This moves the watcher queue declaration from the pipeline level
(where it is no longer valid) to the project level.
https: //lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html
Change-Id: I06923abb00f7eecd59587f44cd1f6a069e88a9fc
(cherry picked from commit 6003322711)
Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.
Until the requirements repository has as stable/yoga branch, tests will
continue to use the upper-constraints list on master.
Change-Id: I96a9c51dc6d88e4c6522d56a9bf9b8a143415bbd
Add file to the reno documentation build to show release notes for
stable/xena.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/xena.
Sem-Ver: feature
Change-Id: If1c02305a153575c6a550844b0c6f45b74ea5ef3
>>> random.sample([5,10], 1.3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.6/random.py", line 321, in sample
result = [None] * k
TypeError: can't multiply sequence by non-int of type 'float'
Change-Id: Ifa5dca06f07220512579e4fe3c5c741aeffc71cc
Block Storage API v2 was deprecated during Pike cycle and is being
removed during Xena cycle, and current v3 API should be used instead.
Change-Id: Ia5247742b31f5f07186ef908588f0972d3ac609f
Python introduced http.HTTPStatus since version 3.5,
and Wallaby has targeted a minimum version of python 3.6.
Change-Id: I45f732f0f59b8fae831bb6c07f4fdd98cdd7409a
Since installing watcher dashboard is fixed in devstack deployments
we can update documentation so it recommends to install dashboard
plugin.
Change-Id: I284a1ec31536ea258cc1979ffd46b22d3e1ac18b
Setuptools v54.1.0 introduces a warning that the use of dash-separated
options in 'setup.cfg' will not be supported in a future version [1].
Get ahead of the issue by replacing the dashes with underscores. Without
this, we see 'UserWarning' messages like the following on new enough
versions of setuptools:
UserWarning: Usage of dash-separated 'description-file' will not be
supported in future versions. Please use the underscore name
'description_file' instead
[1] https://github.com/pypa/setuptools/commit/a2e9ae4cb
Change-Id: Ide4d650a78829a6bc16d86b620e6b3fbed0bba06
The Query.with_lockmode() method is deprecated since version 0.9.0
and will be removed in a future release. [1]
This patch replaces it with Query.with_for_update().
The 'faultstring' was been modified to 'Exactly 5 or 6 columns has to be
specified for iterator expression', so adds one space between "iterator"
and "expression" for 'expected_error_msg'.
Also use upper-constraints in doc build to avoid issues in pdf build.
[1]
https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query.with_lockmode
Closes-Bug: #1933226
Change-Id: I0ad514da647bb08790259fd27e56a41f6dbbbaa0
Add file to the reno documentation build to show release notes for
stable/wallaby.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/wallaby.
Sem-Ver: feature
Change-Id: Ic38b5071799ca733545381e79b956d7f82db2a87
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I207c02ba71fe60635fd3406c9c9364c11f259bae
Current requirements-check job is failing with
below error:
ERROR: Requirement for package PrettyTable excludes a version not excluded in the global list.
Local settings : {'<0.8'}
Global settings: set()
Unexpected : set()
Validating test-requirements.txt
Keeping PrettyTable same as what we have in openstack/requirements repo
Change-Id: I63633d2932757ca23bcea69fd655a2499a5b6d31
There is a commonly shared and proven rpc pattern used
across most OpenStack services that is already implemented
in watcher, but the functions are not used.
This patch basically makes use of the existing
rpc classes and removes some unnecessary code.
Change-Id: I57424561e0675a836d10b712ef1579a334f72018
The check for this call to input() has been removed.
The input method in Python 2 will read from standard input, evaluate and
run the resulting string as python source code. This is similar, though
in many ways worse, than using eval. On Python 2, use raw_input instead,
input is safe in Python 3.
Change-Id: I8654f0c197bfe88796b56e9d85f563cdded6e8a8
Python modules related to coding style checks (listed in blacklist.txt in
openstack/requirements repo) are dropped from lower-constraints.txt
they are not needed during installation.
Change-Id: Iadf4581646131f87803c2cebbc66bd55fdb56685
Add file to the reno documentation build to show release notes for
stable/victoria.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.
Change-Id: I311548732398a680ba50a72273fb98bb16009be4
Sem-Ver: feature
Implements base method as well as some basic implementations to
retrieve time series metrics. Ceilometer can not be supported
as API documentation has been unavailable. Grafana will be
supported in follow-up patch.
Partially Implements: blueprint time-series-framework
Change-Id: I55414093324c8cff379b28f5b855f41a9265c2d3
Create a native Zuul v3 grenade job. It matches the existing job,
even though it doesn't call any local hook as the current legacy
job does (because no local hook exists and it should be rewritten
as zuul configuration if it did).
The new job reuses the variable definition of the devstack watcher
job, so clean up that job as well:
- do not depend on devstack-gate, which is not needed and will be
deprecated soon anyway;
- use the new way (tempest_plugins) to define which tempest plugin
should be installed;
- remove the definition of USE_PYTHON3: true and simply inherit
the value set by devstack;
- remove the definition of PYTHONUNBUFFERED, not really set
anywhere else and only useful back in the days in Jenkins.
Change-Id: Ib0ed3c0f395e1b85b8f25f6e438c414165baab32
It has costs when rollback action_plan.
So give users an option whether to rollback it
when the action_plan fails.
Change-Id: I20c0afded795eda7fb1b57ffdd2ae1ca36c45301
when directly using the `curl` command to create audit template,
strategy name can be accepted.
Closes-Bug: #1884174
Change-Id: I7c0ca760a7fa414faca03c5293df34a84aad6fac
Whether to revert migrate action when the action_plan fails is determained by 'rollback_actionplan' option.
This reverts commit c522e881b1.
Change-Id: I5379018b7838dff4caf0ee0ce06cfa32e7b37b12
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.
Change-Id: I4ee01710d04d650a3ad5ae069015255d3f674c74
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I6cdd4c35a52a014ba3c4dfe4cc2bd4d670c96bc3
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.
Set openstackdocs_auto_name to False to use 'project' variable as name.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: Ia9a3fb804fb59bb70edc150a3eb20c07a279170b
These translation sections are not needed anymore, Babel can
generate translation files without them.
Change-Id: I95bde8575638511449edaa1e546e3399bf0e6451
Since we dropped support for python 2 [1], we no longer need to use the
mock library, which existed to backport py3 functionality into py2.
Which must be done by saying::
from unittest import mock
...because if you say::
import mock
...you definitely will not be getting the standard library mock.
That will always import the third party mock library.
This commit adds hacking check N366 to enforce the former.
This check can be removed in the future (and we can start saying
``import mock`` again) if we manage to purge these transitive
dependencies. I'm not holding my breath.
[1]https://review.opendev.org/#/c/717540
Change-Id: I8c8c99024e8de61d9151480d70543f809a100998
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('watcher/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Co-Authored-By: Sean McGinnis <sean.mcginnis@gmail.com>
Change-Id: Icf35d3a6c10c529e07d1a4edaa36f504e5bf553a
flake8 new release 3.8.0 added new checks and gate pep8
job start failing. hacking 3.0.1 fix the pinning of flake8 to
avoid bringing in a new version with new checks.
Though it is fixed in latest hacking but 2.0 and 3.0 has cap for
flake8 as <4.0.0 which mean flake8 new version 3.9.0 can also
break the pep8 job if new check are added.
To avoid similar gate break in future, we need to bump the hacking min
version.
- http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014828.html
Change-Id: I1fe394ebd1f161eb73f53bfa17d2ccc860b9f51b
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I194eedd505d45137963eb40d1b1d5da2309caeac
Closes-Bug: #1863021
Now that we are running the Victoria tests that include a
voting py38, we can now add the Python 3.8 metadata to the
package information to reflect that support.
Change-Id: Icf85483ff64055d16d35f189755e5fb01fabf574
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Add file to the reno documentation build to show release notes for
stable/ussuri.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/ussuri.
Change-Id: I63fc3e49802f89ac2d967ee089a9dd9dffbe9c78
Sem-Ver: feature
EfficacyIndicator.value is Decimal type, it's
not JSON serializable. So we convert value type
before serialization.
Closed-Bug: #1873377
Change-Id: Id38969775c446bece71f7a85c5c5d3efee9befa0
Error when importing wsmeext.sphinxext
Could not import extension wsmeext.sphinxext
(exception: cannot import name 'l_')
Change-Id: Id23c9c1fd35153d67d4ffb50dc1cd40f30b7ab41
This commit cleanup below:
- Remove python 2.7 stanza from setup.py
- Add requires on python >= 3.6 to setup.cfg so that pypi and pip
know about the requirement
- Add "ignore_basepython_conflict=True" to tox.ini
Change-Id: Ic4fcc1fb15f214ca4204f56ee1ea15dc6a782fc2
Sphinx 3.0.0 breaks the building here, block it for now.
Depends-On: https://review.opendev.org/#/c/717949/
Change-Id: Ibf0c93ea79fec647fbf749257835f1fa99d5f59d
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8.
Remove hacking and friends from lower-constraints, they are not needed
to be installed at run-time.
Change-Id: Ia6af344ec8441dc98a0820176373dcff3a8c80d5
Code in grenade and elsewhere rely on the process/service name
when one runs "ps auxw" and they grep for example "grep -e watcher-api"
to check if the service is running. with uwsgi, let us make sure
we use process name prefix so it is easier to spot the services
and be compatible with code elsewhere that relies on this.
Reference:
https://review.opendev.org/#/c/494531/
Change-Id: I69dbe8840e87a8cb0b2720caa95fb17fb7a30848
There are many warning info in the Watcher api log file,
the reason is that keystonemiddleware only need config
section keystone_authtoken.
refer to https://docs.openstack.org/keystonemiddleware/latest/
Closes-Bug: #1864129
Change-Id: Ie790277d55b3a2d93c26781f7e5e8f66b87227d8
Add a new webhook api and its microversion is 1.4
Partially Implements: blueprint event-driven-optimization-based
Change-Id: I50f7c824e52f3c5fc775d5064898ed422e375a99
This patchset added a new audit type: event,
and the handler to execute event audit.
Partially Implements: blueprint event-driven-optimization-based
Change-Id: I287471ee4d1dcc42af7a6bcc15f8509d4ce73072
Add the releasenote for the general purpose decision engine threadpool.
Including config parameters and how contributors can find relevant
documentation.
Implements: blueprint general-purpose-decision-engine-threadpool
Change-Id: I3560069b4e34f13305950559a0f05f7921f7867e
Now that we are using gitea the contents of our README.rst are
more prominently displayed. Starting it with a "Team and repository
tags" title is a bit confusing. This change makes it start with the
name of the project instead.
Change-Id: Icfce3764aa9e1aabf5e78443cf7ce102de63a052
Documentation with details on general concurrency as well as OpenStack
specific libraries.
Describes how different libraries are effectively used across different
Watcher services. This includes describing how futurist is used in the
Decision Engine and how taskflow is used in the Applier.
Finally, this documentation describes how contributors can use the
new DecisionEngineThreadpool effectively and includes examples.
https://docs.openstack.org/futurist/latest/https://docs.openstack.org/taskflow/latest/
Change-Id: Ic1cd1f3733a0e9a239c9b8d49951e1e4ece49f3a
Partially Implements: blueprint general-purpose-decision-engine-threadpool
We have provided functions to get used and free resources in
class ModelRoot. So strategies can invoke the functions to
get used and free resources.
Change-Id: I3c74d56539ac6c6eb16b0d254a76260bc791567c
Use the general purpose threadpool when building the nova compute
data model. Additionally, adds thorough explanation about theory of
operation.
Updates related test cases to better ensure the correct operation
of add_physical_layer.
Partially Implements: blueprint general-purpose-decision-engine-threadpool
Change-Id: I53ed32a4b2a089b05d1ffede629c9f4c5cb720c8
Implements the singleton general purpose threadpool for the decision
engine and associated tests.
A threadpool is a collection of one or more threads typically called
'workers' to which tasks can be submitted. These submitted tasks will
be scheduled by the threadpool and subsequently executed. How many
tasks will be executed concurrently is managed by the underlying
threadpool and its configuration. In Python the submission of tasks
to a threadpool returns an object called a 'future'. Futures provide
a method to interface with the task being executed that allows to
retrieve information about its state. Such as if it currently is being
executed, if it is waiting on a condition and if it has completed
succesfully. Finally, futures allow to retrieve what has been returned
by the submitted task.
In the case of most OpenStack projects instead of interfacing with native
Python concurrency the futurist library is used. This library provides
very similar interfaces to native concurrency with some extras such as
the wait_for_any method.
For more information about futurist or Python concurrency the following
references can be consulted:
https://docs.python.org/3/library/concurrent.futures.htmlhttps://docs.openstack.org/futurist/latest/reference/index.html#executors
Partially Implements: blueprint general-purpose-decision-engine-threadpool
Change-Id: I94bd9a17290967f011762f2b9c787ee7c46ff930
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.
[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045
Change-Id: If2bbfd8ae6d1fc75cbc494578310c1dc03c367e6
When querying data from datasource, it's possible to miss some data.
In this case if we throw an exception, Audit will failed because of
the exception. We should remove the exception and give the decision
to the strategy.
Change-Id: I1b0e6b78b3bba4df9ba16e093b3910aab1de922e
Closes-Bug: #1847434
Add file to the reno documentation build to show release notes for
stable/train.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.
Change-Id: I0d5ae49a33583514925ad966de067afaa8881ff3
Sem-Ver: feature
Reason:
When there is a compute node but no virtual machine,
the command 'watcher datamodel list' should display
the information of the compute node instead of return None.
Change-Id: Id5ff7f08ac8a9883af9f0313785b756d813ed5a2
Closes-Bug: #1844948
The default planner can not create actions with right order,
The node_resouce_consolidation strategy needs to use its
own planner.
Partially Implements: blueprint node-resource-consolidation
Depends-on: I586e67f782e2965234826634ba3ff51681af4df8
Change-Id: I05b02905a3335a73b6926966de6331c632842293
Add a new pdf-docs environment to enable PDF build.
sphinxcontrib-svg2pdfconverter is used to handle SVG properly.
Change-Id: I1563579486da8912ba8a220bb08a5331e7df910b
It should've been "watcher" instead of "python-watcher" as the
config files are expected to be in /etc/watcher/. Though this is
unlikely to cause problems as this patch corrected the default
config dir.
https://review.opendev.org/#/c/658348/
Nevertheless, we should be using the correct name.
Change-Id: If6b58133eecf2fcc37e11d8c45eaa58f238ea2a8
This component is responsible for selecting an appropriate Planner based
on predefined property value passed to concrete Strategy.
Change-Id: I86de95886df5d7e9558512569601e9ea3babb0e9
Implements: bp watcher-planner-selector
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
This strategy is used to centralize VMs to as few nodes as possible
by VM migration. User can set a input parameter to decide how to
select the destination node.
Implements: blueprint node-resource-consolidation
Closes-Bug: #1843016
Change-Id: I104c864d532c2092f5dc6f0c8f756ebeae12f09e
Many strategies need get node used or free resources, we define
two new method for the purpose in ModelRoot class.
Change-Id: I8cb41fd560dbac9a78d25bfdba51799533db83c2
1. Add datamodel api and policy_enfoce file.
2. Add related unittest for data_model api and policy.
Partially Implements:blueprint show-datamodel-api
Change-Id: I1654685d8cf04db5dd132d43a8640ddf91893cad
1. Add datamodel list endpoint and rpc process.
2. Add datamodel list parased and return.
3. Add related unittest.
Partially Implements:blueprint show-datamodel-api
Change-Id: I758b7ca2bc3d8d596d3457277744336c6629bc4e
The new bp need to get audit type from audit,
so we need to add an audit parameter to do_execute
Partially Implements: blueprint node-resource-consolidation
Change-Id: Ia979781b32202c1821aa1cb91d24253fe6d7bd2d
watcher-tempest-strategies includes all strategies tempest,
we add it and remove all other individual strategy tempest.
Depends-on: I3e45d4a66a6e1bf55499def8550da38ddf01b638
Change-Id: I182bf0ddc528099f5115098b825e9bddae3b187a
As part of Train community goal 'Support IPv6-Only Deployments and Testing'[1],
Tempest has defined the base job 'devstack-tempest-ipv6' which will
deploy services on IPv6.
This commit adds the new job 'watcher-tempest-functional-ipv6-only'
run on gate which is derived from 'devstack-tempest-ipv6'.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- 'watcher-tempest-functional-ipv6-only' will run the tests.
Story: #2005477
Task: #35939
[1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
Change-Id: I42b7e5ff5fd64a21bdb8a32f319759a18c173601
The fields disk and disk_capacity have the same value,
we just need one, so remove disk_capacity field.
Partially Implements: blueprint improve-compute-data-model
Change-Id: If3d385c5e61713bbdc85e22f10cd75e161ff79f0
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I9fe58603692a9850e86a2c36ad7a31c473070100
For Compute node, when calculating resource(VCPU, memory and disk)
capacity, we need to consider reserved resource and allocation ratio.
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I70257dd5fb342a67a3ffda1055eddc54b8360ca3
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I2bb230b5f5a573fb3045261dfdee73f1a8434e0d
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I4f041ad25353d575c276fce87fe13c5e6705754f
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: Id113b4c19792946329e9ff448bfe636cc8eca057
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I7872265b2378e5dc37aa2e086ff1f7fb9071db0b
The node resource(vcpu, memory and disk) used infomation need
to change when creating or deleting instances. Now Placement do
not send notifications, so there is not a good way to capture
the change. We remove these fields and leave the process to strategy.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I3f9a3279a26f3df444117d9265e74cca57b38d6e
This error was discovered by tool coverity. If we don't
initialize this var src_extra_specs, line 225 may sometimes
raise an error.
Change-Id: I992b56b64d56f35c8355b22707c3db5112964b31
The code associated with virtual has been removed before,
and the relevant comments should be removed here.
Change-Id: I7104c1a6752ad0b8c9837a643e51b0a13194a81b
Resource(VCPU, memory and disk) capacity need to be calculated
through formula: capacity = (total-reserved)*ratio.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I15ca66dd2c3a21c5acfebf6f04fa6601aff7918f
We have some new fields(vcpus_ratio, vcpus_used, ...)
in the Watcher ComputeNode. During the process of updating
data model by notifications, we need to get data from
placement.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I10587e93bb3e7be6af78bb3a50509d82d8228f78
The node.free_disk_gb does not take allocation ratios used
for overcommit into account so this value may be negative.
We do not need this field and plan to set disk to total disk
capacity and then remove disk_capacity.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I72c4490f5a8d0fbd1039f70ff20f07b743b6bb2d
check if the resource class(VCPU, memory, disk) in the return
dictionary. If they are, don't need to use dict.get() with a
default value because the parameters are required.
Partially Implements: blueprint improve-compute-data-model
Change-Id: Icb8c672d0e87e6e5f030a2222f928d1bbd069e3c
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: I4101eced9c4bd26741f760e5651204f5d2dfea0f
The fields(vcpus, memory and disk_capacity) in the Watcher ComputeNode
do not take allocation ratios used for overcommit into account so there
may be disparity between this and the used count.
This patch added some new fields to solve this problem.
Partially Implements: blueprint improve-compute-data-model
Change-Id: Id33496f368fb23cb8e744c7e8451e1cd1397866b
Add call_retry method for ModelBuilder classes along with configuration
options. This allows ModelBuilder classes to reattempt any failed calls
to external services such as Nova or Ironic.
Change-Id: Ided697adebed957e5ff13b4c6b5b06c816f81c4a
Actually list_opts() return a list like[1], So we don't need to
convert list to dict and then convert to list[2].
The reason why we need to convert it before is to put together
the same group of configuration objects, but we don't need it
actually.
Now, the list_opts()'s result like this[3].
Reference:
[1]. [(Group1,[cfgObj1,cfgObj2....]),(Group2,[cfgObj3,cfgObj3....])..]
[2]. 375ae32fad/watcher/conf/opts.py (L51-L52)
[3]. [(Group1,[cfgObj1]),(Group1,[cfgObj2]),(Group2,[cfgObj3,cfgObj3....])..]
Change-Id: I50fcc5f812be42038852662639fb10c6dd2f6f72
This lets all the ModelBuilder classes use one baseclass and forces
ClusterDataModelCollector's to pass the scope.
The scopes are still unused in the case of Ironic and Cinder.
The idea is to do several follow ups to this and in the end have a
similar method to query_retry in the datasources baseclass.
Change-Id: Ibbdedd3087fef5298d7f4c9d3abdba05d1fbb2f0
The datasources are only used by the decision_engine, however, they
are placed in a directory one level higher. This patch moves the
datasources code into the decision_engine folder.
Change-Id: Ia54531fb899b79a59bb77adea079ff27c0d518fa
We want to set the value of uuid field of Watcher ComputeNode
to hypversion id(as uuid). So we need to get hypervisor
information by uuid.
Change-Id: I752fbfa560313e28e87d83e46431c283b4db4f23
Related-Bug: #1835192
This error is caused because the condition "is not '':" is not always
true. Sometimes self.aggregation_method['node'] is u'' instead of ''.
This patch ensures that in both cases the behavior is the same.
Change-Id: I7453678cc76892ebeacca23c3501a10a08725d1d
Closes-bug: #1836195
This patch does two things:
1. replace instance's human_id with name.
2. remove ComputeNode human_id.
Now name field in Watcher Compute Data Model is availible.
Use name is better than human_id. For the reason, please see[1].
[1]. https://bugs.launchpad.net/watcher/+bug/1833665
Change-Id: I04f40e7d2a2bda48e9a362f9d0b23f449c40324e
aggregate list and availability_zone list may return ironic type
compute nodes. When building compute data model we should check
the hypervisor_type and remove ironic compute nodes.
Change-Id: Idf404c104c30368baf95ef7d05ad8fc3e7adca38
Related-Bug: #1835183
New datasource to retrieve metrics that can be configured in a
flexible way depending on the deployment. Current implemenation only
works with InfluxDB. Slight changes to datasource manager were
necessary because grafana metric_map can only be built at runtime.
The yaml configuration file can still be used to define metrics
but will require that five different attributes are specified per
metric.
Specific databases accesible through grafana can be defined by
creating 'translators' for these specific databases. This patch
introduces a base class for these translators and their methods.
In addition the first translator specific for InfluxDB is
created.
Depends-on: I68475883529610e514aa82f1881105ab0cf24ec3
Depends-on: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Implements: blueprint grafana-proxy-datasource
Change-Id: Ib12b6a7882703e84a27c301e821c1a034b192508
We want to set the value of uuid field of Watcher ComputeNode
to hypversion id(as uuid). We need a method to get compute
node by name.
Change-Id: I0975500f359de92b6d6fdea2e01614cf0ba73f05
Related-Bug: #1835192
The problem is that watcher is passing limit=-1 to novaclient when
listing servers which will always make at least two API calls to be
sure it's done paging:
https://github.com/openstack/python-novaclient/blob/13.0.1/novaclient/v2/servers.py#L896
If we can determine before we list servers that there are only a
certain number where the number of servers is less than 1000. For
example: 4, we should just pass the limit=len(servers) to novaclient
and avoid the second call for paging which takes extra time and
yields no results.
Change-Id: I797ad934a0f8496dbcbf65798e28b0443f238137
Closes-Bug: #1834679
openstack hypervisor list contains ironic nodes. we should
filter out baremetal nodes when get compute node list.
Change-Id: I4ab3e1a63dc6f61cdc3e99fa2cae749a711459cc
Closes-Bug: #1835183
According to https://review.opendev.org/#/c/251791/,
watcher_messaging group and notifier_driver option
were deprecated.
Change-Id: I2cd114060d1960f77dfa8f4fe0a6d0fc05de5d4c
This is the releasenote for the new grafana datasource it refers to
the documentation on configuring grafana.
Depends-on: Ib12b6a7882703e84a27c301e821c1a034b192508
Change-Id: Icb3939d772f06ad2d66eeba9a59fa8b60822ece0
This is a follow-up to: https://review.opendev.org/#/c/666897/
and makes sure titles and help information get rendered in
the configuration documentation and configuration samples.
The options for the placement_client group are already changed
and left untouched as a result. The changes to grafana_client
are already done in another patch and also untouched.
Change-Id: Ia33cd4576e4b55e651f3f3779a01f2867126138d
"self.assertTrue(action.state, objects.action.State.SUCCEEDED)"
and "self.assertTrue(action.state, objects.action.State.FAILED)"
should use assertEqual.
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
Change-Id: I8e28d651938ca6ed8d12e8a6f5ecf775cf01a39c
This patch implements uWSGI support for Watcher API service.
Because mod_wsgi is deprecated, using uwsgi to replace of mod_wsgi.
Most of Openstack projects have finished it.
Closes-Bug: #1834392
Change-Id: I3fad8d30a15aba493fb91da9337c2515ddea5167
Nova changed the default notification_format from "both" to
"unversioned" in Train [1]. Without configuring nova in the
grenade job we are not testing the nova versioned notification
handler code during upgrades.
Note that grenade only runs stack.sh on the base (old) side so
this change has to depend on a devstack stable/stein change to
add the NOVA_NOTIFICATION_FORMAT variable that we override.
Closes-Bug: #1831917
Depends-On: Ied9d50b07c368d5c2be658c744f340a8d1ee41e0
[1] https://review.opendev.org/603079/
Change-Id: I94c2d14477da185310e0fec596a1ad6436b802f1
This improves the documentation on configuration parameters for the
Grafana datasource.
Follow-up: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Depends-on: I5d1d3129b5d225f0f2fc86d149c046f9aab94d47
Change-Id: Ifd8be7491669c429482d880fdf0219be5ef03163
Nova used to emit versioned and unversioned notiifcations
by default but that changed in https://review.opendev.org/603079/
so now nova emits only unversioned notifications by default.
Watcher listens for versioned notifications so we need to configure
nova to emit both versioned (for Watcher) and unversioned
(for Ceilometer) notifications explicitly.
This adds an override-defaults file so devstack will load up
the nova devstack variable to set the notification_format before
importing and stacking the nova lib script.
Note that this only fixes the non-grenade CI jobs since grenade
requires separate handling for overriding defaults which is proving
hard to do and will be addressed in a separate change.
Partial-Bug: #1831917
Change-Id: I7e441608b38338eecd80e663ed3abe66a89e504f
In the process of handling instance_created.end,
there is a KeyError exception output log. This is because
invoking get_instance_by_uuid before creating the instance
in the data model.
During the review of https://review.opendev.org/#/c/663489/,
reviewers think that it's better to remove the KeyError exception.
This patche seperates the process of instance_created.end from
other Nova notifications and removes the call of get_instance_by_uuid.
Change-Id: Ie9e2d4f5b32ee7a5b52bbcd50abfa81dcabab7bb
Ceilometer removed cpu_util metric in [1].
Another metric compute.node.cpu.percent need to set
compute_monitors option to cpu.virt_driver in the
nova.conf, we should remind user about these.
[1]: https://review.opendev.org/#/c/580709/
Change-Id: I89306ef7c26fa2927945bd4f3ee88b670511d147
This implements the configuration parameters to implement
Grafana as a datasource including the influxdb translator
Change-Id: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Partially-implements: blueprint grafana-proxy-datasource
Replaces the NoSuchMetric exception that was replaced. The exception
is replaced with MetricNotAvailable and test cases are added to prevent
regression.
The changes in the exceptions were introduced in:
https://review.opendev.org/#/c/658127/
Change-Id: Id0f872e916aaa5dec59ed1ae6c0f653553b3fe46
In get_node_by_instance_uuid, an exception ComputeNodeNotFound
will be thrown if can't find a node through instance uuid.
But the exception information replaces the node name with
instance uuid, which is misleading, so we define a new exception.
Closes-Bug: #1832156
Change-Id: Ic6c44ae44da7c3b9a1c20e9b24a036063af266ba
Moves the query_retry method into the baseclass and makes the query
retry and timeout options part of the watcher_datasources config group.
This makes the query_retry behavior uniform across all datasources.
A new baseclass method named query_retry_reset is added so datasources
can define operations to perform when recovering from a query error.
Test cases are added to verify the behavior of query_retry.
The query_max_retries and query_timeout config parameters are
deprecated in the gnocchi_client group and will be removed in a future
release.
Change-Id: I33e9dc2d1f5ba8f83fcf1488ff583ca5be5529cc
We should be starting from stable/stein on the "old" side
of grenade runs now. Rather than hard-code the branch, just
use the BASE_DEVSTACK_BRANCH variable.
Change-Id: I1b0406f870ed0ae5622cfa7421a6cca00d0f891c
When receiving Nova notification instance.create.end,
map instance to its node after adding instance to datamodel.
Related-Bug: #1832156
Change-Id: I6f39e8d935195c611f668f71590e1d9ff52ced0d
In Python, when we use @property, the method will be
decorated by property.
When we call method self.strategy.datasource_backend()[1],
Actually it did two things:
1. call self.strategy.datasource_backend()
2. according to the method's return value[2], call self._datasource_backend()
[1]. https://github.com/openstack/watcher/blob/bd8636f3f/watcher/tests/decision_engine/strategy/strategies/test_base.py#L87
[2]. https://github.com/openstack/watcher/blob/bd8636f3f/watcher/decision_engine/strategy/strategies/base.py#L368
But in this part, we just want it to perform the first step.
So we have to use self.strategy.datasource_backend instead of
self.strategy.datasource_backend()
The reason why the unittest does not report an error is
because the returned value is a mock object, and the second step
is executed without error, for example:
python -m unittest watcher.tests.decision_engine.strategy.strategies.test_base
(Pdb) x=self.strategy.datasource_backend
(Pdb) type(x)
<class 'mock.mock.MagicMock'>
(Pdb) x
<MagicMock name='DataSourceManager().get_backend()' id='139740418102608'>
(Pdb) x()
<MagicMock name='DataSourceManager().get_backend()()' id='139740410824976'>
(Pdb) self.strategy.datasource_backend()
<MagicMock name='DataSourceManager().get_backend()()' id='139740410824976'>
To make the tests more robust, the underlying backend function
is mocked to be not callable.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Change-Id: I3305d9afe8ed79e1dc3affe02ba067ac06cece42
This patch added Placement to Watcher
We plan to improve the data model and strategies in
the future specs.
Change-Id: I7141459eef66557cd5d525b5887bd2a381cdac3f
Implements: blueprint support-placement-api
This makes the ConfFixture extend the Config fixture from
oslo.config which handles cleanup for us. The module level
import_opt calls are also removed since they are no longer
needed.
Change-Id: I869e89c53284c8da45e0b1293f2d35011f5bfbf9
In the process of creating an instance, Nova will emit an
instance.update notification with 'building' state.
This will cause a KeyError exception because this instance
isn't in Watcher datamodel.
So we should ignore the notification instance.update with
'building' state.
Closes-Bug: #1832154
Change-Id: I950eec50d2cee38bd22c47a70ae6f88bbf049080
Now there are some errors when running apidoc,
actually we don't need apidoc, so remove it.
Closes-Bug: #1831515
Change-Id: I3b91a2c05ed62ae7bbd30a29e9db51d0e021410f
The get_compute_node_by_hostname method is given a
compute service hostname and then does two queries to
find the matching hypervisor (compute node) with details:
1. List hypervisors with details and find the one that
matches the given compute service hostname.
2. Using that node, search for hypervisors with the
matching hypervisor_hostname.
There are two issues here:
1. The first query is inefficient in that it has to list
all hypervisors in the deployment to try and match the
one with the compute service hostname client side.
2. The second query is a fuzzy match on the server side [1]
so even though we have matched on the node we want,
get_compute_node_by_name can still return more than
one hypervisor which will result in the helper method
raising ComputeNodeNotFound. Consider having compute
hosts with names compute1, compute10, compute11, compute100,
and so on. The fuzzy match on compute1 would return all of
those hypervisors.
For non-ironic nodes in nova, the compute service host and
hypervisor should be 1:1, meaning the hypervisor.service['host']
should be the same as hypervisor.hypervisor_hostname. Knowing
this, we can simplify the code to search just on the given
compute service hostname and if we get more than one result, it
is because of the fuzzy match and we can then do our client-side
filtering on the compute service hostname.
[1] https://github.com/openstack/nova/blob/d4f58f5eb/nova/db/sqlalchemy/api.py#L676
Change-Id: I84f387982f665d7cc11bffe8ec390cc7e7ed5278
The nova CDM builder code and notification handling
code had some inefficiencies when it came to looking
up a hypevisor to get details. The general pattern
used before was:
1. get the minimal hypervisor information by hypervisor_hostname
2. make another query to get the hypervisor details by id
In the notifications case, it was actually three calls because
the first is listing hyprvisors to filter client-side by service
host.
This change collapses 1 and 2 above into a single API call
to get the hypervisor by hypervisor_hostname with details
which will include the service (compute) host information
which is what get_compute_node_by_id() was being used for.
Now that nothing is using get_compute_node_by_id it is removed.
There is more work we could do in get_compute_node_by_hostname
if the compute API allowed filtering hypervisors by service
host so a TODO is left for that.
One final thing: the TODO in get_compute_node_by_hostname about
there being more than one hypervisor per compute service host
for vmware vcenter is not accurate - nova's vcenter driver
hasn't supported a host:node 1:M topology like that since the
Liberty release [1]. The only in-tree driver in nova that supports
1:M is the ironic baremetal driver, so the comment is updated.
[1] Ifc17c5049e3ed29c8dd130339207907b00433960
Depends-On: https://review.opendev.org/661785/
Change-Id: I5e0e88d7b2dd1a69117ab03e0e66851c687606da
Some of the methods for retrieving data about instances was placed
at the bottom of nova_helper instead of being close to the other
instance based methods.
Change-Id: I68475883529610e514aa82f1881105ab0cf24ec3
This does two things:
1. Rather than make an API call per server on the host,
get all of the servers in a single API call by
filtering on the host. The os-hypervisors API results
to use make this require a bit of refactoring since
get_compute_node_by_name does not have the service
entry in it and get_compute_node_by_id does not have the
servers entry in it. A TODO is added to clean that up
with a single call to os-hypervisors once we have the
support in python-novaclient.
2. Pulls get_node_by_uuid() out of the loop.
A test is added for the nova_helper get_instance_list method
since one did not exist before.
The fake compute node mocks in test_nova_cdmc_execute are
also cleaned up since, as noted above, get_compute_node_by_name
and get_compute_node_by_id don't both return all the details.
Change-Id: Ifd9f83c2f399d4c1765b0c520f4d5a62ad0f5fbd
Fix the list of required metrics from a datasource when testing the
existence of this metric in the metric map.
Change-Id: I19b7408a98893bc942c32edb09f1b3798ec8dc79
With change Id34938c7bb8a5ca934d997e52cac3b365414c006
we require nova API version 2.56 or greater so we can
remove the compatibliity check in the
watcher_non_live_migrate_instance method.
The _check_nova_api_version method is left in place
for future compability checks.
Change-Id: I69040fbc13b03d90b9687c0d11104d4a5bae51d3
The [nova_client]/api_version defaults to 2.56 since
change Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1. There
is compatibility code for that change but if 2.56 is
not available watcher_non_live_migrate_instance will
still fail if a destination host is used.
Since 2.56 has been available since the Queens version of
nova it should be reasonable to require at least that
version of nova is running for using Watcher.
This adds code which enforces the minimum version along
with a release note and "watcher-status upgrade check"
check method.
Note that it's kind of weird for watcher to have a config
option like nova_client.api_version since compute API
microversions are per API request even though novaclient
is constructed with the single configured version. It should
really be something the client (watcher in this case) determines
using version discovery and gracefully enables features if
the required nova API version is available, but that's a bigger
change.
Change-Id: Id34938c7bb8a5ca934d997e52cac3b365414c006
MetricNotAvailable and NoDatasourceAvailable allow to differentiate
between having no datasources configured and a required metric being
unavailable from the datasource. Both exceptions have comments so
that the use case is clear.
The input validation of the get_backend method in the datasource
manager is improved.
Additional logging information allows to identify which metric caused
the available datasource to be discarded.
Tests are updated to validate the correct functionality of the new
exceptions.
Change-Id: I512976cce2401dbcd249d42686b78843e111a0e7
Changes to the baseclass for datasources so strategies can be made
compatible with every datasource. Baseclass methods clearly describe
expected values and types for both parameters and for method returns.
query_retry has been added as base method since every current
datasource implements it.
Ceilometer is updated to work with the new baseclass. Several methods
which are not part of the baseclass and are not used by any strategies
are removed. The signature of these methods would have to be changed
to fit with the new base class while it would limit strategies to
only work with Ceilometer.
Gnocchi is updated to work with the new baseclass.
Gnocchi and Ceilometer will perform a transformation for the
host_airflow metric as it retrieves 1/10 th of the actual CFM
Monasca is updated to work with the new baseclass.
FakeMetrics for Gnocchi, Monasca and Ceilometer are updated to work
with the new method signatures of the baseclass.
FakeClusterAndMetrics for Ceilometer and Gnocchi are updated to work
with the new method signatures of the baseclass.
The strategies workload_balance, vm_workload_consolidation,
workload_stabilization, basic_consolidation, noisy_neighbour,
outlet_temp_control and uniform_airflow are updated to work with the
new datasource baseclass.
This patch will break compatibility with plugin strategies and
datasources due to the changes in signatures.
Depends-on: I7aa52a9b82f4aa849f2378d4d1c03453e45c0c78
Change-Id: Ie30ca3dbf01062cbb20d3be5d514ec6b5155cd7c
Implements: blueprint formal-datasource-interface
As a follow up to the recent test improvements for Ceilometer this
patch ensures that the same test pattern is used for Gnocchi and
Monasca as well. This ensures that the mocked functions will be called
with matching signatures.
Change-Id: Ic14a4c087f3961a4b4f373e2e3d792aba71868f6
Override the metric map of each datasource as soon as it is created by
the manager. This override comes from a file whose path is provided by
a setting in config file.
Loading at creation time allows the correct datasource be used when
get_backend is called, this allows loading a datasource whose metric
names get updated outside the watcher's codebase.
The function 'load_metric_map' returns empty-dict in any error case.
Also in case the file is empty where safe_load is unable finds any
yaml documents, it will return None. [1]
Some minor refactoring in the test_manager file for readability and
added tests for file load and metric override.
1 - https://pyyaml.org/wiki/PyYAMLDocumentation
Change-Id: I1df16245f4c7dfd34066f3ab0553cd67154faa58
Implements: blueprint file-based-metric-map
Some users may want to create keystoneclient by specifying the
type of endpoint and region name, so we need to supply the option
for user to choose.
Implements: blueprint support-keystoneclient-option
Change-Id: I49b33a69ec99d2a91568ce27ef89dc80b75e7091
Change I25b4cb0e1b85379ff0c4da9d0c1474380d75ce3a in
Queens refactored the statistic_aggregation method
and renamed the "aggregate" kwarg to "aggregation",
presumably to match the signature of the GnocchiHelper
statistic_aggregation method (the commit message does
not give details) so a base method could be added to
the parent class for all datasource helpers.
As a result, the CeilometerHelper calls to its
statistic_aggregation started passing the new
"granularity" kwarg but failed to match the rename
to the "aggregation" kwarg, which breaks the
CeilometerHelper. This was missed by the unit tests
because the tests were just asserting the erroneous
call that the runtime code made.
This change fixes the kwarg typo and makes the
tests more robust by using the mock spec kwarg
to define a spec for the statistic_aggregation
mock so that it must be called with the correct
parameters defined in the method signature. The
test is refactored to reduce duplicate mocking.
The same test hardening can and should be done
in the gnocchi and monasca helper tests but that
should be done in a separate change.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Closes-Bug: #1829542
Change-Id: Idfd099f718873d9056fdc35a97954771c9ae5762
As of change Ic4659d1f18af181203439a8bf1b38805ff34c309 the
nova CDM will not be built until an audit is performed.
Instances and services (compute hosts) can be created and
deleted before an audit is performed which will attempt
to use the notification callback function which relies
on the CDM being built already, and if not results in
an AttributeError.
This change side-steps that issue by checking to see that the
nova CDM exists before trying to call the notification
callback function.
An alternative to this is forcefully create the nova CDM when
notifications are received before an audit which is what happend
before change Ic4659d1f18af181203439a8bf1b38805ff34c309.
Change-Id: I16990afb82019821c443c9df26d3e515e52efa69
Closes-Bug: #1828582
_post_live_migration[1] runs on the source host and calls
post_live_migration_at_destination on the dest host which
emits the instance.live_migration_post_dest.end notification:[2]
But it's not the last notification for the live migration operation.
so we should use instance.live_migration_post.end instead of
instance.live_migration_post_dest.end notification.
[1]daa2ac2287/nova/compute/manager.py (L6907)
[2]daa2ac2287/nova/compute/manager.py (L7035)
Change-Id: Id1e2d98f56d5a95d49e32f98d2910660b9f48ce6
The version of bandit in lower-constraints (1.4.0) does
not match the version in test-requirements (1.6.0) however
bandit is a test-only dependency and there is no test coverage
for bandit in the lower-constraints tox job target, so there
is really no good reason to have bandit in lower-constraints.
As such, this change simply removes it from lower-constraints.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Change-Id: I35f66994e9a3a334b342232587d84491542da755
Sphinx 2.0 no longer works on python 2.7, so we need to start capping
it there as well.
The errors are as follow:
Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.5'
does not match "python_version>='3.4'"
Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.5'
does not match "python_version=='2.7'"
Could not find a global requirements entry to match package sphinx. If the package
is already included in the global list, the name or platform markers there may not
match the local settings.
Change-Id: I6dad56ffbb9e85e36cacea1a89565c2fc8248fbf
The final Stein version of Watcher was 2.0.0
so this fixes the version mentioned in the
watcher-status man page docs.
Change-Id: I7fce35471cf31222f9cdafc35e5a7b287bc4598e
The _add_virtual_layer and _add_virtual_servers methods
have not been used since Ic4659d1f18af181203439a8bf1b38805ff34c309
in Stein so this change removes them.
Change-Id: I8c05f29c3c03aa5897cb182bb492948771c42881
This enhances the [collector]/collector_plugins
config option help text to mention the storage
and baremetal in-tree collectors and the ability
to load out-of-tree collectors via extension point.
While doing this, the help text is formatted for
prettier rst rendering in the docs.
Change-Id: Ifd32c95c664c4e9586c250e6bceaeaba2e2df417
CeilometerClient has been deprecated and is no longer available for
master. Without ceilometer client installed docs fail to build with
an exception [1].
This patch marks the import optional.
1 -
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/sphinx/config.py", line 368, in
eval_config_file
execfile_(filename, namespace)
File "/usr/lib/python2.7/site-packages/sphinx/util/pycompat.py", line
150, in execfile_
exec_(code, _globals)
File "/usr/lib/python2.7/site-packages/six.py", line 709, in exec_
exec(""exec _code_ in _globs_, _locs_"")
File "<string>", line 1, in <module>
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/doc/source/conf.py",
line 20, in <module>
objects.register_all()
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/objects/__init__.py",
line 31, in register_all
__import__('watcher.objects.action_plan')
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/objects/action_plan.py",
line 78, in <module>
from watcher import conf
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/conf/__init__.py",
line 28, in <module>
from watcher.conf import datasources
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/conf/datasources.py",
line 21, in <module>
from watcher.datasources import manager
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/datasources/manager.py",
line 19, in <module>
from watcher.datasources import ceilometer as ceil
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/datasources/ceilometer.py",
line 21, in <module>
from ceilometerclient import exc
ImportError: No module named ceilometerclient
)
Change-Id: Idcf582c2495aab39aacf691b687759405bb94dca
Currently default config files are being for initialization of CONF from
oslo_config. However default config dirs are not being passed as a
result watcher components (eg: decision-engine) are unable to load
files from default directories (eg: /etc/watcher/watcher.conf.d)
supported by oslo_config. This is a short-coming on watcher's side.
Also this forces user to have multiple config for each component.
Without this default set, oslo_config will search for conf with string
'python-watcher' in it, eg: /etc/python-watcher/.... Since there is a
because project=python-watcher a couple of lines below
This patch adds the option after evaluating using project as 'watcher'
which is similar to evaluation of default_config_files and also allows
it to be passed in as a function parameter.
Change-Id: I013f9d03978f8716847f8d1ee6888629faf5779b
This fixes the wrong installation guide link from the
user guide which was pointing to the watcherclient docs
for some reason, maybe it was just a copy/paste error.
Change-Id: I38f536e187245523ac37d70054a2df8cdfcbe4b2
Closes-Bug: #1828584
Hard-coding watcher.openstack.common to warning level logging
only makes it hard to debug watcher's interactions with other
services, like when it's triggering and monitoring a server live
migration.
Since debug logging is controlled via the "debug" configuration
option, we can just rely on that to filter out debug logs within
watcher itself.
Note this has been this way since change
I699e0ab082657880998d8618fe29eb7f56c6c661 back in 2015 and there
was no explanation why the watcher.openstack.common logging
was set to WARN level.
Change-Id: I939403a4ae36d1aa9ea83badb9404bc37d18a1a6
Related-Bug: #1828598
The -x option for bandit changed in 1.6.0 and now
supports glob patterns so use that to correctly
exclude test code from bandit scans.
Since this change requires bandit>=1.6.0, we have
to also fix the networkx requirement to pass the
requirements-check job so that the networkx requirement
matches what is in global-requirements from change
I0a9700926c9a0db93e782c853c33f1aaee3d4876.
Change-Id: I4fc1166daee5d8739296419216d11d684be27c0a
Closes-Bug: #1828419
Allows to define a global preference for metric datasources with the
ability for strategy specific overrides. In addition, strategies which
do not require datasources have the config options removed this is
done to prevent confusion.
Some documentation that details the inner workings of selecting
datasources is updated.
Imports for some files in watcher/common have been changed to resolve
circular dependencies and now match the overall method to import
configuration.
Addtional datasources will be retrieved by the manager if the
datasource throws an error.
Implements: blueprint global-datasource-preference
Change-Id: I6fc455b288e338c20d2c4cfec5a0c95350bebc36
To get log formatting like the other openstack projects
running in devstack the setup_logging function should be
used. This will also give us the "Display level" formatting
in the logs via the os-loganalyze packaged used by infra.
Needed by: https://review.opendev.org/657652
Change-Id: I5e9bd5a142e45804e8d915b370746a2142243088
Exceptions should be reraised with just "raise" and not "raise e" to
preserve the traceback. This also addresses a couple cases where the
catching and reraising of the exception was not actually doing anything.
Change-Id: I94ba193f728ee7ca6f689f70fc08317a1dd50c92
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
We have a bp to add resource_name in action input parameter field.
Before doing this, one of this method's parameter should be node
instead of resource_id.
Change-Id: I4ce5ae97efce98d80a460fd6003df3cc5cacab82
This resolves problems with the audit scope such as the scope being
ignored, the scope not merging due to a type in .append, change update
into .add method when adding single elements to a set and making the
access of dict keys and values as lists work in python 3.7.
All these methods from the model builder now have tests to prevent
regressions.
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
Change-Id: I287763d5e426ff860aefabc4a1f3fe3f51accd76
Since commit I8df8921337ea3f4e751c0c822d823e64e3ca7e1c
the check for hardware.cpu.util was removed.
But it can be still used in workload stabilization.
Change-Id: I301487837aac2e1e63bce16a79d0f8136452c313
Our cgit instance will be going away and opendev.org is the new
preferred URL for browsing our git repos. Redirects will exist for the
foreseeable future, but it's more efficient to just go directly to the
new locations.
Change-Id: I7dd9d454da63167832bab02c89be98a2ce03b72a
Now there are only one scheduler for launching audit task and
executing audit jobs. We have found an exception where the scheduler
stops for some reason when executing audit.
In order to keep launching audit task normal, we need to split into
two schedulers.
Change-Id: I45dccaf062290cfc7d7fcfc27fe11d6f87f38afa
When the lower-constraints tox target was added, it was assumed the
install_command was just running the install and that the dependencies
and constraints were being set using "deps = ".
This fixed the install_command and deps to follow the expected pattern
so the lower-constraints job actual does install the lower constraints.
This also raises the oslo.context minimum as
Ic96c1f1e1a80099d9dafa95a014fc47f05b88e42 added a dependency on a newer
versions kwarg.
Depends-On: https://review.openstack.org/#/c/647726/
Change-Id: I4cc2c3ac158a607b22295c50f83896969a4007ee
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
When use tox-elower-constraint, we will meet these errors:
1. line 417, in test_clients_ironic:
b"AssertionError: Expected call: Client('1', endpoint_override...
b"Actual call: Client('1', 'http://localhost:6385/'...
2. line 39, in test_wrong_major_version:
b"KeyError: 'HTTP_ACCEPT'"
3. RUN END RESULT_TIMED_OUT:
[untrusted : git.openstack.org/openstack-infra/
zuul-jobs/playbooks/tox/run.yaml@master]
For the first error, The reason is that the unittest for the
ironicclient is too strict and must be adapted to the latest code.
In fact, the watcher can use the previous ironicclient version.
Therefore, we modified the unittest so that the watcher does not
have to rely on the latest ironicclient version.
For the second error, The reason is that we need to update the minimum
version of pecan and webOb.
For the third error, the reason is that the version of the oslo_messaging
is too low.
Change-Id: Icb3eda3d27fa4452e13e2dcd3c016cc76fc2c7c7
Metrics for datasources now match the name of their corresponding
abstract methods. This ensures that developers know how the method
is named if they know the name of the metric and vice versa.
Change-Id: I0f9d400432d8182b3f10a0da97155e6cb786690e
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I3452a7802dde00d8be32c833d714b2974be58e16
Add file to the reno documentation build to show release notes for
stable/stein.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.
Change-Id: I25931207ed6066f905fe66ca504fa230e40d12dc
Sem-Ver: feature
Although this method does not report an error, this type check 'int'
is redundant and may be misleading.
Refercnce code url: https://github.com/openstack/wsme/blob/master/wsme/api.py
Change-Id: I631b5f9901790666e7f20275e8c8b99f06f06f0a
Many strategies execute very similar statements especially in
pre_execute and some might raise errors that others might not. This
same pattern of many similar statements can also be observed in
strategies their tests.
This patch addresses these issues, firstly; the BaseStrategy class gets
1 additional method _pre_execute which allows for general logic that
most strategies perform at that stage. This method can be executed
before the similarly named method of the superclass. A notable change
is that _pre_execute now handles common exception handling for
ClusterStateStale & ClusterStateNotDefined exceptions.
A similar pattern is applied to the test classes of the strategies
each of these classes now inherits from the TestBaseStrategy class.
This class provides the common attributes almost every test class for
the strategies requires such as: The mocked compute_model, mocked
audit_scope and an instance of FakerModelCollector.
Finally, some minor changes were required in test_strategy_context
& test_audit_handlers and exceptions around 0 nodes in cluster or
storage are removed.
Change-Id: Ia7154376b2448aac65cf17999cc8c3e1c8309b5b
This patch adds a scope to the datamodel, which only gets the VMs
of the specified nodes, and no longer gets all VMs from nova.
Implements: blueprint scope-for-watcher-datamodel
Change-Id: Ic4659d1f18af181203439a8bf1b38805ff34c309
Audit will only failed if an exception occured.
The situation that no solution found will not cause audit failed.
Change-Id: Ib9c3c3505f31c14500926ec13aa865dc8f7aa310
on ImportError set HAS_CEILCLIENT to false
Without this none of the watcher componenets can be started for master
as well as rocky because the ceilometercleint was deprecated.
Using the variable the support for ceilometer can be gradually removed
from master
A backport to rocky will allow using watcher without ceilometerclient.
Change-Id: I3beb0fb8f0a8e8e0a22acaf6bdeca492836bbee2
Moved the metric mappings for Ceilormeter, Gnocchi & Monasca out of
base.py. The datasources manager now uses classes extending base.py
their NAME attribute as key in the dictionary of total available
mappings and datasources. base.py still contains a template
definition of all available mappings so that anyone extending the
base class can identify all the possible endpoints they can map to.
Change-Id: I6a826423031b5a6a60c4cd5fe24f74b8400f6b55
Closes-Bug: #1815769
Prevent workload_stabilization strategy from failing in a network with
0 hosts.
Change-Id: I9f1a9524923c14d958eb50a70dad379a6021b884
Closes-Bug: #1815059
Small cleanups:
* Use openstack-lower-constraints-jobs template, remove individual
jobs.
* Sort list of templates
Change-Id: I63bfcd9bc21011b446fd1c54cb64c5568c601687
Needed-By: https://review.openstack.org/623229
This patch set removes "observable" and "synchronization"
modules cause they aren't used by any Watcher modules so far.
Change-Id: If23cdf0d3d09087919d48f50ab38b0d355c36481
Ceilometer Datasource has been deprecated since its API has been
deprecated in Ocata cycle. Watcher has supported Ceilometer for some
releases after Ocata to let users migrate to Gnocchi/Monasca datasources.
Since U-release, Ceilometer support will be removed.
Change-Id: I944a5a562ab09a36961eb9b75e9a5144ba0b9ca4
file host_maintenance.py
This is to fix spelling error and unsuitable punctuation
in file host_maintenance.py
Change-Id: I9c535059c3a02277be4c7329693db34fb7674b4e
Bare metal cluster data model was introduced in Queens cycle.
Since the model is different from compute data model, we
need add CDM scoper for bare metal cluster data model
Change-Id: Idd041cefb692085d4545252d229ebe8602926b58
Implements: blueprint audit-scoper-for-baremetal-data-model
vm_workload_consolidation.py
Increase the test of the execute method which contains
the pre_execute(), do_execute(), post_execute() methods.
Increase coverage from 82% to 87%.
Change-Id: Ibde67d7b7d7945657ad0b674e06b1edc9eb24a9f
When tls-proxy is enabled, first start the tls-proxy and then then wait
for api to come up.
Without this the api comes up on the internal port as a result the
subsequent curl fails killing the deployment
- create a zuul job to test with tls
- fix apache ports accordingly
Depends-On: Ie665240b53df92b8e5ca509e998e95d859bd5282
Change-Id: I610a7a24daab68c7ab0e30977e3cabd62cdb56a5
Actually, the metric "cpu_util" and "memory.resident" are necessary
in vm_workload_consolidation.py, according to line 75. So modify
this document about this part.
Change-Id: I648f341184a0b42d88dcb4c934af989997fe3fea
This patch updates response header
from OpenStack-API-Version: [VERSION_STRING]
to OpenStack-API-Version: [SERVICE_TYPE] [VERSION_STRING]
Change-Id: I10577ff1123ef781bd4aa0b26577574a3f7e9c39
Add a new config option 'action_execution_rule' which is a dict type.
Its key field is strategy name and the value is 'ALWAYS' or 'ANY'.
'ALWAYS' means the callback function returns True as usual.
'ANY' means the return depends on the result of previous action
execution. The callback returns True if previous action gets failed,
and the engine continues to run the next action. If previous action
executes success, the callback returns False then the next action
will be ignored.
For strategies that aren't in 'action_execution_rule', the callback
always returns True.
If exception is throwing out during the action execution, reverting will
be triggered by taskflow. To continue executing the next action,
we return False instead of throwing an exception.
Change-Id: Ib5afa214d8d097d739aad35d18b3fe5c8e4de8fc
Implements: blueprint enhance-watcher-applier-engine
With these brackets, the statement "raise Exception("Live migration
execution.....") in watcher/common/nova_helper.py, line 379 will never
be executed. So remove it and let the statement make sense.
Change-Id: I42a2fa0c8ffa9c84a918d432c5093470dbd80f82
The method is quite simple and it doesn't need a dostring.
Also existing docstring was incorrect. The name of the expected
parameter is 'name', not 'node'. And it cannot be an object
of the type node.StorageNode
Change-Id: I94124d327c490d45eae4d2ded218beadfbc33ad7
The commands used by constraints need at least tox 2.0.
Update to reflect reality, which should help with local running of
constraints targets.
Change-Id: I0eb9af735f34ad259c7099729d7d465a1276fc5f
The correct type of parameter 'pool' in method build_storage_pool is
<class 'cinderclient.v2.pools.Pool'>
Change-Id: I986f707e4e740ebec94a46c6ee413f9a70197dad
This patch set adds API microversion support along
with the first API microversion: start/end time for
CONTINUOUS audits.
APIImpact
Implements: blueprint api-microversioning
Depends-On: I6bb838d777b2c7aa799a70485980e5dc87838456
Change-Id: I17309d80b637f02bc5e6d33294472e02add88f86
Now we have removed nova legacy notifications in Watcher
and just consume nova versioned notifications,
we don't need notification config in nova.conf
Change-Id: I1c9c141d98d858c36ad8bb7be0b95c38ff1d5752
This commit adds the functionality of watcher-status CLI for performing
upgrade checks as part of the Stein cycle upgrade-checkers goal.
It only includes a sample check which must be replaced by real checks in
future.
Change-Id: Ic3d066af439797d6f705e805334f729b52ce3aac
Story: 2003657
Task: 26164
Add new start_time and end_time fields in the audit table
Partially Implements: blueprint add-start-end-time-for-continuous-audit
Change-Id: I6bb838d777b2c7aa799a70485980e5dc87838456
As rpc_backend config option has been removed from
oslo_config [1], projects should not use it.
Current uses of it cause watcher crash when installing
via devstack.
[1] https://review.openstack.org/#/c/580910/
Change-Id: Iba7471e87e8935f1ea02b363f269e9debdc7cc71
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')
Change-Id: I0371fc2c0878a177c0a9e9c9313ca5b8470bfd98
This patch set fixes process of audit creation and
allows to create audit without Audit Template using only
names of Goal and Strategy. It also provides some additional
unit tests to improve tests covering.
Change-Id: I89a9c7661616f49639151869055d8f5ebe723d5f
Closes-Bug: #1794233
This patch set adds efficacy indicators for workload_balancing
goal (that includes workload_stabilization and workload_balance
strategies so far).
Change-Id: I5b04d084ace7c661001c62f07b8308e5763e144d
oslo_context may add new fields in request context, there are no
need to warning these fields.
Closes-Bug #1790577
Change-Id: Ic96c1f1e1a80099d9dafa95a014fc47f05b88e42
This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I5e75f2ea7dd02065bc18793d974f56fef2daa2c4
Story: #2002586
Task: #24344
As openstack installation guides suggest to run mysql with root shell
user, mysql will not ask for password, so in
controller-install-*.rst "-u root -p" is useless.
Change-Id: I511f39d734702ab3d1a209f7d868f52fb184f1fc
Related-Bug: #1785025
This patch set refactors logs of workload stabilization
strategy to make them more readable and sensible.
Change-Id: I408988712bb7560728157f3b4e4f2b37572128c4
We just need to check decision engine service status
when Rescheduling continuous audits.
This is an update for 1
1:https://review.openstack.org/#/c/586033
Change-Id: I05a17f39b6ff80c6b9382248c72cac571191e395
We have replaced voluptuous with jsonschema in [1].
Now voluptuous can be removed.
[1]: https://review.openstack.org/#/c/561182/
Change-Id: I99c65ed79ef166839838559a808ee7607389e07a
This patch set adds background job that reschedules CONTINUOUS
audits from FAILED to ACTIVE decision engine nodes using round-robin
algorithm. It also contains fix for main[1] HA PS about filtering audits.
[1]: https://review.openstack.org/#/c/578102/
Partially-Implements: blueprint support-watcher-ha-active-active-mode
Change-Id: Ib248a6cd3adbd3927c47db6bb819300361492411
This patch set adds hostname field to Audit and Action Plan
objects to track services which execute these objects.
Change-Id: I786e419952925c380c969b12cc60f9a1004af96b
Partially-Implements: blueprint support-watcher-ha-active-active-mode
This commit allows to consume notifications via notifications pools[1].
Listeners in notification pools recieves a copy of notification. It
will let Watcher to sync Data Models of Decision Engines.
[1]: https://docs.openstack.org/oslo.messaging/ocata/notification_listener.html
Change-Id: Ie37528263181924f84510500fc1277b0237c1df8
Partially-Implements: blueprint support-watcher-ha-active-active-mode
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.
[1] https://etherpad.openstack.org/p/YVR-python-pti
Change-Id: I8f70f7d8a3d18301559c0eb47e6a64c8b5100d39
Non-voting jobs should not be in gate queue at all, remove them.
They are wasting needlessly are resources.
To make clearer which jobs are non-voting, move voting:false
from base job to individual jobs.
Make watcherclient-tempest-functional as voting job and explicitly set
it non-voting in project stanza so that it can be easier to make it
voting in other repos.
Change-Id: Id94e49347006bca850f72a0400794da8c2a67144
The legacy-rally-dsvm-watcher-rally job does not exist but it is listed
in the .zuul.yaml config. This is a zuul configuration error. Remove
this job which does not exist to fix zuul.
Change-Id: I1bbfd373ad12b98696ab2ddb78e56e6503cc4c4d
This patch fixes watcher.tests.common.test_clients.TestClients.
test_clients_ironic unit test error due to python-ironicclient 2.4.0.
Change-Id: I0aaa2047cc8fcf09dee8bc7168e35f7a58c10125
This patch replaces voluptuous with JSON-schema to validate
efficacy indicator since in watcher we want to remove voluptuous
and use JSONSchema as our only JSON validation tool to keep consistency.
Change-Id: Iaa77566f1cdfdac03ce8e7d5a75406274c7d5298
Implements: blueprint replace-voplutuous-with-jsonschema
url in action_plan policy file is:
'path': '/v1/action_plans/{action_plan_uuid}/action'
whereas it shouls be :
'path': '/v1/action_plans/{action_plan_uuid}/start'
Related-Bug: #1756274
Change-Id: Ic15fed9af739b59efb2777b70514697747b2af7f
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: Ia481330b9a889b113b585fca0d4ddb86df9f74d3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The python 3 version of the linter does not allow Windows-style line
endings (\r\n) so replace them with UNIX-style endings (\n).
Change-Id: Ifb97491323d3df92bb1520e373552aeb5e1919a4
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
oslo_messaging's rpc_backend setting, which is set by
ConfFixture.transport_driver has been deprecated since Newton. To allow
oslo_messaging to remove it, switch to setting transport_url instead.
Change-Id: Ie37b20d1be6c177f2f1c26cc473b632d7d934c05
Partial-Bug: #1712399
This patch set adds API Reference along with some
fixes to documentation. It partially fixes bug #1757423.
Change-Id: I107b4fd5daf40aad63fc13864debbbbc82a9826c
Storage CDM can not be build for some environment such as
the one using VMwareVcVmdkDriver, since some attributes of
Storage CDM'S pool element can be 'unknown'.
This patch updates storage CDM collector to raise watcher
specific exception if some attributes of storage CDM'S pool
element is 'unknown'
Change-Id: If75a909025c8d764e4de6e20f058b84e23123c1a
Closes-Bug: #1751206
Now that the v2.0 API has been removed, we don't have a reason to
include deployment instructions for two separate applications on
different ports.
Related-bug: #1754104
Change-Id: I98fae626d39cb62ad51c86435c1a2c60be5c1fb9
maintain one compute node without having the user's application
been interruptted.
It will firstly migrate all instances from the maintenance node
to one backup node. If not, it will migrate all instances,
relying on nova-schduler.
Change-Id: I29ecb65745d5e6ecab41508e9a91b29b39a3f0a8
Implements:blueprint cluster-maintaining
Currently RabbitMQ connection gradually increases by CONTINUOUS audit
with auto-trigger option.
This patch fixes watcher to reuse RabbitMQ connection.
Change-Id: I818fc1ce982f67bac08c815821f1ad67f8f3c893
Currently the REST API to start action plan in watcher
is which is same as for update action plan.
PATCH /v1/action_plans
https://docs.openstack.org/watcher/latest/api/v1.html
we need to make it easy to understand like :
POST /v1/action_plans/{action_plan_uuid}/start
the action should be start in above case.
Change-Id: I5353e4aa58d1675d8afb94bea35d9b953514129a
Closes-Bug: #1756274
Beginning with the Queens release, the keystone install guide
recommends running all interfaces on the same port. This patch
updates the install guide to reflect that change.
Change-Id: Ice155d0b80d2f2ed6c1a9a9738be2184b6e9e76c
Closes-bug: #1754104
In I4d2f44fa149aee564c62a69822c6ad79de5bba8a we introduced new
_get_model_list method that introduces unify way for retrieving models
from db. This commit adds tests that do checks on bug 1761956, when
selecting with filter() method could return deleted entites.
Change-Id: I12df4af70bcc25654a0fb276ea7145d772d891e2
Related-Bug: 1761956
Both Continuous and Oneshot audits made same action in
do_execute, so it's a good idea to move it to the base
class
TrivialFix
Change-Id: Ic0353f010509ce45f94126e4db0e629417128ded
This patch adds project_id in compute CDM, It also adds logic for
excluding project_id in audit scope.
Change-Id: Ife228e3d1855b65abee637516470e463ba8a2815
Implements: blueprint audit-scope-exclude-project
Since Nova API v2.56, Nova migrate Server(migrate Action) API
has host option.
This patch replaces cold migration implementation to use the API.
Change-Id: Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1
Implements: blueprint replace-cold-migrate-to-use-nova-migration-api
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I2ef330d7f9b632e9a81d22a8edec3c88eb532ff5
When we call audittemplate list without filters, it returns all Audit
Templates that are not deleted, as expected. If we add any filter to
query and context.show_deleted is None (we request only current AT),
query.filter_by adds filter to joined table (for example, goals, results
in a query like JOIN goals ... WHERE ... goals.deleted_at IS NULL) not
to model's table (AuditTemplate in our case).
We change call for filter_by to filter, explicitly point to model that
we want to filter.
Also, we moved query generating code to new method _get_model_list(). As
a result we applied same fix to all of the other models.
Change-Id: I4d2f44fa149aee564c62a69822c6ad79de5bba8a
Closes-bug: 1761956
New releases of oslo.config support a 'mutable' parameter to Opts.
Configuration options are mutable if their oslo.config Opt's
mutable=True is set. This mutable setting is respected when the oslo
method mutate_config_files is called instead of reload_config_files.
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e allows services to tell
oslo.service they want mutate_config_files to be called by specifying
the 'restart_method=mutate' parameter, what this patch does.
The default mutable configuration options (set by oslo.config Opts'
mutable=True) are:
- [DEFAULT]/pin_release_version
- [DEFAULT]/debug
- [DEFAULT]/log_config_append
Concrete params, that made mutable in Watcher:
* watcher_decision_engine.action_plan_expiry
* watcher_decision_engine.check_periodic_interval
* watcher_decision_engine.continuous_audit_interval
* gnocchi_client.query_max_retries
* gnocchi_client.query_timeout
* DEFAULT.periodic_interval
Change-Id: If28f2de094d99471a3ab756c947e29ae3d8a28a2
Implements: bp mutable-config
Removed duplicates of strategies descriptions, added references to
that descriptions instead of module descriptions.
Change-Id: Ife396ddce5c3cc926cc111f1ff1abd3a42c22561
Added creation of [DEFAULT]/transport_url value
in devstack.
Also, fixed same topic in docs.
Change-Id: I9ad9475c4fccf023daac40c0b1e841eeeb22f040
Closes-Bug: 1738329
This patch set removes legacy-* jobs and migrates
tempest functional job to ZuulV3 syntax.
Change-Id: I87771737cc713eae20b4d6aaaefefc5e40875666
Implements: blueprint migrate-to-zuulv3
While sorting output of list command ("audittemplate list",
"strategy list", etc) by sort-key that is not belongs
to specific model, this sort-key was passed to db what
caused error (HTTP 500). We added check on such keys and now,
if got one of them, then we make sort on API side
instead of db side.
We removed excess sort and changed all sorting routines
to unified way.
Also added sort tests on every model.
Change-Id: I41faea1622605ee4fa8dc48cd572876d75be8383
Closes-Bug: 1662887
This patch removes the unnecessary maintenance of a date and version
from the CLI documentation.
NOTE: Cinder/Nova teams also did the same removal with
the commit Idf78bbed44f942bb6976ccf4da67c748d9283ed9
and the commit I0a9dd49e68f2d47c58a46b107c77975e7b2aeaf7
Change-Id: I6a0faeb596f1ee3a3b67d1d37a14e1507aa40eba
With current URL [1], default driver will be used.
In order to ensure the compatibility, it is better to include the exact driver [2].
[1] connection = mysql://
[2] connection = mysql+pymysql://
Change-Id: I4f7b3ccbecfb2f1e2b3d125179dbd5c6fbf5e6b9
The change is wrong. We link on purpose to the unversioned version and update that one once rocky is released.
This reverts commit e771ae9e95.
Change-Id: I0f981a8473a47d18ce20be74a8e2d12d22f40061
NotImplementedError are reported in decision-engine log file
when we activate storage data model and see a Guru Meditation Report.
This patch fixes by adding default values.
Change-Id: I06386f8295f7758cbb633612eee8b19225905c92
Closes-Bug: #1750300
The string %(action) is not valid, it misses the conversion specified,
add s for string.
Note that this leads to an untranslatable string, since our translation
tools check for valid formats and fail. In this case the failure comes
from a source code fail.
Change-Id: I2e630928dc32542a8a7c02657a9f0ab1eaab62ff
Replace the old http url links with the lastest https ones according
to the official OpenStack website.
Change-Id: I1abd79bb80dae44ee2ba5946b8a375c7096b39d6
This patch set completes schema by adding restrictions
to different types of schema properties.
It also makes workload_stabilization strategy more
user friendly by setting cpu_util as default metric.
Change-Id: If34cf4b7ee2f70dc9a86309cb94a90b19e3d9bec
Replace 'a instance' with 'an instance' in
watcher/decision_engine/model/collector/nova.py
watcher/decision_engine/model/element/instance.py
Change-Id: I39020f3e7b460dea768f7e38fef9ae9e2a4b7357
Zuul no longer requires the project-name for in-repo configuration.
Omitting it makes forking or renaming projects easier.
Change-Id: Ib3be82015be1d6853c44cf53faacb238237ad701
This patch set fixes issues that have appeared after merging
watcher-multi-datasource and strategy-requirements patches.
It is final commit in watcher-multi-datasource blueprint.
Partially-Implements: blueprint watcher-multi-datasource
Change-Id: I25b4cb0e1b85379ff0c4da9d0c1474380d75ce3a
If hostname is different from uuid in Compute CDM,
get_compute_node_by_hostname method returns empty.
This patch set fixes to return a compute node even if hostname
is different from uuid.
Change-Id: I6cbc0be1a79cc238f480caed9adb8dc31256754a
Closes-Bug: #1746162
This patch remove redundant import aliases and add pep8 hacking function
to check no redundant import aliases.
Co-Authored-By: Dao Cong Tien <tiendc@vn.fujitsu.com>
Change-Id: I3207cb9f0eb4b4a029b7e822b9c59cf48d1e0f9d
Closes-Bug: #1745527
This patch set adds validation of baremetal model.
It also fixes PEP issues with storage capacity balance
strategy.
Change-Id: I53e37d91fa6c65f7c3d290747169007809100304
Depends-On: I177b443648301eb50da0da63271ecbfd9408bd4f
This patch set adds /state resource to strategy API
which allows to retrieve strategy requirements.
Partially-Implements: blueprint check-strategy-requirements
Change-Id: I177b443648301eb50da0da63271ecbfd9408bd4f
This patch set excludes nodes and instances from auditing
if appropriate metrics aren't available.
Change-Id: I87c6c249e3962f45d082f92d7e6e0be04e101799
Closes-Bug: #1736982
This patch set:
1. Removes nova, ceilometer and gnocchi properties.
2. Adds using of datasource_backend properties along with
statistic_aggregation method.
3. Changes type of datasource config.
Change-Id: I09d2dce00378f0ee5381d7c85006752aea6975d2
Partially-Implements: blueprint watcher-multi-datasource
This patch adds Storage Capacity Balance Strategy to balance the
storage capacity through volume migration.
Change-Id: I52ea7ce00deb609a2f668db330f1fbc1c9932613
Implements: blueprint storage-workload-balance
This patch set:
1. Removes nova, ceilometer and gnocchi properties.
2. Adds using of datasource_backend properties along with
statistic_aggregation method.
3. Changes type of datasource config.
Change-Id: I4a2f05772248fddd97a41e27be4094eb59ee0bdb
Partially-Implements: blueprint watcher-multi-datasource
If it uses a specific value(usually 1 or 2) as the min_sd when starting
to simulate migrations. The first simulate_migration case will always be
less than the min_sd and come into the solution, even though the migration
will increase the weighted sd. This is unreasonable, and make migrations
among hosts back and forth
Change-Id: I7813c4c92c380c489c349444b85187c5611d9c92
Closes-Bug: #1739723
This patch set adds data migration version that fills noname audits
with name like strategy.name + '-' + audit.created_at.
Closes-Bug: #1738758
Change-Id: I1d65b3110166e9f64ce5b80a34672d24d629807d
when we update a existing audit's interval, the interval of
'execute_audit' job is still the old value.
We need to update the interval of 'execute_audit' job.
Change-Id: I402efaa6b2fd3a454717c3df9746c827927ffa91
Closes-Bug: #1738140
Now that CDM-scoping was implemented, Getting scoped storage model
have to be updated.
This patch updates getting storage cluster data model.
Change-Id: Iefc22b54995aa8d2f3a7b3698575f6eb800d4289
This commit registers all policies formally kept in policy.json as
defaults in code. This is an effort to make policy management easier
for operators. More information on this initiative can be found
below:
https://governance.openstack.org/tc/goals/queens/policy-in-code.html
bp policy-and-docs-in-code
Change-Id: Ibab08f8e1c95b86e08737c67a39c293566dbabc7
In one integrated cloud env, there would be many solutions, which would
make the compute resource strongly relocated. Watcher should listen to
all the notifications which represent the compute resource changes, to
update compute CDM. If not, the compute CDM will be stale, Watcher
couldn't work steadily and harmoniously.
Change-Id: I793131dd8f24f1ac5f5a6a070bb4fe7980c8dfb2
Implements:blueprint listen-all-necessary-notifications
Migration action 'cold' is not intuitive for the developers and users,
so this patch replaces it with ‘migrate’ and 'retype'.
Change-Id: I58acac741499f47e79630a6031d44088681e038a
Closes-Bug: #1733247
Migration action with migration_type 'cold' does not work.
This patch fixes nova_helper to follow Pike release python-novaclient.
Change-Id: I0637cefad68cd4b7077a5d3e6b5db5562e7f2917
Closes-Bug: #1736624
This patch set follows deprecation steps
and changes default datasource to gnocchi
Closes-Bug: #1735180
Change-Id: I3b7ec47af7707359373a424a6469d28cbf9ce177
This patch removes 'audit_scope' from __init__ of
BaseClusterDataModelCollector class, as it is a singleton class
and cannot be instantiate more than once.
A new method is defined in BaseClusterDataModelCollector in place
of property audit_scope_handler, which takes audit_scope as
argument.
Change-Id: I0664c151d71a711c118d43c180d8b0760b1c81fa
Closes-Bug: #1732849
In one integrated cloud env, there would be many solutions, which would
make the compute resource strongly relocated. Watcher should listen to
all the notifications which represent the compute resource changes, to
update compute CDM. If not, the compute CDM will be stale, Watcher
couldn't work steadily and harmoniously.
Change-Id: I57173f0cce0717aa36c5ff758d972d38013e3ef8
Implements:blueprint listen-all-necessary-notifications
This patch adds notfication changes after changing the type
of global_efficacy in ActionPlan objects.
Change-Id: I1116cf1d19e4b19e19789f943a631d0400871766
Partially-Implements: blueprint multiple-global-efficacy-indicator
Storage pool can have many volume types,
'get_volume_type_by_backendname' should return a list of types.
Closes-Bug: #1733257
Change-Id: I877d5886259e482089ed0f9944d97bb99f375824
The app.wsgi file should be adjusted to meet pep8 standards and
apply pep8 check to app.wsgi automatically.
Change-Id: I34ec9ce56a329ede76d3d69f6e973d66350d85d1
This patch does step 1 in the docs: Move Legacy Jobs to Projects.
Partial-Implements: blueprint migrate-to-zuulv3
Change-Id: I0c0713260c2dfa2ba64a5746c342db01c1bc3d75
The os-testr 1.0.0 release had a couple of required config changes due
to it's internal usage of stestr. This commit fixes those by adding a
.stestr.conf (to remove a warning) sets the fixture env variables in
the tox.ini instead of being hidden in .testr.conf and removing the
quotes around posargs to make passing args to ostestr actually work.
Also to keep the expected dev experience stestr is used directly for
places where ostestr was called directly.
Change-Id: I785b0dc1720328b119f0978aa573ad0b89e54c75
Closes-Bug: #1731155
This patch update the global efficacy for server consolidation
strategy, test cases and general format.
Change-Id: I62af1e4be415998669f938b3d587c1ccf4293419
Implements: blueprint multiple-global-efficacy-indicator
According to "code conventions" [1], do not use "-y" option.
Instead, use apt-get install package, yum install package,
or zypper install package.
[1] https://docs.openstack.org/doc-contrib-guide/writing-style/
code-conventions.html
Change-Id: I174c8a0c653e322bbf951d72445b1a43251baa7e
If server migration fails, the while loop will not break
until retry to 0,
we can check the task_state to avoid this situation.
Closes-Bug: #1728476
Change-Id: I07e1048eb736263a261456ee78c96fee9db13cb5
Use https instead of http to ensure the safety without containing our
account/password information
Change-Id: I2e4f120f1d57c6c666d65a9a279eddefd112bf23
If scheduler uses db job store, the self.scheduler is None
beacuse execute_audit is class method.
We get [] from self.scheduler.get_jobs()
and can't remove the cancelled audit.
Change-Id: I1768c847465bbefbc9794281328edb07fea83b03
Closes-Bug: #1726345
This patch updates document and add release note for cdm-scoping.
Change-Id: I8e9099aa5caf6bf17ea912e7b1a600af3b6d519d
Partially-Implements: blueprint cdm-scoping
Use https instead of http to ensure the safety without containing our
account/password information
Change-Id: I16bdf6067fba4ea841071100d68edae7750946de
1. Update the instance migration link
2. remove the unnecessary install-guide link
configure and install Ceilometer by
https://docs.openstack.org/ceilometer/latest is enough
remove the other link.
Change-Id: I2bf408de1023750a3d1f2c9e25293649d99ac428
There are more than one 'migrating' status in the volume migration.
Others include starting, completing and so on.
So we should check the final status 'success' and 'error'.
Change-Id: I8ee9330aa32ec4516f2bf9e046ea68a72bc8a53d
https://github.com/openstack/python-novaclient/blob/
master/novaclient/v2/servers.py#L468
From nova api version 2.25(Mitaka release), the default value of
block_migration is None which is mapped to 'auto'.
So we can remove 'block_migration' from live_migrate_instance.
Change-Id: I75a581d3146824b1146e2acf5a7fbe4f8368a4e8
There is already define version_string in version.py
So we don't need generate version in other files, just call it.
Change-Id: I7d8294860523eedad92e213ad00569829e120c39
As of Nova API microversion 2.47, response of GET /servers/detail has flavor
which contains a subset of the actual flavor information used to create the
server instance, represented as a nested dictionary.
Since current watcher nova default api version is 2.53(Pike), This patch
follows the API response change.
Change-Id: Ia575950f0702afa1d093f03ca8ddedd3c410b7de
Closes-Bug: #1722462
This patch removes, explicitly enabled neutron from local.conf
as devstack now default uses neutron
Change-Id: Icf6bd944dd2262ff23cbcceb762a9ba80f471dbb
This patch fixes TypeError of range() in function chunkify, as
range() integer step argument expected, but got str.
Change-Id: I2acde859e014baa4c4c59caa6f4ea938c7c4c3bf
Since default workflow engine action container do_execute method
does not raise exception when action failed, workflow engine action
container execute method never raise exception and
action plan state becomes always SUCCEEDED.
This patch fixes default workflow engine action container do_execute
method to raise exception when action does not return True.
Change-Id: I7eeef69dbdfb5d40e3cf0b1004cbfe199a16bf7b
Closes-Bug: #1719793
Since packages for openSUSE and SLES are not provided,
this patch removes installation guide for openSUSE and SLES.
Change-Id: Ic15d8c4b262e935c7acaef41e18960d0b259d5c9
Closes-Bug: #1715032
This patch adds Notifications for cancel action plan
operation.
Change-Id: I5a89a80729349e3db43ca35ff9fbe8579e86b3b1
Implements: blueprint notifications-actionplan-cancel
disk_over_commit flag was removed in Nova API microversion 2.25(Mitaka).
Since current watcher nova default api version is 2.53(Pike),
this patch removes disk_over_commit flag.
Change-Id: Ib141505b9e8cb41997b29c1762e387b1f84f5143
Closes-Bug: #1720054
As of now we can not know what was happend when migrate action
failed critically.
This patch adds exception log when migrate action failed critically.
Change-Id: I54d0bc54ee1df6f13754771775c58255f53f5008
At various places in watcher code, we are using property getters
to set property, in this way the property setters defined are
never used, this patch fixes to use property setters to set
property.
Change-Id: Idb274887f383523cea39277b166ec9b46ebcda85
python-subunit is not used directly anywhere
and it is dependency of both testrepository
and os-testr
(probably was used by some tox wrapper script before)
Change-Id: I89279430554bc522817c4e2685afef0d95c641dd
add 'disabled_reason' filed into 'ComputeNode' resource, to distinguish
which nodes are disabled by Watcher and which are not by Watcher.
Implements:blueprint extend-node-status
Change-Id: I7175f14870834a4582e45309529d7e8d9fbb2e6f
This patch set replaces create_schema with upgrade to fix
apscheduler creation issue. It also fixes pep8 warnings to
d09a5945e4a0_add_action_description_table.py
Change-Id: Ica842d585ee3a9cd67e45eb1d7bb1916573d7c9c
The return value of ironic_client.node.set_power_state is None, so it's
useless to return the result.
We should check the node state until it's changed or timeout.
Change-Id: I31f75a2c4a721ce4481e6ae7fb83d154a443dad9
Closes-Bug: #1713655
This Patch fixes gate failure, encountered in recent version
of oslo_messaging.
Change-Id: I6d8ab882a7c157dcf4f78c805a4ce2d9b1fa3f14
Closes-Bug: #1716476
iso8601.UTC is correct datetime UTC field object.
iso8601 >= 0.1.12 includes only iso8601.UTC for python3
while both UTC and Utc() for python2. Less then 0.1.12
included both UTC and Utc() for both python2/3.
Change-Id: I0f8796fba6725eea013b3f8d9ad33c10a402c524
Closes-Bug: #1715486
AverageCpuLoad and MigrationEfficacy efficacy indicators are not used.
This patch removes unused indicators.
Change-Id: I2b21defd442c135d26f8fd45f6faf9f67c770bde
Gnocchiclient uses keystoneauth1.adapter so that adapter_options
need to be given.
This patch fixes gnocchiclient creation.
Change-Id: I6b5d8ee775929f4b3fd30be3321b378d19085547
Closes-Bug: #1714871
Audit scope JSON schema should restrict key of host_aggregates
to "id" or "name", but that is not working now.
This patch fixes DEFAULT_SCHEMA to validate host_aggregates.
Change-Id: Iea42da41d61435780e247736599a56c026f47914
Closes-Bug: #1714448
Diffrent stratege has diffrent default scope, restrict them to their
default scope will avoid usage problems.
1)workload_balancing/thermal_optimization/airflow_optimization goals
react on enabled nodes, so restrict default scope to compute nodes
with up state and enabled status.
2)server_consolidation goal react on enabled or disabled nodes, So
restrict default scope to compute nodes with up state and
enabled/disabled status.
Change-Id: I7437dee699ee2d3dd227a047196d4d8db811b81e
Closes-Bug: #1714002
Services are now identified by uuid instead of database id to ensure
uniqueness across cells.
GET /os-services returns a uuid in the id field of the response
from API microversion 2.53(maximum in Pike)
This patch set updates default Nova API version to 2.53.
Change-Id: Ib9fefb794eda3c9e75c6a2f5cfdb0e682b8955f3
Closes-Bug: #1709544
Since iter(dict) is equivalent to iter(dict.keys()), it is unnecessary
to call the keys() method of a dict, the dictionary itself is enough
to be referenced. The shorter form is also considered to be more
Pythonic.
This patch removes the unnecessary dict.keys() method calls in api.
This is a part of a larger patch series that removes dict.keys()
method calls.
TrivialFix
Change-Id: I29000f1f05b90d70109fa01393e97e1ebf450c63
During the strategy sync process,
if goal_id can't be found in the goals table,
will throw a KeyError exception.
Change-Id: I62800ac5c69f4f5c7820908f2e777094a51a5541
Closes-Bug: #1711086
Only when True is returned, the action state is set to SUCCEEDED
some actions(such as migrate) will return None if exception raised
Change-Id: I52e7a1ffb68f54594f2b00d9843e8e0a4c985667
By the input parameter "metrics", it makes decision to migrate a VM
base on cpu or memory utilization.
Change-Id: I35cce3495c8dacad64ea6c6ee71082a85e9e0a83
outlet_temperature strategy relies on a datasource config parameter,
which can be either "ceilometer" or "gnocchi". This patch overrides
get_config_opts method of base class to allow specify datasource.
Change-Id: I551401039e26816568a04c7f2151d5b3c7ed269a
Closes-Bug: #1709024
List comprehensions and generator expressions are considered to be more
Pythonic (and usually more readable) than map and filter with lambda.
This patch replaces four usages of [map|filter](lambda ...) with the
appropriate list comprehension or generator expression.
TrivialFix
Change-Id: Ifda9030bb8aa196cb7a5977a57ef46dfefd70fa6
Metric 'memory.usage' depends on balloon driver,
and shows the memory usage inside guest-os,
while 'memory.resident' represents volume of RAM
used by the instance on the physical machine
So, it is more reasonable to use 'memory.resident'
when calculating node utilization
by gathering instances utilization of the node.
Change-Id: I12dff5176bcf6cb103aa44cafd54f9ecd7170864
Watcher fails to load storage plugin in case there is no installed
Cinder in OpenStack services.
This patch set adds collector_plugins parameter under collector
section in watcher.conf. If plugin name is in collector_plugins,
The plugin is loaded.
Change-Id: Ie3c3543216c925d49b772bf5fe3773ca7d5ae437
Closes-Bug: #1707603
compute_model.get_node_by_uuid(node_uuid) will raise a exception
when the compute node isn't in the compute model.
Closes-Bug: #1709004
Change-Id: I667a9dbfcf67f9f895976aadd5300bbea2ffe6f0
monascaclient was recently migrated to
use 'osc' library. Due to that, exception
classes has been changed. This commit adjusts
the exception class to recently released
monascaclient==1.7.0
Depends-On: Ie647221a3ab055e7b150d65ffb9287b44ef834cb
Change-Id: Icfef345c4269ac4cb682049f22a43fdab3d39845
This patch set updates gnocchi repository URL in local.conf.controller
bacause it moved from under openstack to their own repository.
Change-Id: I53c6efcb40b26f83bc1867564b9067ae5f50938d
Currently compute CDM excludes disabled compute node.
This patch set fixes compute CDM to include disabled compute node.
Change-Id: I8236bb73e0d9bb242251c2abfb59ad5693087afa
Closes-Bug: #1685787
This patch updates the state machine diagram for action plan, It
includes new state "cancelling" which is introduced by actionplan
cancel operation
Change-Id: I0af59f2164922c56d59fbad8018e2aecfef97098
In this patch feching resource_id by resource's original_id was added to
statistic_aggregation method.
Closes-Bug: #1707653
Change-Id: I70b9346146f810e2236ccdb31de4c3fedf200568
This strategy now allow us to create action plans with an explicit
set of actions.
Co-Authored-By: Mikhail Kizilov <kizilov.mikhail@gmail.com>
Change-Id: I7b04b9936ce5f3b5b38f319da7f8737e0f3eea88
Closes-Bug: #1659243
1. Update python version from 3.4 to 3.5
2. Update Ubuntu version from 14.04 to 16.04
3. Update Fedora version from 19+ to 24+
Change-Id: Ic5e9bbd126e10697300c6ffd51ff55d0b815d5ca
Currently continuous audit fails once it fails
because continous audit tries to remove job
even if job is not exists.
This patch set fixes it.
Change-Id: Ic461408c97d71e14c57e368f8436b26fe355fa4e
Closes-Bug: #1706857
This patch fixes the hardcoded value of availability zone
in nova-helper, Now nova api is used to get the availability zone
of destination node
Change-Id: I4c5a34946ed404df5bbfe34ce99873d32772dbf4
Add a new table to save the mapping
Add logic to update the table when action loading
Add logic to show the action description
Change-Id: Ia008a8715bcc666ab0fefe444ef612394c775e91
Implements: blueprint dynamic-action-description
Cinder data model was added in Pike cycle and that needs
configuration in cinder.conf for refreshing the model in
real time.
This patch set adds Add Configure Cinder Notifications section
for explaining the configuration.
Change-Id: I41cc870e2d47c56fd7c9fcdd6f03c95fa939c3f2
Now that we replaced voluptuous with JSONSchema in all actions,
this patch replaces voluptuous with JSONSchema in BaseAction
and removes validate_parameters method in each action.
Partially Implements: blueprint jsonschema-validation
Change-Id: I07c907ddfa4a568d7fff42776df02218330d56a0
This patch set fixes devstack plugin by removing head
keyword from watcher-db-manage.
It is related to https://review.openstack.org/#/c/483825/
Change-Id: I3dcea6ae799c94a882e68d66920c5cd87d83d85e
This patch removes the requirement to specify destination node,
for migrate action. There can be some strategies that wants to rely
on nova-scheduler for destination node. It will not impact any current
strategy behavior.
Change-Id: Ia12b2f0b8369e632a7959b28e485d86b6cff83e9
Closes-Bug: #1691056
There are many uselees logging in the decision log file:
Arguments dropped when creating context: {u'global_request_id': None}
Change-Id: I7583b2ff6ea1cc15173536ca1cf08cc9f0ecb20d
Closes-Bug: #1704736
This patch adds abort operation for live migration
to support abort in cancel action plan.
Change-Id: I458e93d9bd09dc4cf80cc941104129fc7600a6b1
Partially-Implements: blueprint cancel-action-plan
wrong parameter passing of function "query_sample"
the default parameter will be recognized as "user_id"
Change-Id: I293b130c3f709dc93cd4b1b7382ae8895a54765d
Closes-Bug: #1703297
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the change_node_power_state action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: If9ffe5e0b107e0da5673247e4d5ec9917790827f
This patch set updates documentation link in README
because of doc migrations.
Depends-On: Iecb4f60efb015a56b9b37331859848b287112842
Change-Id: I5b8a5ec8a328ae275fe6271965ea0077ae301814
This patch set makes the following changes:
* Add index file to each subdirectory of doc/source
* Update doc/source/index.rst with new links
* Move content of install-guide to the doc/source/install
* Minor changes
Depends-On: Ifc5512c0e2373cf3387e0e0498268eab092e52bb
Change-Id: Iecb4f60efb015a56b9b37331859848b287112842
The default neutron endpoint type is publicURL in neutronclient.
This patch replaces default neutron endpoint type from
internalURL to publicURL.
Change-Id: I8162b6178051df5f086488ecfb5d0bdc569ef9cd
Related-Bug: #1686298
The default glance endpoint type is publicURL in glanceclient.
This patch replaces default glance endpoint type from
internalURL to publicURL.
Change-Id: I39451ba89f191693475a694bd5c9045bf3bf539a
Related-Bug: #1686298
There is no action plan if creating audit was cancelled.
In this case, we just return none.
Change-Id: Ia7a93dab5978d181557d7dd7499e83655aec6f40
Closes-Bug: #1700901
In tempest, alias 'manager' has been moved to 'os_primary'
in version Pike, and it will be removed in version Queens.
This patch is to replace the usage of 'manager' with 'os_primary'.
For other details, please check [1] and [2]
[1] https://review.openstack.org/#/c/468036/
[2] https://review.openstack.org/#/c/463484/
Change-Id: I582758c42f61a85a2fd4aac5f7a97cd6021ecf68
When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.
This patch fix this problem. Refer patch set [1]
[1] https://review.openstack.org/#/c/189569
Change-Id: I6c0b896a6de1b7193dd4b77b6bc4433d0c75732d
Some of the available checks are disabled by default, like:
[H106] Don’t put vim configuration in source files
[H203] Use assertIs(Not)None to check for None
Change-Id: I369cff1c0f7f3cd3f5bcf3785b6904c9326c6759
When using 'query.filter_by(deleted_at=None)'
Will generate the incorrect SQL 'audits.deleted_at', as below:
SELECT * FROM action_plans INNER JOIN audits ON audits.id =
action_plans.audit_id WHERE audits.uuid = '' AND
audits.deleted_at IS NULL ORDER BY action_plans.id ASC
The correct filter field is 'action_plans.deleted_at'
Change-Id: Ie05f35233f78e82bc7af2c26c9effd62ea5f86ab
Closes-Bug: #1698720
The primary usage of "node.state" is wrong, it should be 'node.status'.
So correct it and refactor the method 'get_state_str'.
Change-Id: I9004c85bfb09f9b41fc56ecb5a56122d523a661f
Added strategy to identify and migrate a Noisy Neighbor - a low priority VM
that negatively affects peformance of a high priority VM by over utilizing
Last Level Cache.
Partially Implements: blueprint noisy-neighbor-strategy
Change-Id: Ia13f7e91a695024410b8f3b3d3d1646a0dd687d4
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the migration action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I02bff5db9bd06567bcc33b61a316c42c805bb20e
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the change_nova_service_state action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I09a03fff96d9555024a74ba255c6951affc39de8
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use JSONSchema
as our only JSON validation tool to keep consistence and also
to make it easier to expose the validation schema through
our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the resize action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I0ee4ba010a9f437658af81d5c971449aefc7f9c4
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the sleep action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I3032490236536a11e7045a56ad0bd40ef979407e
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the nop action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: Idf42b3359c36ac9480bd1f1bdd31e756214628ef
DeprecationWarning:
Using function/method 'oslo_messaging.transport.get_transport()'
is deprecated: use get_rpc_transport or get_notification_transport.
Replace get_transport with get_rpc_transport.
Change-Id: I13455842235a16463e61e5500c9e250a5cc9f86e
This patch adds feature to cancel action plan in watcher.
A General flow from watcher-api to watcher-applier is implemented.
action plan cancel can cancel any [ongoing, pending, recommended]
action plan, it will update the action states also to "cancelled".
For ongoing actions in action plan, actions needs to be aborted.
Seperate patches will be added to support abort operation
in each action.
Notification part is addressed by a seperate blueprint.
https://blueprints.launchpad.net/watcher/+spec/notifications-actionplan-cancel
Change-Id: I895a5eaca5239d5657702c8d1875b9ece21682dc
Partially-Implements: blueprint cancel-action-plan
Fix the following WARNING:
WARNING:test command found but not installed in testenv
cmd: /bin/rm
env: /home/jenkins/workspace/gate-watcher-python27-ubuntu-xenial/.tox/py27
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
Change-Id: Ie091bd64b6a87c30535ada34daf9d594aa3fdd41
Oslo.config deprecated parameter enforce_type and change
its default value to True. Remove the usage of it to avoid
DeprecationWarning: "Using the 'enforce_type' argument is
deprecated in version '4.0' and will be removed in version
'5.0': The argument enforce_type has changed its default
value to True and then will be removed completely."
Change-Id: I59621664773ee5ad264e6da9b15231f30dbb9c40
Closes-Bug: #1694616
The default cinder endpoint type is publicURL in cinderclient.
This patch replaces default cinder endpoint type from
internalURL to publicURL.
Change-Id: Ie6951086e4656bd83195dab151dbaaf948113a7c
Related-Bug: #1686298
Add action get_description in class BaseAction.
This information will be sent to the API side via notification.
Partially Implements: blueprint dynamic-action-description
Change-Id: I9ce1b18ad8c5eb7db62ec926d1859d0f508074b0
Audit record was deleted but the field 'state' maybe not set DELETED.
get_audit_list's filter used field 'state' will get
wrong result.
filter rule should use field 'deleted_at' instead of field 'state'.
get_action_list and get_action_plan_list have the same filter rule.
Change-Id: I08b2a005ca5fb7c2741ac5ed97c6e6b4279758ed
Closes-Bug: #1693666
1.TypeError: watcher_non_live_migrate_instance() got an unexpected keyword
argument 'dest_hostname'
2.Conflict: Cannot 'stop' instance ****** while it is in vm_state stopped
(HTTP 409)
Closes-Bug: #1693434
Change-Id: I7293dd5d08e33c2e534d072da8592172bc438c9e
Block Storage API v2 is deprecated now. Instead of it, we should use v3
in default (api_version).
This patch replace cinder default api_version from '2' to '3' in
conf/cinder_client.py
Change-Id: I53ffa74cdac7ac31c74937bf18da8ed2fec92223
Closes-Bug: #1691104
[1] moves the attr decorator from test.py to tempest/lib. So, all the
references to tempest.test has to be moved to tempest.lib.decorator.
[2] https://review.openstack.org/#/c/456236/
Change-Id: If977e559d9f3b982baf2974efef3c5b375f263b9
This replaces the deprecated (in python 3.2) unittest.TestCase
method assertRaisesRegexp() with assertRaisesRegex().
Change-Id: I38c3055288034aba51c11bb1bccd3655f760cecc
Closes-Bug: #1436957
for now / just get int value ,this will lead method
filter_destination_hosts return 0 host
because self.threshold / 100 * host.vcpus is 0
so we need use accurate division to change this
close-bug: 1689269
Change-Id: I5663951ce750d6c4580a507ccfc0268baea0685f
This patch set adds WatcherJobStore class that allows to link
jobs and services.
Partially-Implements: blueprint background-jobs-ha
Change-Id: I575887ca6dae60b3b7709e6d2e2b256e09a3d824
Currently if user wants to skip some host_aggregates from audit,
it is not possible. This patch adds host_aggregates into the exclude
rule of audit scope. This patch also implements audit-tag-vm-metadata
using scopes.
TODOs:
1. Add tests
2. Remove old implementation of audit-tag-vm-metadata
Change-Id: Ie86378cb02145a660bbf446eedb29dc311fa29d7
Implements: BP audit-tag-vm-metadata
The default is publicURL in novaclient.
This also caused the failure of
gate-watcher-dsvm-multinode-ubuntu-xenial-nv.
Change-Id: I485dd62fb7199ffeca29a9b573a624bf144484d1
Closes-Bug: #1686298
Closes-Bug: #1686281
Running py2* post py3* tests results in error. Add
'rm -f .testrepository/times.dbm' command in testenv to
resolve this.
Change-Id: Ia43f8d10f157d988c4d2c89f16cac0ea729cabe6
now when action like live-migration retry is reached
result is False but action still success
we can add check in do execute method can fix this problem.
close-bug: 1685757
Change-Id: I8390566ec8dcfa3a71b931d5be1b305802ac0b2a
If instance_ram_util and instance_disk_util can not get data
from datasource. we can use instance data,
just like total_cpu_utilization
Change-Id: I4170b96946b07435411ada5ff4a14c978c0435b4
oslo.messaging allow dispatcher to restrict endpoint
methods with DefaultRPCAccessPolicy to fix FutureWarning
Closes-Bug:1663543
Change-Id: I0288320193b0153ee223664696abca21cbdb0349
This patch set adds ironicclient.
Change-Id: I122a26465d339ee6e36c0f234d3fd6c57cee2afa
Partially-Implements: blueprint build-baremetal-data-model-in-watcher
New audit state SUSPENDED is added in this patch set. If audit
state with continuous mode is changed from ONGOING to SUSPENDED,
audit's job is removed and the audit is not executed.
If audit state changed from SUSPENDED to ONGOING in reverse,
audit is executed again periodically.
Change-Id: I32257f56a40c0352a7c24f3fb80ad95ec28dc614
Implements: blueprint suspended-audit-state
This patch adds gnocchi support in uniform_airflow strategy
and adds unit tests corresponding to that change.
Change-Id: I347d0129da94a2fc88229d09297765795c5eeb1a
Partiallly-Implements: bp gnocchi-watcher
In DefaultApplier.execute(), Action.list should set eager=True.
Otherwise it will trigger an exception when the notification is sent.
This also causes the FAILURE of
gate-watcher-dsvm-multinode-ubuntu-xenial-nv.
Change-Id: I27db9691727671abb582d4f22283ebda5bd51b07
Closes-Bug: #1676308
Use https instead of http to ensure the safety without containing our
account/password information
Change-Id: I1f164848f164d9694c0cfc802cc3980459bdf12f
This patch adds gnocchi support in outlet_temp_control strategy
and adds unit tests corresponding to that change.
Partiallly-Implements: bp gnocchi-watcher
Change-Id: I2c2e9a86c470f3402adc3dbb7eb9995c643d5b37
This patch adds gnocchi support in workload_balance strategy
and adds unit tests corresponding to that change.
Change-Id: I9bc56c7b91b5c3fd0cfe97d75c3bace50ab22532
Partiallly-Implements: bp gnocchi-watcher
This patch enables gnocchi plugin for devstack.
Partiallly-Implements: bp gnocchi-watcher
Closes-Bug: 1662515
Change-Id: I6614ce6999c9681bd6fafc6c85a3755b5ce8e2dd
This patch adds support to run watcher-api
with mod-wsgi. It provides
1. wsgi app script files, to run watcher-api under apache.
2. updated devstack plugin to run watcher-api default with
mod-wsgi.
3. Document to deploy watcher-api behind wsgi.
Change-Id: I8f1026f0b09fd774376220c2d117ee66f72421b8
Closes-Bug: #1675376
In this changeset, I added a tempest test that is in charge of
executing the workload_stabilization strategy.
Change-Id: I61bad268fc5895ddb22312baeb21da5ae3c71de9
This patch adds gnocchi support in VM-Workload-Consolidation strategy
and adds unit tests corresponding to that change.
Change-Id: I4aab158a6b7c92cb9fe8979bb8bd6338c4686b11
Partiallly-Implements: bp gnocchi-watcher
This commit[1] has removed verbose option due to which our ut's are failing.
This patch is fixing this issue.
[1] https://review.openstack.org/#/c/444217/
Change-Id: I784a7f855f42de462e8fc8f829f5526e1483dab4
Some configuration options were accepting both IP addresses
and hostnames. Since there was no specific OSLO opt type to
support this, we were using ``StrOpt``. The change [1] that
added support for ``HostAddressOpt`` type was merged in Ocata
and became available for use with oslo version 3.22.
This patch changes the opt type of configuration options to use
this more relevant opt type - HostAddressOpt.
[1] I77bdb64b7e6e56ce761d76696bc4448a9bd325eb
Change-Id: Idec43189ff8edc539027ba0b0369e54ae883cd2b
This patch adds gnocchi support in workload_stabilization strategy
and adds unit tests corresponding to that change.
Change-Id: I96bd758962bbf67d60e19a99a19451fb80e447b2
Partiallly-Implements: bp gnocchi-watcher
This patch adds the functionality to filter out VMs which have
metadata field 'optimize' set to False. This patch implements the
functionality for basic_consolidation strategy.
Change-Id: Iaf7b63e09534e4a67406e7f092242558b78c0bde
Partially-Implements: BP audit-tag-vm-metadata
This patch adds gnocchi support in basic_consolidation strategy
and adds unit tests corresponding to that change.
Change-Id: Ia1ee55fca8eadffbd244c0247577805b6856369d
Partiallly-Implements: bp gnocchi-watcher
We have two controllers as HA in our OpenStack environment.
There are watcher-applier and watcher-decision-engine in each
controller. So there are two same name in the services table.
In this case, the objects.Service.get_by_name(context, name)
will trigger exception of MultipleResultsFound.
We should use objects.Service.get(context, id) replace of
objects.Service.get_by_name(context, name).
Change-Id: Ic3ce784590d6c2a648cb3b28299744deed281332
Closes-Bug: #1674196
Those are remnants from the oslo-incubator times. Also, oslo.messaging
deprecated [1] transport aliases since 5.2.0+ that is the minimal
version supported for stable/newton. The patch that bumped the minimal
version for Watcher landed 3 months+ ago, so we can proceed ripping
those aliases from the code base.
[1] I314cefa5fb1803fa7e21e3e34300e5ced31bba89
Change-Id: Ie3008cc54b0eb3d1d02f55f388bd1c3b109d126d
Closes-Bug: #1424728
Check the creation time of the actionplan,
and set the state to SUPERSEDED if it has expired.
Change-Id: I900e8dc5011dec4cffd58913b9c5083a6131d70d
Implements: blueprint stale-action-plan
The scenario tests base class from Tempest is not a stable interface
and it's going to be refactored on Tempest side, as notified in
http://lists.openstack.org/pipermail/openstack-dev/2017-February/112938.html
Maintain a local copy of the base class, taken from Tempest with head of
master at c5f1064759fe6c75a4bc5dc251ed1661845936cb.
Change-Id: Idfa5ebe18c794c51e406156fb120d128478d4f1e
actionplan.save() will send a send_update notification.
This notification need eagerly loaded,
so we should set eager=True for actionplan.list.
Change-Id: Iafe35b9782fb0cc52ba5121c155f62c61ef70e1f
Closes-Bug: #1673679
Currently http is used to access git, however http is not safe enough
since the access information contains our account/password information.
Use https instead of http to ensure the safety.
And for others services such as pypi or python, it's still nice to use
https.
Change-Id: I706a4a1873c6bbc05385057757fc5962344f9371
I extracted some of the functionalities into helper functions
to reduce the length and complexity of build_query (in
watcher/datasource/ceilometer.py).
Additionally it became more readable as well, without
changing its functionality.
Change-Id: I9e5c524754cf0f9d718a216465ba1b7536add80e
This patch adds instance metadata in the cluster data model. This
is needed for Noisy Neighbor strategy.
Change-Id: Ia92a9f97ba1457ba844cc37a4d443ca4354069e3
Since pbr already landed and the old version of hacking seems not
work very well with pbr>=2, we should update it to match global
requirement.
Partial-Bug: #1668848
Change-Id: I5de155e6ff255f4ae65deff991cff754f5777a8d
There are not any endpoints for `ceilometer` with devstack/
local.conf.controller. The service `ceilometer-api` should
be enabled explicitly.
Change-Id: I2218a98182001bef65fbc17ae305cfadf341930e
Closes-Bug: #1667678
In the current audit process, after executing the strategy,
will check whether there are currently running actionplan,
and if so, will set the new actionplan SUPERSEDED.
We can optimize the process to perform this check in pre_execute(),
and if any actionplan is running, no further processing is performed.
Change-Id: I7377b53a2374b1dc177d256a0f800a86b1a2a16b
Closes-Bug: #1663150
Since watcher dashboard can be sucessfully installed now by devstack,
we should enable this again. Many of us are get the local.conf from
here,so this change is necessary, we can enable watch dashboard plugin
by default.
Change-Id: Iad5081a97515b3f831d2f468dc514a942e6d3420
test.call_until_true has been deprecated since Newton on Tempest side,
and now Tempest provides test_utils.call_until_true as the stable
library method. So this patch switches to use the stable method before
removing old test.call_until_true on Tempest side.
Change-Id: Iba2130aca93c8e6bccb4f8ed169424c791ebc127
Needed-by: Ide11a7434a4714e5d2211af1803333535f557370
Improves tracking independent workflows. A log adapter
was added to provide an easy means of prepending publisher
ID and event type information.
Change-Id: I5d2d8a369f99497b05c2a683989e656554d01b4f
Closes-Bug: 1642623
This patch adds checking audit state when updating an existing audit
in accordance with audit state machine.
Closes-Bug: #1662406
Change-Id: I20610c83169b77f141974a5cebe33818a4bf0728
The argument to the add_edge function should be instance.uuid
and node.uuid, not instance and node
Change-Id: Ida694f9158d3eb26e7f31062a18844472ea3c6fa
Closes-Bug: #1662810
The gating on python 3.4 is restricted to <= Mitaka. This is due to
the change from Ubuntu Trusty to Xenial, where only python3.5 is
available. There is no need to continue to keep these settings.
Change-Id: I3b3f0b08f6f27322b8a9d99eb25984ccd6bfe7a6
The 'strategy_id' attribute is the one that not exposes
to API. But the 'PATCH' API always update this attribute.
So this change does not work when choose 'remove' op.
This patch sets value for 'strategy_id' attribute.
Change-Id: I1597fb5d4985bb8271ad3cea7ea5f0adb7de65f4
Closes-Bug: #1662395
When action plan is currently running, new action plan is set as
SUPERSEDED and error log reported. This patch changes log level
from error to warning.
Change-Id: I931218843d8f09340bd5363256164807d514446b
Closes-Bug: #1662450
'watcher audit list' returns a incorrect auto trigger flag
auto_trigger field is incorrectly unset to False
Change-Id: Iba4a0bda1acf18cbfde6f1dcdb0985a4c3f7b5bb
Closes-Bug: #1662051
Alembic provides command script to update a relational
database, using SQLAlchemy. I provide in this patchset the
initial version python script for create a Ocata db version.
Implements: blueprint db-migration
Change-Id: I7bda4286bc8141bb4fcfba3837070966ef506b5d
This patch set adds new period strategy input parameter
which allows allows to specify the time length of
statistic aggregation.
Partial-Bug: #1614021
Change-Id: I1a276206e5b2c05d8f94acdeb866c8822fa84f35
The mocks on _get_alarm_client() are now failing because it was renamed
as _get_redirect_client(). This fix reflects this name change.
Change-Id: Id61e74dc4adc0c939661462953b322d65104a651
Here is a new architecture diagram with some updates on the
glossary and on descriptions of architecture elements.
I also fix some warning logs.
Closes-Bug: #1657405
Change-Id: I442082d702fc8667e9397c090da51ca1ead5d86e
This patch set allows to execute Watcher Actions in parallel.
New config option max_workers sets max number of threads
to work.
Implements: blueprint parallel-applier
Change-Id: Ie4f3ed7e75936b434d308aa875eaa49d49f0c613
In this changeset, I use https://review.openstack.org/#/c/362730/
as an example to make the existing ModelRoot fully graph-based.
Change-Id: I3a1ec8674b885d75221035459233722c18972f67
Implements: blueprint graph-based-cluster-model
If we define an audit scope with a exclusion rule composed
with compute nodes, the compute model built is empty. I fix
the algo used to built this scoped compute model.
Change-Id: I675226c58474b7ea52b3f92e7b466adae40643b8
Closes-bug: #1658995
An action plan can be now go to SUPERSEDED state if another
action plan is on going for instance. It could be the case with
tempest test, because we can have several scenario tests running
at the same time.
Change-Id: I5449e545676463f0ea601e67993e9dffa67929a7
Closes-bug: #1657031
As instance migration cost is petty compared to the cost of
compute node release, I update the way to compute the global
efficacy for a server consolidation goal. The new formula is simplest
and it's only based on compute node.
Change-Id: Ibcce31a85af70429f412c96c584a761d681366a2
In the optimize solution stage when launching audit
based vm_workload_consolidation strategy.
The instance map in node should change when migration
action was removed. Otherwise a KeyError exception
will be thrown.
Change-Id: I054b6b12922892a02d155b4ddc001b19890d32c5
Closes-Bug: #1656157
Add a script command to enable Nova compute.instance.update
notifications on VM and task state changes.
Change-Id: I639f003d92e184085a332f53c6783e5eca0002fe
Getting images from Novaclient is refered as proxy API
and was deprecated to use in nova. We should use
Glanceclient to get images instead.
Closes-bug: #1655516
Change-Id: Ie7d89e857d149e11b3c9b44c980b0be5cb0cc35f
In this changeset, I added the support for both Monasca and
Ceilometer for the basic_consolidation strategy.
Partially Implements: blueprint monasca-support
Change-Id: Ide98550fbf4a29954e46650190a05be1b8800317
In this changeset, I implemented a Helper class to deal with
Monasca requests.
Change-Id: I14cfab2c45451b8bb2ea5f1f48254b41fa5abae8
Partially-Implements: blueprint monasca-support
By calling solution.add_action with no input_parameters, TypeError:
'NoneType' object does not support item assignment was occurred.
This patch fix it.
Change-Id: Ia2ad0c18bc20468ca73c0ab70495fac2c90e0640
Closes-Bug: #1647927
This patch set adds support of auto-triggering of action plans.
Change-Id: I36b7dff8eab5f6ebb18f6f4e752cf4b263456293
Partially-Implements: blueprint automatic-triggering-audit
When we receive a incoming 'service.update' notification
from nova, with disabled: false, we should set the related
compute node status to ENABLED.
Change-Id: Ib8075a5cf786f81ec41423805433f83ae721cbbd
Closes-bug: #1650485
Implemented db config module
Partially Implements: blueprint centralise-config-opts. Also moved
basedir_def, bindir_def and state_path_def to watcher.conf.paths
Change-Id: I73d201f6a23bbdb1c6189434b11314a66620e85c
For multinode setup, a couple of updates to /etc/nova/nova.conf was
needed for live migration.
serial_console needed to be disabled and vncserver_listen needed to
be set to any for the vncserver to accept connections from all of the
compute nodes.
Change-Id: I62ebc2a07ca525bd80da130981f337806b2b89ae
Closes-Bug: #1649638
add_migration(self, instance_uuid, source_node,
destination_node, model)
param source_node: node object
param destination_node: node object
but in optimize_solution(), The incoming parameters are
source_node_uuid and destination_node_uuid.
This causes an exception:
AttributeError: 'unicode' object has no attribute 'state'
Change-Id: Ia27f219caa007f2b49ff9efc2544d5b4d894fe65
Closes-Bug: #1649441
Specific exception should be thrown when cluster state
is stale. Current usage is to raise this exception if
compute_model.state is True.
Bug was describeid by Jean-Emile DARTOIS.
Change-Id: Iaddb4cc8007c51bb14759c9da829751e834499d0
Closes-Bug: #1621855
At present, In the sync function, there is no check about the
parameters_spec field in strategy table, if the parameters_spec
field content has changed, Such as increased 'periods' parameter,
strategy table will not be updated, the program will run abnormal.
exception msg:
2016-12-05 11:11:39.138 TRACE watcher.decision_engine.audit.base
raise AttributeError(name)
2016-12-05 11:11:39.138 TRACE watcher.decision_engine.audit.base
AttributeError: periods
Change-Id: I84709c246acbdf44ccac257b07a74084962bb628
Closes-Bug: #1647521
The goal of "vm_consolidation" is not existent by default.
Use the goal of "server_consolidation" instead for shell command.
Change-Id: Icef5536a337fa88a4504e23e4de6d2e96c45d7b6
doc/source/deploy/configuration.rst
* 'The Watcher system is a collection of services
that provides support to optimize your IAAS plateform. '
Fix: Typo, plateform -> platform
* Additionally, the Bare Metal service has certain
external dependencies, which are very similar to
other OpenStack services:
Fix: Bare Metal -> Watcher
* Please check your hypervisor configuration to correctly
handle instance migration.
Fix: ref of ``instance migration`` is http://docs.openstack.org/
admin-guide/compute-live-migration-usage.html
Change-Id: I00ab282a0f9ffcfddf745df1dd67418a43d70b10
Closes-Bug: #1646290
There is not module 'messaging.utils' in 'watcher.common'
but module 'synchronization' instead.
Change-Id: If2d585a4f416614cbe91e4ef61fc7473508d38af
Closes-Bug: #1643862
One space was missed between 'the' and ':ref' in
'the:ref:Watcher Database <watcher_database_definition>'.
Change-Id: I3e46121dc7c30f73df4ca455e2c629929cdbd2ec
Closes-Bug: #1644388
This patch set adds new periods strategy input
parameter that allows to specify the time length of
statistic aggregation.
Change-Id: Id6c7900e7b909b0b325281c4038e07dc695847a1
In this change set, hardware.cpu_util is changed to
compute.node.cpu.percent in workload_stabilization.
By doing so, one can run this strategy on a simple devstack
without having to setup the SNMP plugin.
Change-Id: I8df8921337ea3f4e751c0c822d823e64e3ca7e1c
I this changeset, I fixed the issue whereby object auto fields are not
being updated within the WatcherObject after an update.
Change-Id: I7e65341b386a5c0c58c2109348e39e463cf2f668
Closes-Bug: #1641955
In this change set, Default value 'hardware.cpu_util' of instance_metrics
was changed to 'compute.node.cpu.percent'.
Change-Id: I02f87e5fea663e2e04c61cc36b7d55ff250bf8cc
In this changeset, I implemented the main logic although this is
mainly a shameful copy/paste of Nova's blueprint
https://blueprints.launchpad.net/nova/+spec/centralize-config-options
Partially Implements: blueprint centralise-config-opts
Change-Id: Ib645ad5da5c706336bb6ac37e85b027d05665c32
In this changeset, I implemented the following notifications:
- audit.planner.start
- audit.planner.end
- audit.planner.error
Partially Implements: blueprint audit-versioned-notifications-api
Change-Id: I29d371f8cb3fd7948c87bec97186e24a88e8fdb6
In this changeset, I implemented the following notifications:
- audit.strategy.start
- audit.strategy.end
- audit.strategy.error
Partially Implements: blueprint audit-versioned-notifications-api
Change-Id: I6ae8468caf8d215bc8bc694813beb4dc94f53fdb
In this changeset, I implemented the audit.delete notification.
Partially Implements: blueprint audit-versioned-notifications-api
Change-Id: I8aeb73f4b8d3273d6de7dc9c44674e3041b005ea
In this changeset, I implemented the audit.create notification.
Change-Id: Ia092ca3a3dc951e3313a07f15a98aec5818e9ab0
Partially-Implements: blueprint audit-versioned-notifications-api
In this changeset, I implemented the sending of update notifications
whenever an audit is modified.
Change-Id: I5ccc2516ce896ae7d4ef542b133e8f052eaed602
Partially-Implements: blueprint audit-versioned-notifications-api
In this changeset, I updated the devstack plugin so it has the
necessary configuration options for it to emit the Watcher
notifications straight out of the box.
Partially Implements: blueprint watcher-notifications-ovo
Change-Id: I048e0b82cde40bd3699c90429bdf55ee235fff3b
In this changeset, I implemented the notification_level parameter
that allows you to set the level minimum level of notification to
be emitted by Watcher.
Partially Implements: blueprint watcher-notifications-ovo
Change-Id: I78b30ceb3ee7606078a87beb3a4ce5d9568af1e0
In this changeset, I removed the now obsolete status_topic config
option.
DocImpact
Partially Implements: blueprint watcher-notifications-ovo
Change-Id: Icfc03abd875b77fc456bfa286ac2b5774651e8fa
In this changeset, I cleaned up the Watcher codebase to remove
the old notification mechanism that is actually unused.
Partially Implements: blueprint watcher-notifications-ovo
Change-Id: I1901e65f031441b98a7d6f6c9c1c0364eaaaf481
In this changeset, I added a new Sphinx directive (from Nova) that
allows us to automatically display notification samples for of
all the upcoming notifications in Watcher.
Partially Implements: blueprint watcher-notifications-ovo
Change-Id: Id7d819ee21db6dc616c78faea483b883f77e3bd6
In this changeset, I added all the required modification in order
for Watcher to enable the implementation of versioned notifications.
Change-Id: I600ecbc767583824555b016fb9fc7faf69c53b39
Partially-Implements: blueprint watcher-notifications-ovo
This patch adds documentation on requirements for
vm_workload_consolidation strategy.
Change-Id: I304644138b34658b09f7cd8f29eb089757807b61
Closes-bug: #1640820
This patch set removes extra field from rally tasks
since Watcher team has removed extra field from
watcher and python-watcherclient projects.
Change-Id: Ib1640cbe8668f56f3a3a54e9f73bb1e3e6747d79
This patch set allows in case of NoMetricValuesForInstance exception
to continue the flow of Workload Stabilization strategy
instead of interrupting.
Closes-Bug: #1640096
Change-Id: I2de22bb97defb9c34e409102d06846275793b80d
The function nova.servers.list() will return a huge context in the
large infrastructure. We can use nova.servers.get() instead of it.
Change-Id: If922c31e8aff27bfbc475c63083ee561ee7f1f67
Submitted state is not referenced nor used in the Watcher codebase,
and hence removed the references to the state from the codebase.
Change-Id: Ibb0cf5125e2d93683739abf35030b7cf5cb1b056
Closes-Bug: 1634150
In this changeset, I added the "action_plan" ObjectField which can
either be loaded by setting the new "eager" parameter as True
or not loaded (as before) by setting it to False. The advantage of
introducing this eager parameter is that this way,
we can reduce to a minimum the overhead of DB queries whenever the
related object fields is not actually needed.
Change-Id: Iae255d9a0a919e2b6db710be24dbf9033b72166e
Partially-Implements: blueprint watcher-versioned-objects
In this changeset, I added the "audit" and the "strategy "ObjectField
which can either be loaded by setting the new "eager" parameter as True
or not loaded (as before) by setting it to False.
The advantage of introducing this eager parameter is that this way,
we can reduce to a minimum the overhead of DB queries whenever the
related object fields is not actually needed.
Change-Id: Ieca6fa438cbf2267f5ae2ea0d059b2953e65076b
Partially-Implements: blueprint watcher-versioned-objects
In this changeset, I added the "goal" and the "strategy "ObjectField
which can either be loaded by setting the new "eager" parameter as True
or not loaded (as before) by setting it to False.
The advantage of introducing this eager parameter is that this way,
we can reduce to a minimum the overhead of DB queries whenever the
related goal is not actually needed.
Change-Id: I5a1b25e395e3d904dae954952f8e0862c3556210
Partially-Implements: blueprint watcher-versioned-objects
In this changeset, I added the "goal" and the "strategy "ObjectField
which can either be loaded by setting the new "eager" parameter as True
or not loaded (as before) by setting it to False.
The advantage of introducing this eager parameter is that this way,
we can reduce to a minimum the overhead of DB queries whenever the
related goal is not actually needed.
Change-Id: I97418163f68aea2f1cda80be94e4035f0b3700ae
Partially-Implements: blueprint watcher-versioned-objects
In this changeset, I added the "goal" ObjectField which can
either be loaded by setting the new "eager" parameter as True or
not loaded (as before) by setting it to False.
The advantage of introducing this eager parameter is that this way,
we can reduce to a minimum the overhead of DB queries whenever the
related goal is not actually needed.
Partially-Implements: blueprint watcher-versioned-objects
Change-Id: I103c9ed161d2cedf7b43c55f9e095ef66bf44dea
In this changeset, I modified all existing Watcher objects to now
rely on oslo.versionedobjects as a base.
Change-Id: I3c9b1ca6da529d128743b99020350f28926ea1a2
Partially-Implements: blueprint watcher-versioned-objects
As we are about to version the Watcher objects, we need to make sure
that upcoming model/object modifications are additive in order to
avoid having to bump the major version of the API. Therefore,
this changeset removes 4 unused DB fields that were exposed in their
associated Watcher objects (i.e. AuditTemplate and Audit).
Change-Id: Ifb0783f21cd66db16b31e3c8e376fc9d6c07dea3
Partially-Implements: blueprint watcher-versioned-objects
In this changeset, I added ORM relationships to the DB models
concerning the already-declared foreign keys.
I also modified the DB query building to now handle a new 'eager'
parameter that, if True, is responsible to also fetch the data
relative to these 'parent' DB entities (no cascading).
Change-Id: Ieea181af9a4b173c54621dcc6c549161f5a35aeb
Partially-Implements: blueprint watcher-versioned-objects
This patch is deprecated use xx = [] for the parameter initial value,
this parameter will only be initialized at the first call,this is not
what we expected.
Better choice is to set the initial value to None,
then initialize xx with xx= xx or [] in method body.
More details:http://effbot.org/zone/default-values.htm
Change-Id: Iee0648fd500422f48c3f7f35b5066fdc644ae308
This patch set normalizes CPU for hardware.cpu.util meter and
transform KB into MB fetched from hardware.memory.used meter.
Change-Id: I6dd1d053e81f06c3adb6fb256ac77bbf55a07f3e
Developer should provide a detailled documentation about his strategy
algorithm to make it even easier to use by a Watcher end user.
I propose in this changeset a template for strategy documentation.
you will find also a example with basic consolidation strategy.
Change-Id: I66da1a33b87a94b508dd23ac7dce4cae6f4e068b
watcher already uses PBR:-
setuptools.setup(
setup_requires=['pbr>=1.8'],
pbr=True)
This patch removes `MANIFEST.in` file as pbr generates a
sensible manifest from git files and some standard files
and it removes the need for an explicit `MANIFEST.in` file.
Change-Id: If1b80738e28d71fd9e9bb4939f94adfd721a041a
Closes-Bug:#1608980
This patch set adds audit scope mechanism.
It also removes host_aggregate field.
Change-Id: Ia98ed180a93fc8c19599735e2b41471d322bae9a
Partially-Implements: blueprint define-the-audit-scope
This patch set adds service object to the watcher_db_schema_diagram.txt
New watcher_db_schema_diagram.png is compiled and attached.
Partially-Implements: blueprint watcher-service-list
Change-Id: Ibaac9020b0cb9fb147259a8685a7a072216ff95d
This patch set adds supervisor mechanism for Watcher services
to get ability to track states.
Partially-Implements: blueprint watcher-service-list
Change-Id: Iab1cefb971c79ed27b22b6a5d1bed8698e35f9a4
In this changeset I added a Model abtract base class for models
and updated the plugin docs to reference it. I also added the
CDM and CDMC acronyms to the glossary.
Change-Id: Ie7d76a9f84d545715fa6e5bf350b9d2321bab9e1
Releasenote translation publishing is being prepared. 'locale_dirs'
needs to be defined in conf.py to generate translated version of the
release notes.
Note that this repository might not get translated release notes - or
no translations at all - but we add the entry here nevertheless to
prepare for it.
Change-Id: I67b2202570e23d3f20f7d132b7d7dbb460a2af3f
In this changeset, I added composite contraints on models that
were currently subject to an implicit uniqueness.
This composite constraint takes as an input the name and the deleted
columns.
The 'deleted' column is 0 by default and is updated with the value of
the associated 'id' column upon soft deletion.
What this means is that this composite constraint guarantees the fact
that we can hold many soft deleted records holding the exact same name.
Indeed, the value of the deleted column will be different whilst
forbidding 2 non-soft deleted records with the same name as their
associated 'deleted' value will be 0 and subsequently violates the
constraint.
Change-Id: I3ee39e23aa4ca9bc6b4ea9af8c6b7e6d67af0136
The show_models() function was not covered by any test so this
changeset is adding a new unit test for it.
Change-Id: Ia861f049c1f1ee0aa623404e62fb002392c9052d
In this changeset, I renamed watcher/doc.py file as
doc/ext/term.py which subsequently does not get scanned for coverage
anymore. Hence, we can close this bug.
Change-Id: I9d700da6569b464ce71085fe78002521555002b9
Closes-bug: #1527163
Service available group already exists.Therefore we don't need to
register this group here again.
Change-Id: I2096ed1f69a1f70fbf248fee5fded94e3caca6c6
Closes-Bug: #1621036
HasLength() based assertions are re-written to assertEqual() based
assertion to homogenize the test code base.
For example:
assertThat(.., HasLength()) becomes assertEqual(len(),...)
Change-Id: Ia3bf24e34fffe445ff4b0ee61fb7cbc0eb36a57a
Closes-Bug:#1629898
In tests/api/test_hooks.py
the 'test_hook_without_traceback_debug'
and 'test_hook_without_traceback_debug_tracebacks'
have the same function.
Change-Id: I5209bc8b1251b77b7444d51531c3902529d11453
As some of our utils function are actually available in oslo.utils,
this patchset simply gets rid of our local copy and instead references
the oslo.utils one. This will help us deport the code maintenance of
these to a 3rd party library.
Change-Id: I3760ebab69d35c37bbe78500918d4e46b5bd9119
GMR was ignoring the config because the conf wasn't passed when
starting any of the Watcher services. This changeset fixes this issue.
Change-Id: If386c5f0459c4278a2a56c8c3185fcdafce673a0
This adds a pip install command to tox.ini that is only used when the
tox env is passed with the 'constraints' factor appended onto it.
As such this will not effect developer workflows or current unit tests.
The initial use of this will be in a non-voting job, to verify that the
constrained checks with tox are stable. DevStack is already running
constrained jobs, as such problems are no expected.
To run a tox with pip using constraints on a developer system a
developer should run the desired tox environment with -constraints.
For example: $(tox -epy27-constraints)
Pip will pull the current version of the upper-constraints.txt file down
from the git.openstack.org, however this method can be overriden to use
a local file setting the environment variable "UPPER_CONSTRAINTS_FILE"
to the local path or a different URL, it is passed directly to pip.
This is currently not enabled in the default tox run, however it is
possible to enable it as a default by adding it to 'envlist' in tox.ini
This also removes requirements.txt from tox.ini deps
This is redundant, per lifeless email:
http://lists.openstack.org/pipermail/openstack-dev/2015-July/069663.html
Change-Id: I79c0ceb46fc980840a8baf5fa4a303bb450bfbec
cfg.CONF.debug should be set False as default,
If it's True, some unittests can't pass.
Change-Id: Ib098250af3aec48aa9d9152e20c80460f3bd641e
Closes-Bug: #1625560
I deactivated the watcher-dashboard plugin in the docs since it
doesn't at the moment. Will be reactivated once it gets fixed
(see https://bugs.launchpad.net/devstack/+bug/1540328).
Change-Id: I65de8ca4a08cdb294d0e48eac52927eed13ec66c
root.description = ("Watcher is an OpenStack project which aims to"
"to improve physical resources usage through"
Here more one 'to'
Change-Id: I3dac6cb15af43583a1360b7f2f14d4e2c9c26cf9
In this changeset, I fixed the issue with the basic server
consolidation strategy to now loop over all compute nodes
as expected instead of stopping after the first one.
Change-Id: If594f0df41e39dfb0ef8f0fce41822018490c4ec
Closes-bug: #1548874
ceilosca seems to not correctly handle group_by statement.
I remove it, because we don't need it for now.
Change-Id: Ie18551ba5a6bd155f9f4ae9c29c25b379affe019
Closes-Bug: #1623983
This patch set adds rally-jobs folder with watcher.yaml scenario
to get rally support for Watcher project
Change-Id: Icb8eace045d86a9b78b543a8a49fe747f4b00772
In this changeset, I added debug logs to dump the cluster data model
copy structure (as XML) before and after the execution of the strategy.
By doing so, we can see the cluster data model structure that is
expected after the execution of the corresponding action plan.
Change-Id: I81c23e148a78d9b176154f7620087a322a5bce28
Config option sqlite_db is deprecated in
0a1bae9859079fb21a03716be947c5f1da6db0a2, and deprecate argumentsqlite_db in method set_defaults in
https://review.openstack.org/#/c/350945/, should use config option
connection instead. For watcher, we test database with sqlite in memory
mode [1] as below:
cfg.CONF.set_override("connection", "sqlite://",
group="database", enforce_type=True)
and don't use config option sqlite_db, so remove unused code.
[1]http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#sqlite
Change-Id: I9b1f995e1b7004bcfe6c5a854c2f83b24e5bfb56
When the action plan is empty,its state is still 'RECOMMENDED'.
I think it's should be 'SUCCEED'.
Closes-Bug: #1619155
Change-Id: Icd10707c7892089999f8b37775ca62bde47db0c4
This patch set allows to use audit parameters for
workload-stabilization strategy and makes some little
refactoring.
Closes-Bug: #1620604
Change-Id: I60e34611d4dd001beed31666fd11d2ab11c1723c
This change is updating the glossary with Scoring Module terms
and adding an information page about implementing scoring engine
plugin.
Partially-Implements: blueprint scoring-module
Change-Id: I411370dcc003ed837d8ce67659ecbfef3548ee11
In this changeset, I implemented a small GMR plugin that converts
the cluster data model structure into an XML structure.
Change-Id: I75548952635a0aa3c7dbd6d068831b5765a5db1a
Closes-Bug: #1620551
There were a few typos and some lines of code that weren't formatted
properly. Now, the typos are fixed and the misformatted code is
corrected.
Change-Id: I6009e44f65b9be1ae9edc806618342c334b2fa92
The default value of 'is_admin_project' should be 'True'
As in the file '/oslo_context/context.py':
def __init__(self,...
is_admin_project=True)
Change-Id: Id5e3f7347819deff27c6423679f9964fbb35f047
Watcher consumes now notications sent by Nova services.
We have to configure Nova to publish its notifications into
the dedicated Watcher notification queue.
Change-Id: I29f2fa12dfe3a7ce0b014778109a08bbe78b4679
Partially-Implements: blueprint cluster-model-objects-wrapper
In this changeset, I updated the CDMC plugin documentation to explain
how to implement and register new notification endpoints.
Change-Id: Ib8c014e82051647edef5c1272f63429f76673227
Partially-Implements: blueprint cluster-model-objects-wrapper
In this changeset, I implemented the notification handling (Rx only)
system for consuming incoming notifications, more especially the Nova
ones. The notifications handlers also contain the logic which
incrementally updates the Compute model.
Change-Id: Ia036a5a2be6caa64b7f180de38821b57c624300c
Partially-implements: blueprint cluster-model-objects-wrapper
In this changeset, I added the start_time and end_time params
to the Ceilometer helper which can drastically reduce the execution
time of the queries.
Change-Id: I39cb3eef584acfca1b50ff6ec1b65f38750802d2
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where
an exception occured initially instead of place where
the exception re-raised
Change-Id: I662583cd3cda2424d5a510cae7e815c97a51c2fe
Currently, create audit with unspecified parameters will success.
This is not reasonable, we shoud return a FAILED status to
notify the admin user.
Change-Id: Ifbcb3b8d9e736607b05b1eb408ec0f41bdf58a2f
Closes-Bug: #1599879
When the load a plugin, we need to reload once the watcher
configuration data, in order to include the plugin parameters in
cfg.CONF data dict. To reload the conf, we just call self.conf().
But every time we call this method, cfg.CONF is cleaned, and we
lost previously loaded parameters. This generated the exception
RequiredOptError and the plugin was not correctly loaded.
To fix it, we have just to add the watcher configuration
filename as argument of self.conf().
Change-Id: Ic2384b68f6d604640127fe06893d0f808aee34b7
Closes-Bug: #1617240
This change is adding the main logic for the scoring module,
defines entry points for the scoring engine plugins and provides
a watcher-sync tool to enable Watcher database synchronization
without needing to restart any Watcher service.
Partially-Implements: blueprint scoring-module
Change-Id: If10daae969ec27a7008af5173359992e957dcd5e
Fixed the flaky test_delete_audit test which was failing if the delete
query was sent before the audit reaches an 'idle' state.
Change-Id: I244643b1f7c9b31baa5c25753e62dd3da0a53544
We do not store pot files anymore in repositories,
instead they are published at
http://tarballs.openstack.org/translation-source/watcher/master/ after
each commit and thus always accurate.
Remove the outdated and obsolete file.
Change-Id: I24f18c0a62f2c5339479d07904fb2ce0a888c696
In this changeset, I implemented the logic which cancels
any audit or action plan whose goal has been re-synced
(upon restarting the Decision Engine).
Partially Implements: blueprint efficacy-indicator
Change-Id: I95d2739eb552d4a7a02c822b11844591008f648e
Due to the recent code refactor in the cluster data model
compute model is failing to 'get_latest_cluster_data_model'
attribute
Change-Id: Iba848db6d03cf1b682c4000ca48cf846b0ffa79b
Closes-Bug: #1614296
This patch set renames 'OS-EXT-STS:instance_state' to 'OS-EXT-STS:vm_state'
for correct working of decision_engine scheduler.
Change-Id: I20805a079a991d5f3b8565f52d5f7280c2389bee
Closes-Bug: #1614511
In some part in the code we import objects.
In the Openstack style guidelines they recommand to import
only modules. We need to fix that.
Change-Id: I268c5045f00d25b4dfbf77c1f599c7baca8373ed
Partial-Bug:1543101
This patch updates the applier's abstract methods to be consistent
with other abstract methods of similar nature.
Also included are a few other miscellaneous changes.
Change-Id: Ia1527c00332011412aba2ab326ec986f1e773001
Closes-bug: 1606634
The nova helper does not have a sufficient code coverage.
This commit raises its coverage from 39% to 89%.
And unit tests find two bugs.
Closes-Bug: #1607198
Change-Id: Iebb693cd09e512ce44702eddca8ead0c7310b263
Closes-Bug: #1599849
Since there are so many components in openstack,
by describing the URL info in the configuration file,
it is more convenient to find the developer documentation
than from the unified portal
Change-Id: Id6bc554db8bb0cd71851773b9cea71aada4eb9e2
In this changeset, I refactored the whole Watcher codebase to
adopt a naming convention about the various elements of the
Compute model so that it reflects the same naming convention
adopted by Nova.
Change-Id: I28adba5e1f27175f025330417b072686134d5f51
Partially-Implements: blueprint cluster-model-objects-wrapper
Many places, there are more specific asserts which can be used.
I replaced the generic assert with more specific ones, where
it was possible.
This change enhances readibility, and on fail, more useful
message is displayed
Change-Id: I86a6baeae2cd36610a2be10ae5085555246c368a
In this changeset, I merged the metrics_engine package into
the decision_engine one alongside the required changes to make
the tests pass.
Change-Id: Iac1cd266a854212bf4fa8b21c744b076c3b834a8
Partially-Implements: blueprint cluster-model-objects-wrapper
In this changeset, I updated the architecture documentation
about the Decision Engine by adding a new sequence diagram
outlining the CDM synchronization workflow.
I also explained the default 'period' parameter used in the
CDMC plugin.
Change-Id: I09790281ba9117e302ab8e66a887667929c6c261
Partially-Implements: blueprint cluster-model-objects-wrapper
In this changeset, I implemented a background scheduler service
for Watcher and more particularly for the Decision Engine where
I made it create 2 types of job per cluster data model collector
plugin:
- An initial job that is asynchronously executed upon starting the
Decision Engine
- A periodical job that gets triggered every configurable interval
of time
Change-Id: I3f5442f81933a19565217b894bd86c186e339762
Partially-Implements: bluprint cluster-model-objects-wrapper
In this changeset, I wrote down the documentation on how to implement
a cluster data model collector plugin for Watcher.
This documentation corresponds to part 1 of the associated
specification.
Change-Id: Iac72b933df95252163033cd559d13348075a9b16
Partially-Implements: blueprint cluster-model-objects-wrapper
In this changeset, I made BaseClusterDataModelCollector instances
pluggable. This corresponds to "part 1" of the work items detailed
in the specifications.
Change-Id: Iab1c7e264add9e2cbbbb767e3fd6e99a0c22c691
Partially-Implements: blueprint cluster-model-objects-wrapper
A Scoring Module needs to expose a list of available
scoring engines through API and Watcher CLI. This list
is stored in database and synchronized by Decision Engine.
Partially-Implements: blueprint scoring-module
Change-Id: I32168adeaf34fd12a731204c5b58fe68434ad087
APIImpact
Modifying the api controller for audit objects to allow
creation of audit objects by specifying either an
audit_template uuid/id and/or a goal_id.
strategy_id is optional.
Partially Implements: blueprint persistent-audit-parameters
Change-Id: I7b3eae4d0752a11208f5f92ee13ab1018d8521ad
use parameters to set the threshold for strategies
of Uniform Airflow,Workload balance.
Change-Id: I5a547929eb1e2413468e9a5de16d3fd42cabadf9
Close-bug: #1600707
The hacking checks enforce during the pep8 run functional validations of
the code to ensure deeper filters and code consistency. This change set
adds the hacking checks to the wathcer project. These checks were
seeded from the neutron project, which had a good set of base defaults.
This change set also updates the watcher project to be compliant with
these new hacking checks.
Change-Id: I6f4566d384a7400bddf228aa127a53e6ecc82c2e
This patch fixes various typos and other nits in the strategies. It
also updates some of the log messages to be a little more operator
friendly.
Change-Id: Ic9268c6d7376dad215a1a40798485b1d836ba7ae
Closes-Bug: #1604248
In watcher/db/api.py, some abstract methods are specifying a 'columns'
parameter that is actually ignored in db/sqlalchemy/api.py. Since we
do not need this parameter, realignment was done for the signatures
of these methods, by removing the 'column' parameter (and its
docstring) from every single one of the following methods.
get_audit_template_list
get_audit_list
get_action_list
get_action_plan_list
Closes-Bug: #1597641
Change-Id: If706e24d5714f0139fd135bdc41d17d0e431e302
gate-watcher-python27 FAILURE
The reason is that oslo_context.context.__init__()
added parameter 'domain_name'.
There are some mismatching in file '/watcher/common/context.py'
Change-Id: If0be05943e7c89788d6ccba3385474ccb036e6f5
Closes-Bug: #1604214
The dict.keys()[0] will raise a TypeError in PY3,
as dict.keys() doesn't return a list any more in PY3
but a view of list.
Change-Id: If15a153c9db9b654e761f8ad50d5d66a427efa4e
Closes-Bug: #1583419
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv.
Change-Id: Idf6cd632bcb6f4f61dba65fedc9309d0184f46b7
The goal name 'airflow optimization' must be synched with the
entry point name.
The strategy name 'uniform airflow' must be synched with the
entry point name.
Partial-Bug: #1602262
Change-Id: Ibc6f0a7c5047e57549910818200c323b878152b0
tox -e py27 fails with this error:
socket.error: [Error 98] Address already in use
Change-Id: I1cf4a000fb82228bed6a7da61feed48ab679b891
Closes-Bug: #1599556
This patch set adds implementation for CONTINUOUS type
of audit.
Change-Id: I5f4ec97b2082c8a6b3ccebe36b2a343fa4a67d19
Implements: blueprint continuously-optimization
Add docs of how to define and input parameters of strategy.
Change-Id: I918305b89721d141b7f37630459e6cf5999f4211
Partially-Implements: optimization-threshold
Debian experimental packages are now available for Watcher.
I added a new section to help users to quickly deploy these
packages and test them.
Change-Id: Ib920b7dbf968c36941c0d30c9001bc150df543f8
We can now use policies to allow or not users to invoke Watcher API
methods. I added, in the Admin guide section, a new documentation
introducing watcher policy rules.
Change-Id: Ie1f8a9e15561756f23242c6f6b0d53cdacf3505a
blueprint: watcher-policies
Default 'blocking' executor handles only one message at once.
'eventlet' executor is recommended.
Change-Id: Id738d0462fbb3c7fd6c78ee2f0dd0f1e79131ca7
Closes-Bug: #1517843
Updates the module name to reflect the right defination of efficacy
specification, and adds doc string to some class.
Change-Id: If54e65a5e98521d74d8e19a21dd900ecd37edfe5
Closes-Bug: #1599368
There's a number of places where a field of an object is named "type",
which isn't good thing. Actually type is keyword in python, so wherever
I found type used in models/api/object/test, I have fixed it to
audit_type in the audit files
Change-Id: Iea6bd3acb0b2af2a833b3916701aad88f6064bba
Closes-Bug: #1533392
line 93:An ref:'action plugin <implement_strategy_plugin>
here implement_strategy_plugin should be 'implement_action_plugin'
Change-Id: I3487578b91bd169335ccf336443e6174905728b0
Closes-Bug: #1599029
Strategy provides parameters to customize algorithm behavior. End user
could query then specify parameters for their requirements.
Change-Id: Id097db5f6e79c94b57674c8e5d55b06098abf18c
Implements-bp: optimization-threshold
In the base-setup.rst about 3rd-party plugins,
we should add the watcher dependency requirement.
Change-Id: I8f898b2e3e7e30c88f4f14f7bd3ae38b8aa3ebd3
Closes-Bug: #1594705
Currently we have to amend the code of the Planner
for each new added Watcher Action. This patch set will
modifying the DefaultPlanner class and leverage the
plugins-parameters blueprint so that we can configure
weights via the configuration file.
Documentation and unittest also be update in this patch.
Co-Authored-By: Vincent Françoise <vincent.francoise@b-com.com>
Change-Id: Ib794488fcafd1e153a7d7b1f7253686136501872
blueprint: configurable-weights-default-planner
This is one of the algorithm of Intel thermal POC.
It's based on the Airflow of hypervisors.
Change-Id: I48bbab75b20129d85b1fb15ede823474546fe399
blueprint: uniform-airflow-migration-strategy
Refactor all plugin loaders in watcher/applier into
watcher/applier/loading/default.py.
Change-Id: I0f0a87d4f72ead45d34aca1b14219fd2ede42a6f
Closes-Bug: #1591095
This patch removes import of classes and replaces them with import of modules.
Change-Id: Id2502cf96ac7227cf1036cd54a74f3d7acd83479
Closes-Bug: #1594434
The VM workload consolidation strategy sends 'down' instead
of 'disabled' to change nova-compute service state. This patch
will correct it. The same applies for enabling nova-compute service.
Change-Id: I257411ef711b215bd9b56d0bf0479c79a9ef61d8
Closes-Bug: #1591901
This patch updates VM workload consolidation strategy adding a new
condition to check whether nova-compute service is disabled before
creating the action.
Change-Id: I1accbc7bbd62684dce14ca3b35c92121e923a73a
Closes-Bug: #1591927
Add bandit tox environment and amend pep8 env to run it.
Also, fix bandit errors with "0.0.0.0".
Change-Id: Ieb5785abd945663e07c07f0ddd3d9a074004f46a
Closes-Bug: #1594423
In this changeset, I refactored the DB filter system to support
comparison operators using a django-like syntax.
A filter can take 2 forms:
- "<FIELDNAME>" which is a syntactic sugar for "<FIELDNAME>__eq"
- "<FIELDNAME>__<OPERATOR>" where <OPERATOR> is the comparison operator
to be used.
Here is the list of the supported operators:
- 'eq' (==)
- 'neq' (!=)
- 'gt' (>)
- 'gte' (>=)
- 'lt' (<)
- 'lte' (<=)
- 'in' (in)
- 'notin' (not in)
Change-Id: I53a61d50a3253342a40f0ff87cb5612ed57a3bd1
This patch fixes the StrategyContext to use the optional
attribute strategy_id.
Change-Id: Ib78581f564282de6cfc7f07495c846615ec1866a
Closed-bug: #1590357
Fix several problems, tox -e releasenotes did not run at all.
Fixes:
* Fix version import so that release note building works.
* There's no stable/mitaka branch for watcher, so remove the mitaka file,
otherwise release notes build will fail.
* Add _static directory with empty placeholder for sphinx to run.
Change-Id: Ia4dd2f7a7743dfd7f7022be04552c401ade9953f
In this changeset, I wrote a documentation detailing how one can
implement a new goal plugin. I also mention to define the efficacy
specification for a given goal.
Partially Implements: blueprint efficacy-indicator
Change-Id: Iba267ae312f248b49d4600504f11678cdc225622
I this changeset, I added the efficacy indicators both at the DB
and at the API level alongside the associated logic.
Partially Implements: blueprint efficacy-indicator
Change-Id: I824553637621da67966103c1b0c01348b09bd836
In this changeset, I broke down the execute() method to sequentially
call 3 methods:
- pre_execute()
- do_execute()
- post_execute()
This changeset also removes the cluster model parameter from the
execute() method to now become a `model` property of a strategy which
is lazy loaded whenever needed.
Partially Implements: blueprint efficacy-indicator
Change-Id: I2f697938db693acfa95b2c2fbecfdc1b733c93fd
In this changeset, I created the Watcher object for the efficacy
indicators as well as the API object which will be embedded into the
/action_plans endpoint.
Partially Implements: blueprint efficacy-indicator
Change-Id: Ibbe47613317d51a3829fe9de12540c048e8d7117
In this changeset, I added the new Efficacy, EfficacySpecification
and IndicatorSpecification classes which are the main components for
computing an efficacy.
Partially Implements: blueprint efficacy-indicator
Change-Id: I3a1d62569de2dd6bb6f9a52f6058313fa2b886ce
In this changeset, I created a new model named EfficacyIndicator
which is responsible for storing information regarding the efficacy
indicators that were computed by the strategy in its solution. Every
efficacy indicator should relate to a single Action Plan.
Partially Implements: blueprint efficacy-indicator
Change-Id: Ifc14ea5e16e92f032d7912c9b3fdc270af79cab6
In this changeset, I decoupled the notion of Goal from the Strategy
by making it a distinct object. Goals are plugins that can be loaded
just like for the strategies.
Partially Implements: blueprint efficacy-indicator
Change-Id: I4378dccd508170b305aa968843228bbc8af78895
In this changeset, I added the missing entries for the configuration
sample file generation:
- oslo.reports
- oslo.cache
- oslo.concurrency
- oslo.policy
- oslo.service.periodic_task
- oslo.service.service
- oslo.service.wsgi
Change-Id: I826f5cc1185d75f545c96242022a07089cf3042e
In this changeset, I added the 'audit_template' filter to the
/audits/detail endpoint so it has the same set of parameters as
the /audits endpoint.
Change-Id: I2568be18854bd97d4b7879532cb36a7cac532719
This patch set removes normalizing for hardware.cpu.util meter
since the values that comes from ceilometer hardware.cpu.util
are already normalized.
Closes-Bug: #1588257
Change-Id: I9494f2cc9bbaa6dfd168fb515f679eb6d7f2398a
Following https://review.openstack.org/#/c/263376/, I update the
way Watcher tempest test retrieve the admin credentials.
Change-Id: I9633752ff69a27edf8912227ea2fd24f358dec7e
In this changeset, I updated the documentation to explain how to
add configuration options for each type of plugin.
Partially Implements: plugins-parameters
Change-Id: Ifd373da64207110492b4a62f1cb7f13b029a45d2
This is one of the algorithm of Intel thermal POC.
It's based on the VM workloads of hypervisors.
Change-Id: I45ab0cf0f05786e6f68025bdd315f38381900a68
blueprint: workload-balance-migration-strategy
In this changeset, I added the list of all the available plugins
for the current instance of any given Watcher service.
Partially Implements: blueprint plugins-parameters
Change-Id: I58c9724a229712b0322a578f0f89a61b38dfd80a
In this changeset, I added the possibility for all plugins to define
configuration parameters for themselves.
Partially Implements: blueprint plugins-parameters
Change-Id: I676b2583b3b4841c64c862b2b0c234b4eb5fd0fd
The main purpose of this strategy is to choose the pair VM:dest_host that
minimizes the standard deviation in a cluster best.
Change-Id: I95a31b7bcab83411ef6b6e1e01818ca21ef96883
Implements: blueprint watcher-overload-sd
We introduced a new watcher plugin for OpenStack CLI. This patchset
updates accordingly the watcher documentation and schemas.
Partially Implements: blueprint openstackclient-plugin
Change-Id: Ib00469c8645fff21f5ba95951379827dbd359c69
Cold migration enables migrating some of the VMs which are not in active state (e.g. stopped). Cold migration can also be used for migrating active VM, although VM is shut down and hence unaccessible while migrating.
Change-Id: I89ad0a04d41282431c9773f6ae7feb41573368e3
Closes-Bug: #1564297
In this changeset, I added both the 'goal_name' and the 'strategy_name'
field.
Change-Id: Ic164df84d4e23ec75b2b2f4b358cf827d0ad7fa5
Related-Bug: #1573582
Replace assertEqual(None, *) with assertIsNone in tests to have
more clear messages in case of failure.
Change-Id: I98261ef7cca06447ea9d443a2c287c046f380f77
Closes-Bug: #1280522
In this changeset, I fix the issue caused by the use of lazy
translations within the 'watcher-db-manage purge' subcommand.
This is caused by the PrettyTable dependency which performs
addition operations to format its tables and the __add__ magic
method is not supported by oslo_i18n._message.Message objects.
Change-Id: Idd590e882c697957cfaf1849c3d51b52797230f6
Closes-Bug: #1584652
In this changeset, I fixed the test_create_audit_with_no_state
tempest test which was randomly failing because of a race condition.
Change-Id: Ibda49944c79fcd406fa81870dbbff6064b5dc4fa
Since telemetry was removed from tempest, this changeset removes the
telemetry tags from the watcher integration tests
Change-Id: I6229ee23740c3d92a66fc04c8de8b0ed25911022
This patch set fixes aggregate parameter for statistic_aggregation
function so we may use it with any aggregate function.
Change-Id: If586d656aadd3d098a1610a97a2f315e70351de5
Closes-Bug: #1583610
In this changeset, I just generate the .pot file for all the new
translations that were added during the implementation of this BP
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I2192508afda037510f8f91092c5cfde0115dae1d
In this changeset, I remove the now unused [watcher_goals] section.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I91e4e1ac3a58bb6f3e30b11449cf1a6eb18cd0ca
In this changeset, I removed the now useless [watcher_goals] section
from the devstack plugin.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Iaa986f426dc47f6cbd04e74f16b67670e3563967
In this changeset, I updated the Watcher documentation to reflect
the changes that are introduced by this blueprint.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I40be39624097365220bf7d94cbe177bbf5bbe0ed
In this changeset, I updated the purge script to now take into
account the registered goals and strategies.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I2f1d58bb812fa45bc4bc6467760a071d8612e6a4
In this changeset, I introduce the syncing of audit templates.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Ie394c12fe51f73eff95465fd5140d82ebd212599
In this changeset, I updated the 'goal_id' field into the AuditTemplate
to now become a mandatory foreign key towards the Goal model. I also
added the 'strategy_id' field into the AuditTemplate model to be an
optional foreign key onto the Strategy model.
This changeset also includes an update of the /audit_template
Watcher API endpoint to reflect the previous changes.
As this changeset changes the API, this should be merged alongside the
related changeset from python-watcherclient.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Ic0573d036d1bbd7820f8eb963e47912d6b3ed1a9
In this changeset, I refactored the strategy selector to now
look into the Watcher DB instead of looking into the configuration
file.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I2bcb63542f6237f26796a3e5a781c8b62820cf6f
In this changeset, I added the /strategies endpoint to the Watcher
API service.
This also includes the related Tempest tests.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I1b70836e0df2082ab0016ecc207e89fdcb0fc8b9
In this changeset, I changed the Strategy base class to add new
abstract class methods. I also added an abstract strategy class
per Goal type (dummy, server consolidation, thermal optimization).
This changeset also includes an update of the /goals Watcher API
endpoint to now use the new Goal model (DB entries) instead of
reading from the configuration file.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Iecfed58c72f3f9df4e9d27e50a3a274a1fc0a75f
In this changeset, I added the ability to synchronize the strategies
into the Wather DB so that it can later be served through the Watcher
API.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Ifeaa1f6e1f4ff7d7efc1b221cf57797a49dc5bc5
In this changeset, I add the Strategy model as well as the DB
functionalities we need to manipulate strategies.
This changeset implies a DB schema update.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I438a8788844fbc514edfe1e9e3136f46ba5a82f2
In this changeset, I added the Goal object into Watcher along with
a sync module that is responsible for syncing the goals with the
Watcher DB.
Partially Implements: blueprint get-goal-from-strategy
Change-Id: Ia3a2032dd9023d668c6f32ebbce44f8c1d77b0a3
In this changeset, I added the Goal model into Watcher.
This implies a change into the Watcher DB schema
Partially Implements: blueprint get-goal-from-strategy
Change-Id: I5b5b0ffc7cff8affb59f17743e1af0e1277c2878
When starting the Watcher API service, the URI it served to is shown in
a log message. In this log message (in watcher/cmd/api.py) take into
account the case where SSL has been enabled with CONF.api.enable_ssl_api
set to True and format this log message accordingly.
Change-Id: I98541810139d9d4319ac89f21a5e0bc25454ee62
Closes-Bug: #1580044
This patch set removes the possibility of using UUID field
in POST methods of Watcher API.
Closes-Bug: #1572625
Change-Id: I88a8aa5346e937e3e9409b55da3316cbe1ed832a
In this PS, I have refactored the Decision Engine and the Applier
to use the oslo service utility.
Change-Id: If29158cc9b5e5e50f6c69d67c232cceeb07084f2
Closes-Bug: #1541850
This patchset introduces the use of oslo.service to run the
Watcher API service.
Change-Id: I6c38a3c1a2b4dc47388876e4c0ba61b7447690bd
Related-Bug: #1541850
We have to check Audit Type and Audit State to make sure
these parameters are in valid status.
Also, we provide default states for the next attributes:
- 'audit_template' is required and should be either UUID or text field
- 'state' is readonly so it raises an error if submitted in POST
and is set by default to PENDING
- 'deadline' is optional and should be a datetime
- 'type' is a required text field
Change-Id: I2a7e0deec0ee2040e86400b500bb0efd8eade564
Closes-Bug: #1532843
Closes-Bug: #1533210
Due to importing modules rather than functions and decorators directly,
@abc.abstract and 'raise NotImplementedError' were added to the
.coveragerc file. Since abstract methods are not testable, this will
give us a more accurate representation of our coverage.
Change-Id: Id5ed5e1f5e142d10f41ad18d20228399226ec20d
Co-Authored-By: Jin Li <jl7351@att.com>
Closes-Bug: #1563717
We need to update sqlalchemy/api and sqlalchemy/models (and appropriate tests)
to support deleting audit templates and recreating them with the same names.
Change-Id: Icf54cf1ed989a3f2ad689e25be4474b16a3a3eb2
Related-Bug: #1510179
In some modules the global LOG is not used any more. And the import
of logging is not used. This patch removes the unused logging import
and LOG vars.
Change-Id: I794ee719d76f04e70154cf67f726152fbb1ba15a
Unicode type resource state is now handled in the same fashion as resource state specified by general string.
Change-Id: I35ffa09015283b51c935515436735aecbe83a9d6
Closes-Bug: #1565764
I removed the POST, PATCH and DELETE verbs from the actions
controller as they should only be modified internally.
Change-Id: Ia72484249240f829423056f66c5c0f9632d02106
Closes-Bug: #1533281
Added extra information regarding the plugin mechanism for:
action, strategy, and Watcher planner.
Change-Id: I9a7523282e229b83c16b06e3806ff795a0699c78
Closes-Bug: #1558470
The list of possible states for Action Plan objects was outdated, and
was updated to match the state machine diagram. A reference to the
state machines for Audits and Action Plans were added to the glossary,
and the descriptions of each state were moved to the sections containing
the state machines within the Architecture page.
Change-Id: I27043ad864c02fff50fb31868b27dc4b4897dbd4
Closes-Bug: #1558464
This patch adds a new load consolidation strategy based on a heuristic
algorithm which focuses on measured CPU utilization and tries to
minimize hosts which have too much or too little load.
A new goal "vm_workload_consolidation" was added which executes
the strategy "VM_WORKLOAD_CONSOLIDATION".
This work depends on the implemetation of the bug:
https://bugs.launchpad.net/watcher/+bug/1553124
Change-Id: Ide05bddb5c85a3df05b94658ee5bd98f32e554b0
Implements: blueprint basic-cloud-consolidation-integration
Watcher sample configuration file groups parameters from
various projects and the watcher project ones. This makes it
tricky to review updates on configuration parameters.
It is inconvenient for developer if the add/remove/change some
configuration options cause they need to take care about the
config.sample file.
The sample configuration file should be available into HTML doc.
This patchset:
. removes the file /etc/watcher/watcher.conf.sample
. adds an admin script tool to be able to built it, by using tox
. includes a new section 'Watcher sample configuration files' into
the doc source files
. uses sphinx extension oslo_config.sphinxgenconfig
Change-Id: If2180de3614663f9cbc5396961a8d2175e28e315
Closes-Bug: #1541734
In this patchset, I added a small subsection which highlights the fact
that actions are using Voluptuous Schemas to validate their input
parameters.
Change-Id: I96a6060cf167468e4a3f7c8d8cd78330a20572e3
Closes-Bug: #1545643
The variable vm_avg_cpu_util was renamed to host_avg_cpu_util for
clarity, as it was really referring to the host average cpu util.
Change-Id: I7aaef9eb2c8421d01715c86afa36ab67f2fd5f30
Closes-Bug: #1559113
In watcher/applier/workflow_engine/default.py, we are checking the
return value of action.execute(). As the "TODO" above indicates it
(line 118), we should get rid of this and only flag an action as
failed if an exception was raised during its execute(). We will
need to update the related unit tests.
Change-Id: Ia8ff7abd9994c3504e733ccd1d629cafe9d4b839
Closes-Bug: #1548383
This patchset add a new entry for the purge into the Watcher
documentation.
Change-Id: Ifb74b379bccd59ff736bf186bdaaf74de77098f1
Implements: blueprint db-purge-engine
This patchset implements the purge script as specified in its
related blueprint:
- The '--age-in-days' option allows to specify the number of
days before expiry
- The '--max-number' option allows us to specify a limit on the number
of objects to delete
- The '--audit-template' option allows you to only delete objects
related to the specified audit template UUID or name
- The '--dry-run' option to go through the purge procedure without
actually deleting anything
- The '--exclude-orphans' option which allows you to exclude from the
purge any object that does not have a parent (i.e. and audit without
a related audit template)
A prompt has been added to also propose to narrow down the number of
deletions to be below the specified limit.
Change-Id: I3ce83ab95277c109df67a6b5b920a878f6e59d3f
Implements: blueprint db-purge-engine
As a pre-requisite for being able to query the database for objects
that are expired, I need a way to express date comparison on the
'deleted_at' field which is common for every Watcher object. As they
are coming from mixins, I decided to implement these filters with a
syntax borrowed from the Django ORM where the field is suffixed by the
comparison operator you want to apply:
- The '__lt' suffix stands for 'less than'
- The '__lte' suffix stands for 'less than or equal to'
- The '__gt' suffix stands for 'greater than'
- The '__gte' suffix stands for 'greater than or equal to'
- The '__eq' suffix stands for 'equal to'
I also added a 'uuid' filter to later on be able to filter by uuid.
Partially Implements: blueprint db-purge-engine
Change-Id: I763f330c1b8ea8395990d2276b71e87f5b3f3ddc
When creating a new audit template, the verification of its goal
existence was previously done in watcher/objects/audit_template.py.
This check was moved to api/controllers/v1/audit_template.py, rather
than in the DAO class.
Change-Id: I6efb0657f64c46a56914a946ec78013b9e47331b
Closes-Bug: #1536191
In watcher/metrics_engine/cluster_history/api.py,
we can find the BaseClusterHistory abstract base class.
To follow the same naming convention observed throughout the rest
of the project, I renamed watcher/metrics_engine/cluster_history/api.py
to watcher/metrics_engine/cluster_history/base.py
Change-Id: If18f8db7f0982e47c1998a469c54952670c262f5
Closes-Bug: #1548398
There are translations that are missing from watcher.pot.
This patchset includes them.
Change-Id: Ia418066b5653b4c81885d3eb150613ba357f9b7b
Related-Bug: #1510189
Fetched information of total disk capacity from nova and added a new
resource 'disk_capacity' to NovaClusterModelCollector cluster. Also a
new resource type 'disk_capacity' was added to ResourceType.
https://bugs.launchpad.net/watcher/+bug/1553124
Change-Id: I85750f25c6d2693432da8e5e3a3d0861320f4787
Closes-Bug: #1553124
This documentation is a pre-requisite to all plugin documentation
as it guides you through the creation of a project from scratch
instead of simply forcusing on the implementation of the plugin
itself.
Change-Id: Id2e09b3667390ee6c4be42454c41f9d266fdfac2
Related-Bug: #1534639
Related-Bug: #1533739
Related-Bug: #1533740
In coverage target in tox.ini, there is the following argument:
--omit="watcher/tests/*"
However, in .coveragerc, the tests are also omitted,
according to line 4 in .coveragerc:
omit = watcher/tests/*
So the watcher/tests/* directory is omitted twice. As it can be
seen in other modules (e.g.: swift, nova) omitting only in
.coveragerc should be enough.
Change-Id: I72951196a346fb73a90c998138fc91dd171432cd
Closes-Bug: #1552617
As we modified the way a strategy gets implemented in
blueprint watcher-add-actions-via-conf, this patchset updates the
documentation regarding the implementation of a strategy plugin.
Change-Id: I517455bc34623feff704956ce30ed545a0e1014b
Closes-Bug: #1533740
This documentation describes step-by-step the process for implementing
a new planner in Watcher.
Change-Id: I8addba53de69be93730924a58107687020c19c74
Closes-Bug: #1533739
We provide a list of Watcher related projects
on the generated doc. This commit add links to
the Watcher dashboard project in various locations.
Change-Id: I1993cd5a11d3fcc8ca2c40b1779700359adab4ea
This documentation describes step-by-step the process for implementing
a new action in Watcher.
Change-Id: I978b81cdf9ac6dcf43eb3ecbb79ab64ae4fd6f72
Closes-Bug: #1534639
In several places, assertEqual is used the following way:
assertEqual(observed, expected)
However, the correct way to use assertEqual is:
assertEqual(expected, observed)
Change-Id: I5a7442f4adf98bf7bc73cef1d17d20da39d9a7f8
Closes-Bug: #1551861
The DevStack documentation should provide basic steps for setting up
Watcher with DevStack assuming the reader has no previous knowledge of
DevStack.
Change-Id: I830b1d9accb0e65bba73944697cba9c53ac3263e
Closes-Bug: #1538291
Watcher applier should be able to live migrate instances on any storage
type. To do this watcher will catch error 400 returned from nova if we
try to live migrate instance which is not on shared storage and live
migrate instance using block_migrate.
Added unit tests, changed action in watcher applier.
Closes-bug: #1549307
Change-Id: I97e583c9b4a0bb9daa1d39e6d652d6474a5aaeb1
The Watcher Tempest tests are only mentioned inside a README.rst.
They are now part of the main documentation.
Change-Id: Ieca85dc7f7307b45e4b99af4a4600a8c2d2b59d7
Closes-Bug: #1536993
This patchset introduces a new custom directive called 'drivers-doc'
which loads all available drivers under a given namespace and import
their respective docstring into the .rst document.
This patchset also contains some modification/addition to the
docstring of these drivers to make the final document complete.
Change-Id: Ib3df59fa45cea9d11d20fb73a5f0f1d564135bca
Closes-Bug: #1536218
Closes-Bug: #1536735
"TRIGGERED" is not the correct word to use to describe the action
state, we should use "PENDING" instead.
Change-Id: If24979cdb916523861324f7bcc024e2f1fc28b05
Closes-Bug: #1548377
Removed an extra underscore in the
get_audit_template_by__name method name in
watcher/db/api.py
Change-Id: I2687858ff4510c626c4dd2e2e9a5701405b5da55
Closes-Bug: #1548765
In watcher/tests/__init__.py has a totally unused,
misleading class so I removed it, as it is never used.
Change-Id: Ib878252453489eb3e1b1ff06f4d6b5e2b0726be5
Closes-Bug: #1549920
The primitive ChangeNovaServiceState allows us to change the state of
the nova-compute by calling nova api. The state of a nova-compute can
be ENABLED or DISABLED, however in the current implementation we use
OFFLINE and ONLINE.
Update the code to use ENABLED or DISABLED.
Change-Id: If3d9726bc5ae980b66c7fd4c5b7986f89d8bc690
Closes-Bug: #1523891
This commit only applies to documentation, state name
must be updated in code accordingly.
Change-Id: I027fd55d968c12992d800de3657543be417c71b0
Related-Bug: #1548377
When I used `tox -e debug <test-name>` to test a case, the case did not
run properly and there is an error like this:
ImportError: Start directory is not importable: './python-watcher/tests'
If we use '-t wartcher/tests' to point out the test path, the case will
run properly under debug.
So we'd better add this start directory for oslo_debug_helper.
Change-Id: I04d9937f72a95f8f045129af08df0cd0d0870d39
In watcher/api/controllers/v1/utils.py, in
the validate_sort_dir method has a return statement,
however the return value is exactly the parameter's value.
This is misleading, so I removed it.
Change-Id: I18c5c7853a5afedac88431347712a4348c9fd5dd
Closes-Bug: #1546917
In watcher/common/ceilometer_help.py:129,
there is an unused parameter, called limit:
def get_last_sample_values(self, resource_id, meter_name, limit=1):
In the next line, there is a method call which can take this
currently unused 'limit' parameter. Probably, passing this 'limit'
parameter to the query was originally intended, however it wasn't
added to the query call.
Closes-Bug: #1541415
Change-Id: I025070c6004243d6b8a6ea7a1d83081480c4148b
Although it was proposed via python-watcherclient, the feature was
not implemented on the Watcher API.
As the notion of host aggregate is currently unused in Watcher,
decision was made to only implement the filtering of goal within
the Watcher API whilst removing the host_aggregate filter from the
Watcher client.
Thus, this patchset adds this missing functionality by adding the
'goal' parameter to the API.
Change-Id: I54d248f7e470249c6412650ddf50a3e3631d2a09
Related-Bug: #1510189
Renamed many of the variables and method parameters
in the strategy implementations to make the names
more meaningful. Also changed the abstract method
signature in base.py to reflect these changes.
Closes-Bug: #1541615
Change-Id: Ibeba6c6ef6d5b70482930f387b05d5d650812355
As we had a low test coverage on actions, I added some more tests
with this patchset. This actually revealed a small bug (typo) in
"change_nova_service_state" which has been fixed in here.
Note that Tempest test also cover these action via the
basic_consolidation strategy.
Change-Id: I2d7116a6fdefee82ca254512a9cf50fc61e3c80e
Closes-Bug: #1523513
In some part in the code we import objects.
In the Openstack style guidelines they recommand
to import only modules.
We need to fix that.
Change-Id: I4bfee2b94d101940d615f78f9bebb83310ed90ba
Partial-Bug:1543101
We want a simplest way to validate the input parameters of an
Action through a schema.
APIImpact
DocImpact
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: I139775f467fe7778c7354b0cfacf796fc27ffcb2
Via Sean Dague in the mailing list:
In Newton, the KEYSTONE_CATALOG_BACKEND variable will be removed.
This commit removes the if check and variable from watcher
Change-Id: I73c5dd25feff9ba9824c267a8817a49e4ad3a06a
Closes-Bug: #1544433
In watcher/tests/objects/test_objects.py, there is a class
called "_TestObject(object)" which is probably an older test class.
This test class runs never (as the name starts with "_"
and inherits from object), and has some really old test,
like test_orphaned_object method, which is testing an exception
that doesn't exist at the current codebase.
Change-Id: I7559a004e8c136a206fc1cf7ac330c7d4157f94f
Closes-Bug: #1544685
When running our Tempest tests, we are now cleaning up all the
objects we have created within the WatcherDB via a soft_delete.
Partially Implements: blueprint deletion-of-actions-plan
Change-Id: Ibdcfd2be37094377d09ad77d5c20298ee2baa4d0
A problem was found during manual integration tests which were failing
because we couldn't instantiate the ceilometer client when trying to
execute an action plan using the 'basic_consolidation' strategy.
This patchset fixes the problem with an update of the related tests
Change-Id: I2b1f1dcc16fd8dfbf508c4d5661c1fce194254e4
Closes-Bug: #1544652
In order to accept the requirements contract defined in
openstack/requirements, we need to sync with the master branch of that
project's global-requirements.txt and test-requirements.txt files.
Most of the changes are just version changes and nothing major. The
only major change is:
1) The twine dependency is removed, and therefore the pypi tox
environment was also removed.
Change-Id: Idbe9e73ddc5a34ac49aa6f6eff0779d46a75f583
Closes-Bug: #1533282
When a user deletes an action plan, we now delete all related actions.
Partially Implements: blueprint deletion-of-actions-plan
Change-Id: I5d519c38458741be78591cbec04dbd410a6dc14b
Watcher uses now auth_type 'password' plugin for authentication.
Configuration related to credentials used to validate and apply
for a token has been updated.
Change-Id: If71bb908741130cb01d5d1525a12cf9a68b58a58
Closes-Bug: #1541296
Change I6c43eba941022a88851a199b56a6c20f017b9e71 seemed to have remove
most references to the SERVERS_CONSOLIDATION goal. Since this goal does
not currently exist in the actual code and all usages of it are for
samples or for tests, it is replaced with the DUMMY goal to avoid
confusion.
Change-Id: I4d2240d3b22c42ebf4e6120e2cd7677ec49d8e98
Closes-Bug: #1538388
The OpenStackClients class provides a convenient way to create and
cache client instances. The idea behind this code comes from Magnum
[0].
The OpenStackClients class will act as the manager of other project's
clients, providing an easy way to fetch instances of said clients. This
will allow the clients to be cached.
An instance of OpenStackClients is created for every call that comes
into the decision engine and the applier, using the request context to
pass needed (domain id) parameters to get a Keystone session. This
instance should be shared as much as possible to avoid additional
unneccessary connections to the other services.
This class will also allow for the version of each client to be
configurable via the watcher.conf file.
The method by which a Keystone session is also changed to use the
keystoneauth1.loading library. In order to avoid DuplicateOptErrors
with the keystone_authtoken group used for the keystonemiddleware in the
API code, a new conf group named "watcher_clients_auth" is created. A
typical configuration using a password authentication scheme will look
like:
[watcher_clients_auth]
auth_type = password
auth_url = http://<server-ip>:<port>
username = <username>
password = <password>
project_domain_id = default
user_domain_id = default
[0]: https://github.com/openstack/magnum/blob/master/magnum/common/clients.py
DocImpact
Change-Id: Iab9d0b304099686da2e9e2b19e8b1de4332ff378
Implements: blueprint external-api-versioning
Closes-Bug: #1530790
Closes-Bug: #1539670
Closes-Bug: #1522774
As of now we only have a single scenario which creates and
successfully executes the DUMMY goal.
This patchset adds a new scenario which creates and executes the
BASIC_CONSOLIDATION goal mapped to the 'basic' (sercon) strategy.
The documentation has also been updated to take into account the
multinode configuration.
Change-Id: Ie246aed288ade56a8fe9c0d9b08365d72e60ada1
Closes-Bug: #1538606
The current code will not work if WATCHER_CONF_DIR or
WATCHER_AUTH_CACHE_DIR already exist but are owned by a different user
such as root. Use install instead of mkdir to handle this scenario.
Change-Id: Ie582a4b393e898e007d73f31de490c4b77e40be3
Closes-Bug: #1539422
In the file watcher/objects/utils.py, on line 120,
there is an unused parameter:
def dt_deserializer(instance, val):
I removed that parameter, and modified the test.
Change-Id: Ibc7ab703d37d7f9248a84e41508820453c8954b7
Closes-Bug: #1540521
Currently self.client is referenced within MessagingCore,
but no definition is made in its constructor. Additionally
self.client is defined in children classes of MessagingCore.
This patchset defines self.client in the constructor of
MessagingCore and removes the redefinition in its children.
-self.client lazily loaded
Co-Authored-By: v-francoise <Vincent.FRANCOISE@b-com.com>
Change-Id: I14525a175bf1ebde3d2636024ad2f2219c79d6e1
Closes-Bug: #1521636
The InvalidParameterValue exception does not define a meaningful
msg_fmt. It is currently _("%(err)s"), which is the equivalent of
nothing and does not help with translation.
Replace InvalidParameterValue with Invalid exceptions.
Change-Id: If8b064e446cbc97e380127f360f262be9e8877a1
Closes-Bug: #1538398
Following the blueprint tempest-basic-set-up which implemented a first
batch of tests, this one adds a new set of API tests on actions.
I also added extra check on actions within the dummy strategy
scenario.
Change-Id: Ib9bf093d0ed457ecba32e8251c019d2cf5c98128
Closes-Bug: #1538074
Whenever trying to get the first action related to a given action
plan, we were getting back a 'null' value from the API even though
we knew there were actions to be linked to it in the DB.
So I fixed this issue and added a related unit test.
Change-Id: I1fa755f24fbf37ecd6ce2cc2396658fca8743a1c
Closes-Bug: #1538130
The msg_fmt of ActionPlanNotFound was missing an "_" which caused
errors upon trying to format it, so I fixed it.
Change-Id: I515c2097a563f809e319d2e57480fd340b878cef
Closes-Bug: #1538065
This patchset fixes the lack of field validation that are provided
by an API user.
Via a PATCH on /action_plans, the only field that can be modified
is now the 'state'. This field can only perform to the following
state transitions:
- RECOMMENDED --> TRIGGERED
- RECOMMENDED --> CANCELLED
- ONGOING --> CANCELLED
- TRIGGERED --> CANCELLED
The DELETED state can only be set using a DELETE request.
Closes-Bug: #1531106
Change-Id: I6669cbe63407f0bbb792fb2e2ce6b1e8a7365238
Workers has been introduced into Decision Engine and Applier.
We can now tune ithe number of workers (threads) used by the
Decision Engine and the Applier services, by updating the Watcher
configuration file.
Change-Id: I81523666e4373e7b1dd59b36daf19067fd61b48f
Closes-Bug: #1528132
This patchset adds CRUD tests on goals via the API.
Partially Implements: blueprint tempest-basic-set-up
Change-Id: Ief544e738d4530bcf981824803de059ae554a059
As the STARTING state was not clear on its meaning, I renamed it
to TRIGGERED.
Change-Id: I99cceeb57f3d7d42c1543b21fad88a6872bc4e55
Closes-Bug: #1533245
This patchset implements the first scenario for Watcher which does
the following actions:
- create an audit template with the dummy strategy
- run the audit to create an action plan
- get the action plan
- run the action plan
- get results and make sure it succeeded
Partially Implements: blueprint tempest-basic-set-up
Change-Id: Iee74ede0bd1bcd03e8938f2ec8c6884f99e7f99a
For a better doc QoS, we now use doc8 as part of the testing
procedure while removing the existing tests we had on doc formatting.
I also updated tox.ini to run doc8 as within 'pep8' and 'docs' venvs.
Change-Id: Ia0ad99541509f4c026e26d28c41ff0210b12a504
Closes-Bug: #1524228
Strategies can require, from Watcher, metrics collected on the IAAS.
Watcher uses only Ceilometer API to retrieve metrics.
Change the metrics database backend, add a new metric is not in
the scope of Watcher, but rather in the Ceilometer one. We add some
links to Ceilometer documentation about these topics.
Change-Id: If37c7df8e5852f5ecf94b4a9eb9c8c91fe6637eb
This patchset adds CRUD tests on Action Plans via the API.
Partially Implements: blueprint tempest-basic-set-up
Change-Id: I8ff3c3f0dbf7d301be2e3f401edf24bca44914bd
Change the title of the page.
Create a 'Getting Started' section.
Create a 'Plugins' section.
Add links to related source code repositories.
Fix bad link to watcher client installation doc.
Change-Id: Ie0548149751d53b5fca235da69798dd0d333b14c
Partial-Bug: #1535244
'Module Index' link points to HTML doc generated from source/api
RST files. These RST files are generated by pbr, by setting the
parameter 'autodoc_index_modules' to True.
Partial-Bug: #1535244
Change-Id: Ifceb5a140599d3968ea3d353b12c0bbe99d955e6
This patchset adds CRUD tests on Audits via the API.
Many of them are currently skipped as they revealed some underlying
bugs which are referenced on launchpad.
Partially Implements: blueprint tempest-basic-set-up
Change-Id: I5769f601f9d1cb94bb541959f94f0fa2e17d15c9
We must set up Tempest for Watcher
(http://docs.openstack.org/developer/tempest/configuration.html)
to run integration tests inside devstack environment.
This patchset is a refactoring of the stale Tempest tests to now
use the latest Tempest coding standards (like using plugins and
credentials factory).
This commit will have an effect on the doc as we need to integrate
Tempest in the Watcher documentation.
DocImpact
Partially Implements: blueprint tempest-basic-set-up
Change-Id: I7600ff8a28d524b56c7dd4903ac4d203634ae412
As we can see in the codebase, we have 3 "Status" enums which are
located at:
- watcher/objects/action.py
- watcher/objects/action_plan.py
- watcher/objects/audit.py
So I renamed them from "Status" to "State" to be consistent with
the DB schema.
Change-Id: If3d180c9daba6ae9083775ad6813aa4c21763dbf
Closes-Bug: #1522733
Added a new sequence diagram to explain showing the main steps
regarding Watcher usage:
- create a new audit template
- launch an audit and receive a recommended action plan
- launch the action plan
Change-Id: I1a0031b957d34908020be06870ed1d2772cd5af6
Closes-Bug: 1536731
The aim of this patchset is to integrate taskflow in
the Watcher Applier. Taskflow will help us a lot to make
Action Plan execution easy, consistent, scalable and reliable.
DocImpact
Partially implements: blueprint use-taskflow
Change-Id: I903d6509d74a61ad64e1506b8a7156e6e91abcfb
Closes-Bug: #1535326
Closes-Bug: #1531912
In the Audit controller (watcher/api/controllers/v1/audit.py),
we have an unused "audit_uuid" parameter in both get_all()
and detail()
Similarly, the Action Plan controller
(watcher/api/controllers/v1/action_plan.py) has an unused
"action_plan_uuid" parameter in both get_all() and detail().
I also removed them from the @wsme_pecan.wsexpose decorator
Change-Id: I7f1edfd44dd95c9768249650e19b16fcbe916b89
Closes-Bug: #1534615
The audit template UUID should be validated during the creation of an
audit. An HTTP 400 error is returned to the client if an invalid audit
template UUID is passed as part of the body when creating an audit.
APIImpact
Closes-Bug: #1510188
Change-Id: I0543d22751b77f6641ddef6a7f0f4acce61180fd
Added some data model diagrams, sequence diagrams and state machine
diagrams.
The state machine diagrams and sequence diagrams are built with
PlantUML whereas data model diagrams are built with Dia.
Also added some textual description with the sequence diagrams.
Change-Id: Iffbb47b0f2d12ce63eeaa1531a1bd1a790d69e79
Closes-Bug: #1531802
Same ID have been set to reference different RST blocks.
To avoid this, I added the prefix 'archi_' within ID referencing
architecture RST block.
Bad indentation warnings have been fixed.
Change-Id: I17f43f2f564ffd83fd5c345aed96fad06ee56b1d
Partial-Bug: #1522034
The execute() method is very large - almost 150 lines,
and McCabe's cyclomatic complexity 22.
(watcher/decision_engine/strategy/strategies/basic_consolidation.py)
I extracted some of the functionalities into helper functions
to reduce the length and complexity of execute().
http://openqa.sed.hu/dashboard/index/544838?did=1
Additionally it became more readable as well, without
changing its functionality.
Change-Id: I760929f56e258b87d7f1d4586bcc90665f1e0d8f
Closes-Bug: 1535729
The __init__ of these 3 classes are missing the super() call
watcher/watcher/api/controllers/v1/goal.py:113
watcher/api/controllers/v1/audit.py:217
watcher/watcher/api/controllers/v1/audit_template.py:166
Change-Id: I2fec1d5dac29a311f617ef141d5bf91f00b96219
Closes-Bug: #1535354
The BaseException class defined in exceptions.py is shadowing the
built-in BaseException class of the Python exception hierarchy, which
could potentially cause confusion.
This removes the BaseException definition and replaces it with the
existing WatcherException object. Instantiations of the
IllegalArgumentException are also changed to use the message kwarg.
Change-Id: I20abf135805c7a354924de8a5194b59fc040460a
Closes-Bug: #1535504
The custom exceptions were defining the message format string as the
'message' attribute, which was confusing as 'message' is also being used
as an input argument to the class. This combined with the fact that the
the 'message' attribute of Python's BaseException has been deprecated,
msg_fmt replaces message to hold the message format of the exception.
The root cause of the bug in question was caused by the __unicode__
method returning self.message, which did not have the kwargs substituted
into the actual message format. The fix is to use self.args[0] instead,
which will contain the message format with the kwargs substituted in
since that is passed to the super's (Exception) __init__ method. See
PEP 0352 for more information on how Exception and BaseException work.
The _cleanse_dict method is also removed as it is not used anywhere.
Change-Id: Ie8ac96afaecc732693a184d0e06e77c56ca8eeb9
Closes-Bug: #1535473
LOG.warn is deprecated. We were still used it some places.
So I replaced the LOG.warn method to LOG.warning, which is not
deprecated.
Change-Id: I9461cec569445ad6c40db9ce2feeeba1ef0af0e3
Closes-Bug: #1508442
In watcher, an audit generates a set of actions which
aims at achieving a given goal (lower energy consumption, ...).
It is possible to configure different strategies in order to achieve
each goal. Each strategy is written as a Python class which produces
a set of actions. Today, the set of possible actions is fixed for a
given version of Watcher and enables optimization algorithms to
include actions such as instance migration, changing hypervisor state,
changing power state (ACPI level, ...).
The objective of this patchset is to give the ability to load
the actions dynamically in order to apply the Action Plan.
DocImpact
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: Idf295b94dca549ac65d4636e8889c8ab2ecc0df6
Renamed diskInfo.py to disk_info.py and the associated test to
test_disk_info. Also changed the usage in the test to reflect
the name change.
Closes-bug: #1533189
Change-Id: Ice63cf8ea6cd4fcc770f88952cf784e5d46cca5c
In order to start translating we should gradually update the messages.
To fix this issue, we need to add _LI _LW _LE _LC in setup.cfg
Change-Id: Ia0bdea4dd3ecc12a6b804add9163926dc34bb581
Related-Bug: #1534164
As of now, the glossary defined in our documentation reflects the
current state of the codebase. In order to avoid any discrepancy
between the codebase and each definition, the objective here is to
gather both in a single place and link it into the rst documentation
via a custom directive.
Also re-aligned the requirements with liberty for doc.
Change-Id: I3089fd9f948b948115672f10937b1500b96ce180
Partial-Bug: #1526671
Replacing dict.iteritems()/.itervalues() with
six.iteritems(dict)/six.itervalues(dict) was preferred in the past,
but there was a discussion suggesting to avoid six for this.
ref:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I63b1e51597307862968f37803ffdbba63306d8c6
Each config option has limitation for type and value.
We make enforce_type=True to check whether we pass wrong type.
Also fixes a type error issue in test_db_manager.py.
Change-Id: I6e111e21588525d32b05eeba75b06583d4e605ed
Related-Bug: #1517839
The documentation for the DevStack plugin suggests changing the
libvirt_opts value of /etc/default/libvirt-bin to allow for live
migration between compute nodes. This changeset removes the incorrect
spaces surrounding the equals sign in the file, which would cause the
libvirt-bin service to crash.
Change-Id: Id073da5cdc84a900a398f260d7f7b40f027eb9d3
Closes-Bug: #1533761
In watcher, an audit generates a set of actions which
aims at achieving a given goal (lower energy consumption, ...).
It is possible to configure different strategies in order to achieve
each goal. Each strategy is written as a Python class which produces
a set of actions. Today, the set of possible actions is fixed for a
given version of Watcher and enables optimization algorithms to
include actions such as instance migration, changing hypervisor state,
changing power state (ACPI level, ...).
This patchset propose a generic and extensible way to describe
the actions and his parameters that we want to add to Action Plan.
It also remove the static actions because they are now deprecated.
The documentation regarding strategy plugin need to be
updated (plugins.rst).
DocImpact
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: I3d641080e8ad89786abca79a942c8deb2d53355b
In watcher, an audit generates a set of actions which
aims at achieving a given goal (lower energy consumption, ...).
It is possible to configure different strategies in order to achieve
each goal. Each strategy is written as a Python class which produces
a set of actions. Today, the set of possible actions is fixed for a
given version of Watcher and enables optimization algorithms to
include actions such as instance migration, changing hypervisor state,
changing power state (ACPI level, ...).
The objective of this patchset is to give the ability to extend the
default set of planner algorithms currently available in Watcher
using Stevedore.
The doc need to explain how create a new planner.
DocImpact
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: I2fd73f8c4a457ee391d764a7a3f494deecd2634f
In watcher, an audit generates a set of actions which
aims at achieving a given goal (lower energy consumption, ...).
It is possible to configure different strategies in order to achieve
each goal. Each strategy is written as a Python class which produces
a set of actions. Today, the set of possible actions is fixed for a
given version of Watcher and enables optimization algorithms to
include actions such as instance migration, changing hypervisor state,
changing power state (ACPI level, ...).
This patchset add a common generic dynamic loader for plugins,
such as for custom Actions, Strategies, Planners, etc.
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: I59d031b93865fff2540e3973921e1bdafa95f88e
In watcher, an audit generates a set of actions which
aims at achieving a given goal (lower energy consumption, ...).
It is possible to configure different strategies in order to achieve
each goal. Each strategy is written as a Python class which produces
a set of actions. Today, the set of possible actions is fixed for a
given version of Watcher and enables optimization algorithms to
include actions such as instance migration, changing hypervisor state,
changing power state (ACPI level, ...).
This patchset add the possibility to store several parameters
for an Action.
The parameters store info that the custom Action needs.
The parameters provided as tuples with the following fields:
(parameter_name, parameter_type).
It remove also the deprecated attributes
(src,dst,description)
APIImpact
Partially implements: blueprint watcher-add-actions-via-conf
Change-Id: Ic6727341822f8ac62f212d337814b2dca76044e3
In the section "WATCHER_GOALS_OPTS" the dict option "goals"
is not mandatory. However it should be.
Change-Id: I2e0770cf7787fed449c012bc45462e3138992ebf
Closes-Bug: #1531116
The usage of assertEqual(True/False, ***) should be changed
to a meaningful format of assertTrue/False(***).
Change-Id: Id708a94ac461adf021893a05796163bd2ced153c
Closes-Bug:#1512207
Use DevStack's plugin model to allow for developers to easily set up
the Watcher services within DevStack.
Provides documentation on how to use the plugin as well as documentation
on how to configure a multi-node DevStack installation to use. Also
provides an example local.conf for both the controller and compute
nodes.
Implements blueprint devstack-plugin
Change-Id: Ide663813f7a49d645877a21a0d1914be3218385e
Some Python class and packages need to be renamed
for a better compliance with the shared terminology
which provides a better understanding of Watcher objects
and components by every contributor.
Partially implements: blueprint glossary-related-refactoring
Change-Id: Ie0e33562f5e990c264a50ab3f533cfa62eac1d19
The nova wrapper2 just uses the same credential to create another
nova client session with the same capability. And it is hardcode
for keystone API v3.
Change-Id: I52b11a9b48ce2bb37a7872e2335ac3bae3f742c7
Closes-Bug: #1528142
2015-12-22 14:57:58 +08:00
891 changed files with 85198 additions and 15883 deletions
The list of efficacy indicators associated to this action plan.
in:body
required:false
type:array
actionplan_global_efficacy:
description:|
The global efficacy of this action plan.
in:body
required:false
type:array
actionplan_hostname:
description:|
Hostname the actionplan is running on
in:body
required:false
type:string
actionplan_state:
description:|
State of this action plan. To get more information about states and
action plan's lifecycle, visit `Action Plan State Machine page <https://docs.openstack.org/watcher/latest/architecture.html#action-plan-state-machine>`_.
in:body
required:false
type:string
# Audit
audit_autotrigger:
description:|
Auto execute action plan once audit is succeeded.
in:body
required:false
type:boolean
audit_endtime_req:
description:|
The local time after which audit can't be executed.
It will be converted to UTC time by Watcher.
in:body
required:false
type:string
min_version:1.1
audit_endtime_resp:
description:|
The UTC time after which audit can't be executed.
in:body
required:false
type:string
min_version:1.1
audit_force:
description:|
Launch audit even if action plan is ongoing.
in:body
required:false
type:boolean
min_version:1.2
audit_goal:
description:|
The UUID or name of the Goal.
in:body
required:false
type:string
audit_hostname:
description:|
Hostname the audit is running on
in:body
required:false
type:string
audit_interval:
description:|
Time interval between audit's execution.
Can be set either in seconds or cron syntax.
Should be defined only for CONTINUOUS audits.
in:body
required:false
type:string
audit_name:
description:|
Name of this audit.
in:body
required:false
type:string
audit_next_run_time:
description:|
The next time audit launch. Defined only for CONTINUOUS audits.
in:body
required:false
type:string
audit_parameters:
description:|
The strategy parameters for this audit.
in:body
required:false
type:JSON
audit_starttime_req:
description:|
The local time after which audit can be executed in accordance
with interval. It will be converted to UTC time by Watcher.
in:body
required:false
type:string
min_version:1.1
audit_starttime_resp:
description:|
The UTC time after which audit can be executed in accordance with interval.
in:body
required:false
type:string
min_version:1.1
audit_state:
description:|
State of this audit. To get more information about states and
audit's lifecycle, visit `Audit State Machine page <https://docs.openstack.org/watcher/latest/architecture.html#audit-state-machine>`_.
in:body
required:true
type:string
audit_strategy:
description:|
The UUID or name of the Strategy.
in:body
required:false
type:string
audit_type:
description:|
Type of this audit. Can only be either ONESHOT or CONTINUOUS.
in:body
required:true
type:string
# Audit Template
audittemplate_description:
description:|
Short description of the Audit Template.
in:body
required:false
type:string
audittemplate_goal:
description:|
The UUID or name of the Goal.
in:body
required:true
type:string
audittemplate_name:
description:|
The name of the Audit template.
in:body
required:true
type:string
audittemplate_scope:
description:|
Audit Scope.
in:body
required:false
type:JSON
audittemplate_strategy:
description:|
The UUID or name of the Strategy.
in:body
required:false
type:string
audittemplate_uuid:
description:|
The UUID of the Audit template.
in:body
required:true
type:string
created_at:
description:|
The date and time when the resource was created. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
deleted_at:
description:|
The date and time when the resource was deleted. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
# Goal
goal_display_name:
description:|
Localized name of the goal.
in:body
required:true
type:string
goal_efficacy_specification:
description:|
Efficacy specifications as result of stategy's execution.
in:body
required:true
type:array
goal_name:
description:|
Name of the goal.
in:body
required:true
type:string
goal_uuid:
description:|
Unique UUID for this goal.
in:body
required:true
type:string
links:
description:|
A list of relative links. Includes the self and bookmark links.
in:body
required:true
type:array
# Data Model Node
node_disk:
description:|
The Disk of the node(in GiB).
in:body
required:true
type:integer
node_disk_ratio:
description:|
The Disk Ratio of the node.
in:body
required:true
type:float
node_hostname:
description:|
The Host Name of the node.
in:body
required:true
type:string
node_memory:
description:|
The Memory of the node(in MiB).
in:body
required:true
type:integer
node_memory_ratio:
description:|
The Memory Ratio of the node.
in:body
required:true
type:float
node_state:
description:|
The State of the node. The value is up or down.
in:body
required:true
type:string
node_uuid:
description:|
The Unique UUID of the node.
in:body
required:true
type:string
node_vcpu_ratio:
description:|
The Vcpu ratio of the node.
in:body
required:true
type:float
node_vcpus:
description:|
The Vcpu of the node.
in:body
required:true
type:integer
# Scoring Engine
scoring_engine_description:
description:|
A human readable description of the Scoring Engine.
in:body
required:true
type:string
scoring_engine_metainfo:
description:|
A metadata associated with the scoring engine
in:body
required:true
type:string
scoring_engine_name:
description:|
The name of the scoring engine.
in:body
required:true
type:string
# Data Model Server
server_disk:
description:|
The Disk of the server.
in:body
required:true
type:integer
server_memory:
description:|
The Memory of server.
in:body
required:true
type:integer
server_name:
description:|
The Name of the server.
in:body
required:true
type:string
server_state:
description:|
The State of the server.
in:body
required:true
type:string
server_uuid:
description:|
The Unique UUID of the server.
in:body
required:true
type:string
server_vcpus:
description:|
The Vcpu of the server.
in:body
required:true
type:integer
# Service
service_host:
description:|
The Name of host where service is placed on.
in:body
required:true
type:string
service_id:
description:|
The ID of service.
in:body
required:true
type:integer
service_last_seen_up:
description:|
The Time when Watcher service sent latest heartbeat.
in:body
required:true
type:string
service_name:
description:|
The Name of service like ``watcher-applier``.
in:body
required:true
type:string
service_status:
description:|
The State of service. It can be either in ACTIVE or FAILED state.
in:body
required:true
type:string
# Strategy
strategy_check_comment:
description:|
Requirement comment.
in:body
required:true
type:string
strategy_check_mandatory:
description:|
Whether this requirement mandatory or not.
in:body
required:true
type:boolean
strategy_check_state:
description:|
State of requirement for Strategy.
in:body
required:true
type:string or JSON
strategy_check_type:
description:|
Type of requirement for Strategy.
in:body
required:true
type:string
strategy_display_name:
description:|
Localized name of the strategy.
in:body
required:true
type:string
strategy_name:
description:|
Name of the strategy.
in:body
required:true
type:string
strategy_parameters_spec:
description:|
Parameters specifications for this strategy.
in:body
required:true
type:JSON
strategy_uuid:
description:|
Unique UUID for this strategy.
in:body
required:true
type:string
updated_at:
description:|
The date and time when the resource was updated. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
uuid:
description:|
The UUID for the resource.
in:body
required:true
type:string
# Version
version:
description:|
Versioning of this API response, eg. "1.1".
in:body
required:true
type:string
version_description:
description:|
Descriptive text about the Watcher service.
in:body
required:true
type:string
version_id:
description:|
Major API version, eg, "v1"
in:body
required:true
type:string
versions:
description:|
Array of information about currently supported versions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
