when directly using the `curl` command to create audit template,
strategy name can be accepted.
Closes-Bug: #1884174
Change-Id: I7c0ca760a7fa414faca03c5293df34a84aad6fac
(cherry picked from commit 3f7a508a2e)
if scope is None, don't create data model
Change-Id: Icf611966c9b0a3882615d778ee6c72a8da73841d
Closed-Bug: #1881920
(cherry picked from commit 9f0138e1cf)
When querying data from datasource, it's possible to miss some data.
In this case if we throw an exception, Audit will failed because of
the exception. We should remove the exception and give the decision
to the strategy.
Change-Id: I1b0e6b78b3bba4df9ba16e093b3910aab1de922e
Closes-Bug: #1847434
(cherry picked from commit f685bf62ab)
Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.
Until the requirements repository has as stable/train branch, tests will
continue to use the upper-constraints list on master.
Change-Id: Ia9e0181e1013445b62abd8128c189eafa83906db
Reason:
When there is a compute node but no virtual machine,
the command 'watcher datamodel list' should display
the information of the compute node instead of return None.
Change-Id: Id5ff7f08ac8a9883af9f0313785b756d813ed5a2
Closes-Bug: #1844948
The default planner can not create actions with right order,
The node_resouce_consolidation strategy needs to use its
own planner.
Partially Implements: blueprint node-resource-consolidation
Depends-on: I586e67f782e2965234826634ba3ff51681af4df8
Change-Id: I05b02905a3335a73b6926966de6331c632842293
Add a new pdf-docs environment to enable PDF build.
sphinxcontrib-svg2pdfconverter is used to handle SVG properly.
Change-Id: I1563579486da8912ba8a220bb08a5331e7df910b
It should've been "watcher" instead of "python-watcher" as the
config files are expected to be in /etc/watcher/. Though this is
unlikely to cause problems as this patch corrected the default
config dir.
https://review.opendev.org/#/c/658348/
Nevertheless, we should be using the correct name.
Change-Id: If6b58133eecf2fcc37e11d8c45eaa58f238ea2a8
This component is responsible for selecting an appropriate Planner based
on predefined property value passed to concrete Strategy.
Change-Id: I86de95886df5d7e9558512569601e9ea3babb0e9
Implements: bp watcher-planner-selector
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
This strategy is used to centralize VMs to as few nodes as possible
by VM migration. User can set a input parameter to decide how to
select the destination node.
Implements: blueprint node-resource-consolidation
Closes-Bug: #1843016
Change-Id: I104c864d532c2092f5dc6f0c8f756ebeae12f09e
Many strategies need get node used or free resources, we define
two new method for the purpose in ModelRoot class.
Change-Id: I8cb41fd560dbac9a78d25bfdba51799533db83c2
1. Add datamodel api and policy_enfoce file.
2. Add related unittest for data_model api and policy.
Partially Implements:blueprint show-datamodel-api
Change-Id: I1654685d8cf04db5dd132d43a8640ddf91893cad
1. Add datamodel list endpoint and rpc process.
2. Add datamodel list parased and return.
3. Add related unittest.
Partially Implements:blueprint show-datamodel-api
Change-Id: I758b7ca2bc3d8d596d3457277744336c6629bc4e
The new bp need to get audit type from audit,
so we need to add an audit parameter to do_execute
Partially Implements: blueprint node-resource-consolidation
Change-Id: Ia979781b32202c1821aa1cb91d24253fe6d7bd2d
watcher-tempest-strategies includes all strategies tempest,
we add it and remove all other individual strategy tempest.
Depends-on: I3e45d4a66a6e1bf55499def8550da38ddf01b638
Change-Id: I182bf0ddc528099f5115098b825e9bddae3b187a
As part of Train community goal 'Support IPv6-Only Deployments and Testing'[1],
Tempest has defined the base job 'devstack-tempest-ipv6' which will
deploy services on IPv6.
This commit adds the new job 'watcher-tempest-functional-ipv6-only'
run on gate which is derived from 'devstack-tempest-ipv6'.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- 'watcher-tempest-functional-ipv6-only' will run the tests.
Story: #2005477
Task: #35939
[1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
Change-Id: I42b7e5ff5fd64a21bdb8a32f319759a18c173601
The fields disk and disk_capacity have the same value,
we just need one, so remove disk_capacity field.
Partially Implements: blueprint improve-compute-data-model
Change-Id: If3d385c5e61713bbdc85e22f10cd75e161ff79f0
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I9fe58603692a9850e86a2c36ad7a31c473070100
For Compute node, when calculating resource(VCPU, memory and disk)
capacity, we need to consider reserved resource and allocation ratio.
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I70257dd5fb342a67a3ffda1055eddc54b8360ca3
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I2bb230b5f5a573fb3045261dfdee73f1a8434e0d
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I4f041ad25353d575c276fce87fe13c5e6705754f
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: Id113b4c19792946329e9ff448bfe636cc8eca057
For Compute node, we can use the new property to calculate
resource(VCPU, memory and disk).
Partially Implements: blueprint improve-compute-data-model
Depends-on: I3f9a3279a26f3df444117d9265e74cca57b38d6e
Change-Id: I7872265b2378e5dc37aa2e086ff1f7fb9071db0b
The node resource(vcpu, memory and disk) used infomation need
to change when creating or deleting instances. Now Placement do
not send notifications, so there is not a good way to capture
the change. We remove these fields and leave the process to strategy.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I3f9a3279a26f3df444117d9265e74cca57b38d6e
This error was discovered by tool coverity. If we don't
initialize this var src_extra_specs, line 225 may sometimes
raise an error.
Change-Id: I992b56b64d56f35c8355b22707c3db5112964b31
The code associated with virtual has been removed before,
and the relevant comments should be removed here.
Change-Id: I7104c1a6752ad0b8c9837a643e51b0a13194a81b
Resource(VCPU, memory and disk) capacity need to be calculated
through formula: capacity = (total-reserved)*ratio.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I15ca66dd2c3a21c5acfebf6f04fa6601aff7918f
We have some new fields(vcpus_ratio, vcpus_used, ...)
in the Watcher ComputeNode. During the process of updating
data model by notifications, we need to get data from
placement.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I10587e93bb3e7be6af78bb3a50509d82d8228f78
The node.free_disk_gb does not take allocation ratios used
for overcommit into account so this value may be negative.
We do not need this field and plan to set disk to total disk
capacity and then remove disk_capacity.
Partially Implements: blueprint improve-compute-data-model
Change-Id: I72c4490f5a8d0fbd1039f70ff20f07b743b6bb2d
check if the resource class(VCPU, memory, disk) in the return
dictionary. If they are, don't need to use dict.get() with a
default value because the parameters are required.
Partially Implements: blueprint improve-compute-data-model
Change-Id: Icb8c672d0e87e6e5f030a2222f928d1bbd069e3c
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: I4101eced9c4bd26741f760e5651204f5d2dfea0f
The fields(vcpus, memory and disk_capacity) in the Watcher ComputeNode
do not take allocation ratios used for overcommit into account so there
may be disparity between this and the used count.
This patch added some new fields to solve this problem.
Partially Implements: blueprint improve-compute-data-model
Change-Id: Id33496f368fb23cb8e744c7e8451e1cd1397866b
Add call_retry method for ModelBuilder classes along with configuration
options. This allows ModelBuilder classes to reattempt any failed calls
to external services such as Nova or Ironic.
Change-Id: Ided697adebed957e5ff13b4c6b5b06c816f81c4a
Actually list_opts() return a list like[1], So we don't need to
convert list to dict and then convert to list[2].
The reason why we need to convert it before is to put together
the same group of configuration objects, but we don't need it
actually.
Now, the list_opts()'s result like this[3].
Reference:
[1]. [(Group1,[cfgObj1,cfgObj2....]),(Group2,[cfgObj3,cfgObj3....])..]
[2]. 375ae32fad/watcher/conf/opts.py (L51-L52)
[3]. [(Group1,[cfgObj1]),(Group1,[cfgObj2]),(Group2,[cfgObj3,cfgObj3....])..]
Change-Id: I50fcc5f812be42038852662639fb10c6dd2f6f72
This lets all the ModelBuilder classes use one baseclass and forces
ClusterDataModelCollector's to pass the scope.
The scopes are still unused in the case of Ironic and Cinder.
The idea is to do several follow ups to this and in the end have a
similar method to query_retry in the datasources baseclass.
Change-Id: Ibbdedd3087fef5298d7f4c9d3abdba05d1fbb2f0
The datasources are only used by the decision_engine, however, they
are placed in a directory one level higher. This patch moves the
datasources code into the decision_engine folder.
Change-Id: Ia54531fb899b79a59bb77adea079ff27c0d518fa
We want to set the value of uuid field of Watcher ComputeNode
to hypversion id(as uuid). So we need to get hypervisor
information by uuid.
Change-Id: I752fbfa560313e28e87d83e46431c283b4db4f23
Related-Bug: #1835192
This error is caused because the condition "is not '':" is not always
true. Sometimes self.aggregation_method['node'] is u'' instead of ''.
This patch ensures that in both cases the behavior is the same.
Change-Id: I7453678cc76892ebeacca23c3501a10a08725d1d
Closes-bug: #1836195
This patch does two things:
1. replace instance's human_id with name.
2. remove ComputeNode human_id.
Now name field in Watcher Compute Data Model is availible.
Use name is better than human_id. For the reason, please see[1].
[1]. https://bugs.launchpad.net/watcher/+bug/1833665
Change-Id: I04f40e7d2a2bda48e9a362f9d0b23f449c40324e
aggregate list and availability_zone list may return ironic type
compute nodes. When building compute data model we should check
the hypervisor_type and remove ironic compute nodes.
Change-Id: Idf404c104c30368baf95ef7d05ad8fc3e7adca38
Related-Bug: #1835183
New datasource to retrieve metrics that can be configured in a
flexible way depending on the deployment. Current implemenation only
works with InfluxDB. Slight changes to datasource manager were
necessary because grafana metric_map can only be built at runtime.
The yaml configuration file can still be used to define metrics
but will require that five different attributes are specified per
metric.
Specific databases accesible through grafana can be defined by
creating 'translators' for these specific databases. This patch
introduces a base class for these translators and their methods.
In addition the first translator specific for InfluxDB is
created.
Depends-on: I68475883529610e514aa82f1881105ab0cf24ec3
Depends-on: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Implements: blueprint grafana-proxy-datasource
Change-Id: Ib12b6a7882703e84a27c301e821c1a034b192508
We want to set the value of uuid field of Watcher ComputeNode
to hypversion id(as uuid). We need a method to get compute
node by name.
Change-Id: I0975500f359de92b6d6fdea2e01614cf0ba73f05
Related-Bug: #1835192
The problem is that watcher is passing limit=-1 to novaclient when
listing servers which will always make at least two API calls to be
sure it's done paging:
https://github.com/openstack/python-novaclient/blob/13.0.1/novaclient/v2/servers.py#L896
If we can determine before we list servers that there are only a
certain number where the number of servers is less than 1000. For
example: 4, we should just pass the limit=len(servers) to novaclient
and avoid the second call for paging which takes extra time and
yields no results.
Change-Id: I797ad934a0f8496dbcbf65798e28b0443f238137
Closes-Bug: #1834679
openstack hypervisor list contains ironic nodes. we should
filter out baremetal nodes when get compute node list.
Change-Id: I4ab3e1a63dc6f61cdc3e99fa2cae749a711459cc
Closes-Bug: #1835183
According to https://review.opendev.org/#/c/251791/,
watcher_messaging group and notifier_driver option
were deprecated.
Change-Id: I2cd114060d1960f77dfa8f4fe0a6d0fc05de5d4c
This is the releasenote for the new grafana datasource it refers to
the documentation on configuring grafana.
Depends-on: Ib12b6a7882703e84a27c301e821c1a034b192508
Change-Id: Icb3939d772f06ad2d66eeba9a59fa8b60822ece0
This is a follow-up to: https://review.opendev.org/#/c/666897/
and makes sure titles and help information get rendered in
the configuration documentation and configuration samples.
The options for the placement_client group are already changed
and left untouched as a result. The changes to grafana_client
are already done in another patch and also untouched.
Change-Id: Ia33cd4576e4b55e651f3f3779a01f2867126138d
"self.assertTrue(action.state, objects.action.State.SUCCEEDED)"
and "self.assertTrue(action.state, objects.action.State.FAILED)"
should use assertEqual.
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
Change-Id: I8e28d651938ca6ed8d12e8a6f5ecf775cf01a39c
This patch implements uWSGI support for Watcher API service.
Because mod_wsgi is deprecated, using uwsgi to replace of mod_wsgi.
Most of Openstack projects have finished it.
Closes-Bug: #1834392
Change-Id: I3fad8d30a15aba493fb91da9337c2515ddea5167
Nova changed the default notification_format from "both" to
"unversioned" in Train [1]. Without configuring nova in the
grenade job we are not testing the nova versioned notification
handler code during upgrades.
Note that grenade only runs stack.sh on the base (old) side so
this change has to depend on a devstack stable/stein change to
add the NOVA_NOTIFICATION_FORMAT variable that we override.
Closes-Bug: #1831917
Depends-On: Ied9d50b07c368d5c2be658c744f340a8d1ee41e0
[1] https://review.opendev.org/603079/
Change-Id: I94c2d14477da185310e0fec596a1ad6436b802f1
This improves the documentation on configuration parameters for the
Grafana datasource.
Follow-up: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Depends-on: I5d1d3129b5d225f0f2fc86d149c046f9aab94d47
Change-Id: Ifd8be7491669c429482d880fdf0219be5ef03163
Nova used to emit versioned and unversioned notiifcations
by default but that changed in https://review.opendev.org/603079/
so now nova emits only unversioned notifications by default.
Watcher listens for versioned notifications so we need to configure
nova to emit both versioned (for Watcher) and unversioned
(for Ceilometer) notifications explicitly.
This adds an override-defaults file so devstack will load up
the nova devstack variable to set the notification_format before
importing and stacking the nova lib script.
Note that this only fixes the non-grenade CI jobs since grenade
requires separate handling for overriding defaults which is proving
hard to do and will be addressed in a separate change.
Partial-Bug: #1831917
Change-Id: I7e441608b38338eecd80e663ed3abe66a89e504f
In the process of handling instance_created.end,
there is a KeyError exception output log. This is because
invoking get_instance_by_uuid before creating the instance
in the data model.
During the review of https://review.opendev.org/#/c/663489/,
reviewers think that it's better to remove the KeyError exception.
This patche seperates the process of instance_created.end from
other Nova notifications and removes the call of get_instance_by_uuid.
Change-Id: Ie9e2d4f5b32ee7a5b52bbcd50abfa81dcabab7bb
Ceilometer removed cpu_util metric in [1].
Another metric compute.node.cpu.percent need to set
compute_monitors option to cpu.virt_driver in the
nova.conf, we should remind user about these.
[1]: https://review.opendev.org/#/c/580709/
Change-Id: I89306ef7c26fa2927945bd4f3ee88b670511d147
This implements the configuration parameters to implement
Grafana as a datasource including the influxdb translator
Change-Id: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Partially-implements: blueprint grafana-proxy-datasource
Replaces the NoSuchMetric exception that was replaced. The exception
is replaced with MetricNotAvailable and test cases are added to prevent
regression.
The changes in the exceptions were introduced in:
https://review.opendev.org/#/c/658127/
Change-Id: Id0f872e916aaa5dec59ed1ae6c0f653553b3fe46
In get_node_by_instance_uuid, an exception ComputeNodeNotFound
will be thrown if can't find a node through instance uuid.
But the exception information replaces the node name with
instance uuid, which is misleading, so we define a new exception.
Closes-Bug: #1832156
Change-Id: Ic6c44ae44da7c3b9a1c20e9b24a036063af266ba
Moves the query_retry method into the baseclass and makes the query
retry and timeout options part of the watcher_datasources config group.
This makes the query_retry behavior uniform across all datasources.
A new baseclass method named query_retry_reset is added so datasources
can define operations to perform when recovering from a query error.
Test cases are added to verify the behavior of query_retry.
The query_max_retries and query_timeout config parameters are
deprecated in the gnocchi_client group and will be removed in a future
release.
Change-Id: I33e9dc2d1f5ba8f83fcf1488ff583ca5be5529cc
We should be starting from stable/stein on the "old" side
of grenade runs now. Rather than hard-code the branch, just
use the BASE_DEVSTACK_BRANCH variable.
Change-Id: I1b0406f870ed0ae5622cfa7421a6cca00d0f891c
When receiving Nova notification instance.create.end,
map instance to its node after adding instance to datamodel.
Related-Bug: #1832156
Change-Id: I6f39e8d935195c611f668f71590e1d9ff52ced0d
In Python, when we use @property, the method will be
decorated by property.
When we call method self.strategy.datasource_backend()[1],
Actually it did two things:
1. call self.strategy.datasource_backend()
2. according to the method's return value[2], call self._datasource_backend()
[1]. https://github.com/openstack/watcher/blob/bd8636f3f/watcher/tests/decision_engine/strategy/strategies/test_base.py#L87
[2]. https://github.com/openstack/watcher/blob/bd8636f3f/watcher/decision_engine/strategy/strategies/base.py#L368
But in this part, we just want it to perform the first step.
So we have to use self.strategy.datasource_backend instead of
self.strategy.datasource_backend()
The reason why the unittest does not report an error is
because the returned value is a mock object, and the second step
is executed without error, for example:
python -m unittest watcher.tests.decision_engine.strategy.strategies.test_base
(Pdb) x=self.strategy.datasource_backend
(Pdb) type(x)
<class 'mock.mock.MagicMock'>
(Pdb) x
<MagicMock name='DataSourceManager().get_backend()' id='139740418102608'>
(Pdb) x()
<MagicMock name='DataSourceManager().get_backend()()' id='139740410824976'>
(Pdb) self.strategy.datasource_backend()
<MagicMock name='DataSourceManager().get_backend()()' id='139740410824976'>
To make the tests more robust, the underlying backend function
is mocked to be not callable.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Change-Id: I3305d9afe8ed79e1dc3affe02ba067ac06cece42
This patch added Placement to Watcher
We plan to improve the data model and strategies in
the future specs.
Change-Id: I7141459eef66557cd5d525b5887bd2a381cdac3f
Implements: blueprint support-placement-api
This makes the ConfFixture extend the Config fixture from
oslo.config which handles cleanup for us. The module level
import_opt calls are also removed since they are no longer
needed.
Change-Id: I869e89c53284c8da45e0b1293f2d35011f5bfbf9
In the process of creating an instance, Nova will emit an
instance.update notification with 'building' state.
This will cause a KeyError exception because this instance
isn't in Watcher datamodel.
So we should ignore the notification instance.update with
'building' state.
Closes-Bug: #1832154
Change-Id: I950eec50d2cee38bd22c47a70ae6f88bbf049080
Now there are some errors when running apidoc,
actually we don't need apidoc, so remove it.
Closes-Bug: #1831515
Change-Id: I3b91a2c05ed62ae7bbd30a29e9db51d0e021410f
The get_compute_node_by_hostname method is given a
compute service hostname and then does two queries to
find the matching hypervisor (compute node) with details:
1. List hypervisors with details and find the one that
matches the given compute service hostname.
2. Using that node, search for hypervisors with the
matching hypervisor_hostname.
There are two issues here:
1. The first query is inefficient in that it has to list
all hypervisors in the deployment to try and match the
one with the compute service hostname client side.
2. The second query is a fuzzy match on the server side [1]
so even though we have matched on the node we want,
get_compute_node_by_name can still return more than
one hypervisor which will result in the helper method
raising ComputeNodeNotFound. Consider having compute
hosts with names compute1, compute10, compute11, compute100,
and so on. The fuzzy match on compute1 would return all of
those hypervisors.
For non-ironic nodes in nova, the compute service host and
hypervisor should be 1:1, meaning the hypervisor.service['host']
should be the same as hypervisor.hypervisor_hostname. Knowing
this, we can simplify the code to search just on the given
compute service hostname and if we get more than one result, it
is because of the fuzzy match and we can then do our client-side
filtering on the compute service hostname.
[1] https://github.com/openstack/nova/blob/d4f58f5eb/nova/db/sqlalchemy/api.py#L676
Change-Id: I84f387982f665d7cc11bffe8ec390cc7e7ed5278
The nova CDM builder code and notification handling
code had some inefficiencies when it came to looking
up a hypevisor to get details. The general pattern
used before was:
1. get the minimal hypervisor information by hypervisor_hostname
2. make another query to get the hypervisor details by id
In the notifications case, it was actually three calls because
the first is listing hyprvisors to filter client-side by service
host.
This change collapses 1 and 2 above into a single API call
to get the hypervisor by hypervisor_hostname with details
which will include the service (compute) host information
which is what get_compute_node_by_id() was being used for.
Now that nothing is using get_compute_node_by_id it is removed.
There is more work we could do in get_compute_node_by_hostname
if the compute API allowed filtering hypervisors by service
host so a TODO is left for that.
One final thing: the TODO in get_compute_node_by_hostname about
there being more than one hypervisor per compute service host
for vmware vcenter is not accurate - nova's vcenter driver
hasn't supported a host:node 1:M topology like that since the
Liberty release [1]. The only in-tree driver in nova that supports
1:M is the ironic baremetal driver, so the comment is updated.
[1] Ifc17c5049e3ed29c8dd130339207907b00433960
Depends-On: https://review.opendev.org/661785/
Change-Id: I5e0e88d7b2dd1a69117ab03e0e66851c687606da
Some of the methods for retrieving data about instances was placed
at the bottom of nova_helper instead of being close to the other
instance based methods.
Change-Id: I68475883529610e514aa82f1881105ab0cf24ec3
This does two things:
1. Rather than make an API call per server on the host,
get all of the servers in a single API call by
filtering on the host. The os-hypervisors API results
to use make this require a bit of refactoring since
get_compute_node_by_name does not have the service
entry in it and get_compute_node_by_id does not have the
servers entry in it. A TODO is added to clean that up
with a single call to os-hypervisors once we have the
support in python-novaclient.
2. Pulls get_node_by_uuid() out of the loop.
A test is added for the nova_helper get_instance_list method
since one did not exist before.
The fake compute node mocks in test_nova_cdmc_execute are
also cleaned up since, as noted above, get_compute_node_by_name
and get_compute_node_by_id don't both return all the details.
Change-Id: Ifd9f83c2f399d4c1765b0c520f4d5a62ad0f5fbd
Fix the list of required metrics from a datasource when testing the
existence of this metric in the metric map.
Change-Id: I19b7408a98893bc942c32edb09f1b3798ec8dc79
With change Id34938c7bb8a5ca934d997e52cac3b365414c006
we require nova API version 2.56 or greater so we can
remove the compatibliity check in the
watcher_non_live_migrate_instance method.
The _check_nova_api_version method is left in place
for future compability checks.
Change-Id: I69040fbc13b03d90b9687c0d11104d4a5bae51d3
The [nova_client]/api_version defaults to 2.56 since
change Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1. There
is compatibility code for that change but if 2.56 is
not available watcher_non_live_migrate_instance will
still fail if a destination host is used.
Since 2.56 has been available since the Queens version of
nova it should be reasonable to require at least that
version of nova is running for using Watcher.
This adds code which enforces the minimum version along
with a release note and "watcher-status upgrade check"
check method.
Note that it's kind of weird for watcher to have a config
option like nova_client.api_version since compute API
microversions are per API request even though novaclient
is constructed with the single configured version. It should
really be something the client (watcher in this case) determines
using version discovery and gracefully enables features if
the required nova API version is available, but that's a bigger
change.
Change-Id: Id34938c7bb8a5ca934d997e52cac3b365414c006
MetricNotAvailable and NoDatasourceAvailable allow to differentiate
between having no datasources configured and a required metric being
unavailable from the datasource. Both exceptions have comments so
that the use case is clear.
The input validation of the get_backend method in the datasource
manager is improved.
Additional logging information allows to identify which metric caused
the available datasource to be discarded.
Tests are updated to validate the correct functionality of the new
exceptions.
Change-Id: I512976cce2401dbcd249d42686b78843e111a0e7
Changes to the baseclass for datasources so strategies can be made
compatible with every datasource. Baseclass methods clearly describe
expected values and types for both parameters and for method returns.
query_retry has been added as base method since every current
datasource implements it.
Ceilometer is updated to work with the new baseclass. Several methods
which are not part of the baseclass and are not used by any strategies
are removed. The signature of these methods would have to be changed
to fit with the new base class while it would limit strategies to
only work with Ceilometer.
Gnocchi is updated to work with the new baseclass.
Gnocchi and Ceilometer will perform a transformation for the
host_airflow metric as it retrieves 1/10 th of the actual CFM
Monasca is updated to work with the new baseclass.
FakeMetrics for Gnocchi, Monasca and Ceilometer are updated to work
with the new method signatures of the baseclass.
FakeClusterAndMetrics for Ceilometer and Gnocchi are updated to work
with the new method signatures of the baseclass.
The strategies workload_balance, vm_workload_consolidation,
workload_stabilization, basic_consolidation, noisy_neighbour,
outlet_temp_control and uniform_airflow are updated to work with the
new datasource baseclass.
This patch will break compatibility with plugin strategies and
datasources due to the changes in signatures.
Depends-on: I7aa52a9b82f4aa849f2378d4d1c03453e45c0c78
Change-Id: Ie30ca3dbf01062cbb20d3be5d514ec6b5155cd7c
Implements: blueprint formal-datasource-interface
As a follow up to the recent test improvements for Ceilometer this
patch ensures that the same test pattern is used for Gnocchi and
Monasca as well. This ensures that the mocked functions will be called
with matching signatures.
Change-Id: Ic14a4c087f3961a4b4f373e2e3d792aba71868f6
Override the metric map of each datasource as soon as it is created by
the manager. This override comes from a file whose path is provided by
a setting in config file.
Loading at creation time allows the correct datasource be used when
get_backend is called, this allows loading a datasource whose metric
names get updated outside the watcher's codebase.
The function 'load_metric_map' returns empty-dict in any error case.
Also in case the file is empty where safe_load is unable finds any
yaml documents, it will return None. [1]
Some minor refactoring in the test_manager file for readability and
added tests for file load and metric override.
1 - https://pyyaml.org/wiki/PyYAMLDocumentation
Change-Id: I1df16245f4c7dfd34066f3ab0553cd67154faa58
Implements: blueprint file-based-metric-map
Some users may want to create keystoneclient by specifying the
type of endpoint and region name, so we need to supply the option
for user to choose.
Implements: blueprint support-keystoneclient-option
Change-Id: I49b33a69ec99d2a91568ce27ef89dc80b75e7091
Change I25b4cb0e1b85379ff0c4da9d0c1474380d75ce3a in
Queens refactored the statistic_aggregation method
and renamed the "aggregate" kwarg to "aggregation",
presumably to match the signature of the GnocchiHelper
statistic_aggregation method (the commit message does
not give details) so a base method could be added to
the parent class for all datasource helpers.
As a result, the CeilometerHelper calls to its
statistic_aggregation started passing the new
"granularity" kwarg but failed to match the rename
to the "aggregation" kwarg, which breaks the
CeilometerHelper. This was missed by the unit tests
because the tests were just asserting the erroneous
call that the runtime code made.
This change fixes the kwarg typo and makes the
tests more robust by using the mock spec kwarg
to define a spec for the statistic_aggregation
mock so that it must be called with the correct
parameters defined in the method signature. The
test is refactored to reduce duplicate mocking.
The same test hardening can and should be done
in the gnocchi and monasca helper tests but that
should be done in a separate change.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Closes-Bug: #1829542
Change-Id: Idfd099f718873d9056fdc35a97954771c9ae5762
As of change Ic4659d1f18af181203439a8bf1b38805ff34c309 the
nova CDM will not be built until an audit is performed.
Instances and services (compute hosts) can be created and
deleted before an audit is performed which will attempt
to use the notification callback function which relies
on the CDM being built already, and if not results in
an AttributeError.
This change side-steps that issue by checking to see that the
nova CDM exists before trying to call the notification
callback function.
An alternative to this is forcefully create the nova CDM when
notifications are received before an audit which is what happend
before change Ic4659d1f18af181203439a8bf1b38805ff34c309.
Change-Id: I16990afb82019821c443c9df26d3e515e52efa69
Closes-Bug: #1828582
_post_live_migration[1] runs on the source host and calls
post_live_migration_at_destination on the dest host which
emits the instance.live_migration_post_dest.end notification:[2]
But it's not the last notification for the live migration operation.
so we should use instance.live_migration_post.end instead of
instance.live_migration_post_dest.end notification.
[1]daa2ac2287/nova/compute/manager.py (L6907)
[2]daa2ac2287/nova/compute/manager.py (L7035)
Change-Id: Id1e2d98f56d5a95d49e32f98d2910660b9f48ce6
The version of bandit in lower-constraints (1.4.0) does
not match the version in test-requirements (1.6.0) however
bandit is a test-only dependency and there is no test coverage
for bandit in the lower-constraints tox job target, so there
is really no good reason to have bandit in lower-constraints.
As such, this change simply removes it from lower-constraints.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Change-Id: I35f66994e9a3a334b342232587d84491542da755
Sphinx 2.0 no longer works on python 2.7, so we need to start capping
it there as well.
The errors are as follow:
Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.5'
does not match "python_version>='3.4'"
Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.5'
does not match "python_version=='2.7'"
Could not find a global requirements entry to match package sphinx. If the package
is already included in the global list, the name or platform markers there may not
match the local settings.
Change-Id: I6dad56ffbb9e85e36cacea1a89565c2fc8248fbf
The final Stein version of Watcher was 2.0.0
so this fixes the version mentioned in the
watcher-status man page docs.
Change-Id: I7fce35471cf31222f9cdafc35e5a7b287bc4598e
The _add_virtual_layer and _add_virtual_servers methods
have not been used since Ic4659d1f18af181203439a8bf1b38805ff34c309
in Stein so this change removes them.
Change-Id: I8c05f29c3c03aa5897cb182bb492948771c42881
This enhances the [collector]/collector_plugins
config option help text to mention the storage
and baremetal in-tree collectors and the ability
to load out-of-tree collectors via extension point.
While doing this, the help text is formatted for
prettier rst rendering in the docs.
Change-Id: Ifd32c95c664c4e9586c250e6bceaeaba2e2df417
CeilometerClient has been deprecated and is no longer available for
master. Without ceilometer client installed docs fail to build with
an exception [1].
This patch marks the import optional.
1 -
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/sphinx/config.py", line 368, in
eval_config_file
execfile_(filename, namespace)
File "/usr/lib/python2.7/site-packages/sphinx/util/pycompat.py", line
150, in execfile_
exec_(code, _globals)
File "/usr/lib/python2.7/site-packages/six.py", line 709, in exec_
exec(""exec _code_ in _globs_, _locs_"")
File "<string>", line 1, in <module>
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/doc/source/conf.py",
line 20, in <module>
objects.register_all()
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/objects/__init__.py",
line 31, in register_all
__import__('watcher.objects.action_plan')
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/objects/action_plan.py",
line 78, in <module>
from watcher import conf
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/conf/__init__.py",
line 28, in <module>
from watcher.conf import datasources
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/conf/datasources.py",
line 21, in <module>
from watcher.datasources import manager
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/datasources/manager.py",
line 19, in <module>
from watcher.datasources import ceilometer as ceil
File
"/home/abuild/rpmbuild/BUILD/python-watcher-2.1.0.dev45/watcher/datasources/ceilometer.py",
line 21, in <module>
from ceilometerclient import exc
ImportError: No module named ceilometerclient
)
Change-Id: Idcf582c2495aab39aacf691b687759405bb94dca
Currently default config files are being for initialization of CONF from
oslo_config. However default config dirs are not being passed as a
result watcher components (eg: decision-engine) are unable to load
files from default directories (eg: /etc/watcher/watcher.conf.d)
supported by oslo_config. This is a short-coming on watcher's side.
Also this forces user to have multiple config for each component.
Without this default set, oslo_config will search for conf with string
'python-watcher' in it, eg: /etc/python-watcher/.... Since there is a
because project=python-watcher a couple of lines below
This patch adds the option after evaluating using project as 'watcher'
which is similar to evaluation of default_config_files and also allows
it to be passed in as a function parameter.
Change-Id: I013f9d03978f8716847f8d1ee6888629faf5779b
This fixes the wrong installation guide link from the
user guide which was pointing to the watcherclient docs
for some reason, maybe it was just a copy/paste error.
Change-Id: I38f536e187245523ac37d70054a2df8cdfcbe4b2
Closes-Bug: #1828584
Hard-coding watcher.openstack.common to warning level logging
only makes it hard to debug watcher's interactions with other
services, like when it's triggering and monitoring a server live
migration.
Since debug logging is controlled via the "debug" configuration
option, we can just rely on that to filter out debug logs within
watcher itself.
Note this has been this way since change
I699e0ab082657880998d8618fe29eb7f56c6c661 back in 2015 and there
was no explanation why the watcher.openstack.common logging
was set to WARN level.
Change-Id: I939403a4ae36d1aa9ea83badb9404bc37d18a1a6
Related-Bug: #1828598
The -x option for bandit changed in 1.6.0 and now
supports glob patterns so use that to correctly
exclude test code from bandit scans.
Since this change requires bandit>=1.6.0, we have
to also fix the networkx requirement to pass the
requirements-check job so that the networkx requirement
matches what is in global-requirements from change
I0a9700926c9a0db93e782c853c33f1aaee3d4876.
Change-Id: I4fc1166daee5d8739296419216d11d684be27c0a
Closes-Bug: #1828419
Allows to define a global preference for metric datasources with the
ability for strategy specific overrides. In addition, strategies which
do not require datasources have the config options removed this is
done to prevent confusion.
Some documentation that details the inner workings of selecting
datasources is updated.
Imports for some files in watcher/common have been changed to resolve
circular dependencies and now match the overall method to import
configuration.
Addtional datasources will be retrieved by the manager if the
datasource throws an error.
Implements: blueprint global-datasource-preference
Change-Id: I6fc455b288e338c20d2c4cfec5a0c95350bebc36
To get log formatting like the other openstack projects
running in devstack the setup_logging function should be
used. This will also give us the "Display level" formatting
in the logs via the os-loganalyze packaged used by infra.
Needed by: https://review.opendev.org/657652
Change-Id: I5e9bd5a142e45804e8d915b370746a2142243088
Exceptions should be reraised with just "raise" and not "raise e" to
preserve the traceback. This also addresses a couple cases where the
catching and reraising of the exception was not actually doing anything.
Change-Id: I94ba193f728ee7ca6f689f70fc08317a1dd50c92
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
We have a bp to add resource_name in action input parameter field.
Before doing this, one of this method's parameter should be node
instead of resource_id.
Change-Id: I4ce5ae97efce98d80a460fd6003df3cc5cacab82
This resolves problems with the audit scope such as the scope being
ignored, the scope not merging due to a type in .append, change update
into .add method when adding single elements to a set and making the
access of dict keys and values as lists work in python 3.7.
All these methods from the model builder now have tests to prevent
regressions.
Co-Authored-By: Canwei Li <li.canwei2@zte.com.cn>
Change-Id: I287763d5e426ff860aefabc4a1f3fe3f51accd76
Since commit I8df8921337ea3f4e751c0c822d823e64e3ca7e1c
the check for hardware.cpu.util was removed.
But it can be still used in workload stabilization.
Change-Id: I301487837aac2e1e63bce16a79d0f8136452c313
Our cgit instance will be going away and opendev.org is the new
preferred URL for browsing our git repos. Redirects will exist for the
foreseeable future, but it's more efficient to just go directly to the
new locations.
Change-Id: I7dd9d454da63167832bab02c89be98a2ce03b72a
Now there are only one scheduler for launching audit task and
executing audit jobs. We have found an exception where the scheduler
stops for some reason when executing audit.
In order to keep launching audit task normal, we need to split into
two schedulers.
Change-Id: I45dccaf062290cfc7d7fcfc27fe11d6f87f38afa
When the lower-constraints tox target was added, it was assumed the
install_command was just running the install and that the dependencies
and constraints were being set using "deps = ".
This fixed the install_command and deps to follow the expected pattern
so the lower-constraints job actual does install the lower constraints.
This also raises the oslo.context minimum as
Ic96c1f1e1a80099d9dafa95a014fc47f05b88e42 added a dependency on a newer
versions kwarg.
Depends-On: https://review.openstack.org/#/c/647726/
Change-Id: I4cc2c3ac158a607b22295c50f83896969a4007ee
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
When use tox-elower-constraint, we will meet these errors:
1. line 417, in test_clients_ironic:
b"AssertionError: Expected call: Client('1', endpoint_override...
b"Actual call: Client('1', 'http://localhost:6385/'...
2. line 39, in test_wrong_major_version:
b"KeyError: 'HTTP_ACCEPT'"
3. RUN END RESULT_TIMED_OUT:
[untrusted : git.openstack.org/openstack-infra/
zuul-jobs/playbooks/tox/run.yaml@master]
For the first error, The reason is that the unittest for the
ironicclient is too strict and must be adapted to the latest code.
In fact, the watcher can use the previous ironicclient version.
Therefore, we modified the unittest so that the watcher does not
have to rely on the latest ironicclient version.
For the second error, The reason is that we need to update the minimum
version of pecan and webOb.
For the third error, the reason is that the version of the oslo_messaging
is too low.
Change-Id: Icb3eda3d27fa4452e13e2dcd3c016cc76fc2c7c7
Metrics for datasources now match the name of their corresponding
abstract methods. This ensures that developers know how the method
is named if they know the name of the metric and vice versa.
Change-Id: I0f9d400432d8182b3f10a0da97155e6cb786690e
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I3452a7802dde00d8be32c833d714b2974be58e16
Add file to the reno documentation build to show release notes for
stable/stein.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.
Change-Id: I25931207ed6066f905fe66ca504fa230e40d12dc
Sem-Ver: feature
Although this method does not report an error, this type check 'int'
is redundant and may be misleading.
Refercnce code url: https://github.com/openstack/wsme/blob/master/wsme/api.py
Change-Id: I631b5f9901790666e7f20275e8c8b99f06f06f0a
Many strategies execute very similar statements especially in
pre_execute and some might raise errors that others might not. This
same pattern of many similar statements can also be observed in
strategies their tests.
This patch addresses these issues, firstly; the BaseStrategy class gets
1 additional method _pre_execute which allows for general logic that
most strategies perform at that stage. This method can be executed
before the similarly named method of the superclass. A notable change
is that _pre_execute now handles common exception handling for
ClusterStateStale & ClusterStateNotDefined exceptions.
A similar pattern is applied to the test classes of the strategies
each of these classes now inherits from the TestBaseStrategy class.
This class provides the common attributes almost every test class for
the strategies requires such as: The mocked compute_model, mocked
audit_scope and an instance of FakerModelCollector.
Finally, some minor changes were required in test_strategy_context
& test_audit_handlers and exceptions around 0 nodes in cluster or
storage are removed.
Change-Id: Ia7154376b2448aac65cf17999cc8c3e1c8309b5b
This patch adds a scope to the datamodel, which only gets the VMs
of the specified nodes, and no longer gets all VMs from nova.
Implements: blueprint scope-for-watcher-datamodel
Change-Id: Ic4659d1f18af181203439a8bf1b38805ff34c309
Audit will only failed if an exception occured.
The situation that no solution found will not cause audit failed.
Change-Id: Ib9c3c3505f31c14500926ec13aa865dc8f7aa310
on ImportError set HAS_CEILCLIENT to false
Without this none of the watcher componenets can be started for master
as well as rocky because the ceilometercleint was deprecated.
Using the variable the support for ceilometer can be gradually removed
from master
A backport to rocky will allow using watcher without ceilometerclient.
Change-Id: I3beb0fb8f0a8e8e0a22acaf6bdeca492836bbee2
Moved the metric mappings for Ceilormeter, Gnocchi & Monasca out of
base.py. The datasources manager now uses classes extending base.py
their NAME attribute as key in the dictionary of total available
mappings and datasources. base.py still contains a template
definition of all available mappings so that anyone extending the
base class can identify all the possible endpoints they can map to.
Change-Id: I6a826423031b5a6a60c4cd5fe24f74b8400f6b55
Closes-Bug: #1815769
Prevent workload_stabilization strategy from failing in a network with
0 hosts.
Change-Id: I9f1a9524923c14d958eb50a70dad379a6021b884
Closes-Bug: #1815059
Small cleanups:
* Use openstack-lower-constraints-jobs template, remove individual
jobs.
* Sort list of templates
Change-Id: I63bfcd9bc21011b446fd1c54cb64c5568c601687
Needed-By: https://review.openstack.org/623229
This patch set removes "observable" and "synchronization"
modules cause they aren't used by any Watcher modules so far.
Change-Id: If23cdf0d3d09087919d48f50ab38b0d355c36481
Ceilometer Datasource has been deprecated since its API has been
deprecated in Ocata cycle. Watcher has supported Ceilometer for some
releases after Ocata to let users migrate to Gnocchi/Monasca datasources.
Since U-release, Ceilometer support will be removed.
Change-Id: I944a5a562ab09a36961eb9b75e9a5144ba0b9ca4
file host_maintenance.py
This is to fix spelling error and unsuitable punctuation
in file host_maintenance.py
Change-Id: I9c535059c3a02277be4c7329693db34fb7674b4e
Bare metal cluster data model was introduced in Queens cycle.
Since the model is different from compute data model, we
need add CDM scoper for bare metal cluster data model
Change-Id: Idd041cefb692085d4545252d229ebe8602926b58
Implements: blueprint audit-scoper-for-baremetal-data-model
vm_workload_consolidation.py
Increase the test of the execute method which contains
the pre_execute(), do_execute(), post_execute() methods.
Increase coverage from 82% to 87%.
Change-Id: Ibde67d7b7d7945657ad0b674e06b1edc9eb24a9f
When tls-proxy is enabled, first start the tls-proxy and then then wait
for api to come up.
Without this the api comes up on the internal port as a result the
subsequent curl fails killing the deployment
- create a zuul job to test with tls
- fix apache ports accordingly
Depends-On: Ie665240b53df92b8e5ca509e998e95d859bd5282
Change-Id: I610a7a24daab68c7ab0e30977e3cabd62cdb56a5
Actually, the metric "cpu_util" and "memory.resident" are necessary
in vm_workload_consolidation.py, according to line 75. So modify
this document about this part.
Change-Id: I648f341184a0b42d88dcb4c934af989997fe3fea
This patch updates response header
from OpenStack-API-Version: [VERSION_STRING]
to OpenStack-API-Version: [SERVICE_TYPE] [VERSION_STRING]
Change-Id: I10577ff1123ef781bd4aa0b26577574a3f7e9c39
Add a new config option 'action_execution_rule' which is a dict type.
Its key field is strategy name and the value is 'ALWAYS' or 'ANY'.
'ALWAYS' means the callback function returns True as usual.
'ANY' means the return depends on the result of previous action
execution. The callback returns True if previous action gets failed,
and the engine continues to run the next action. If previous action
executes success, the callback returns False then the next action
will be ignored.
For strategies that aren't in 'action_execution_rule', the callback
always returns True.
If exception is throwing out during the action execution, reverting will
be triggered by taskflow. To continue executing the next action,
we return False instead of throwing an exception.
Change-Id: Ib5afa214d8d097d739aad35d18b3fe5c8e4de8fc
Implements: blueprint enhance-watcher-applier-engine
With these brackets, the statement "raise Exception("Live migration
execution.....") in watcher/common/nova_helper.py, line 379 will never
be executed. So remove it and let the statement make sense.
Change-Id: I42a2fa0c8ffa9c84a918d432c5093470dbd80f82
The method is quite simple and it doesn't need a dostring.
Also existing docstring was incorrect. The name of the expected
parameter is 'name', not 'node'. And it cannot be an object
of the type node.StorageNode
Change-Id: I94124d327c490d45eae4d2ded218beadfbc33ad7
The commands used by constraints need at least tox 2.0.
Update to reflect reality, which should help with local running of
constraints targets.
Change-Id: I0eb9af735f34ad259c7099729d7d465a1276fc5f
The correct type of parameter 'pool' in method build_storage_pool is
<class 'cinderclient.v2.pools.Pool'>
Change-Id: I986f707e4e740ebec94a46c6ee413f9a70197dad
This patch set adds API microversion support along
with the first API microversion: start/end time for
CONTINUOUS audits.
APIImpact
Implements: blueprint api-microversioning
Depends-On: I6bb838d777b2c7aa799a70485980e5dc87838456
Change-Id: I17309d80b637f02bc5e6d33294472e02add88f86
Now we have removed nova legacy notifications in Watcher
and just consume nova versioned notifications,
we don't need notification config in nova.conf
Change-Id: I1c9c141d98d858c36ad8bb7be0b95c38ff1d5752
This commit adds the functionality of watcher-status CLI for performing
upgrade checks as part of the Stein cycle upgrade-checkers goal.
It only includes a sample check which must be replaced by real checks in
future.
Change-Id: Ic3d066af439797d6f705e805334f729b52ce3aac
Story: 2003657
Task: 26164
Add new start_time and end_time fields in the audit table
Partially Implements: blueprint add-start-end-time-for-continuous-audit
Change-Id: I6bb838d777b2c7aa799a70485980e5dc87838456
As rpc_backend config option has been removed from
oslo_config [1], projects should not use it.
Current uses of it cause watcher crash when installing
via devstack.
[1] https://review.openstack.org/#/c/580910/
Change-Id: Iba7471e87e8935f1ea02b363f269e9debdc7cc71
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')
Change-Id: I0371fc2c0878a177c0a9e9c9313ca5b8470bfd98
This patch set fixes process of audit creation and
allows to create audit without Audit Template using only
names of Goal and Strategy. It also provides some additional
unit tests to improve tests covering.
Change-Id: I89a9c7661616f49639151869055d8f5ebe723d5f
Closes-Bug: #1794233
This patch set adds efficacy indicators for workload_balancing
goal (that includes workload_stabilization and workload_balance
strategies so far).
Change-Id: I5b04d084ace7c661001c62f07b8308e5763e144d
oslo_context may add new fields in request context, there are no
need to warning these fields.
Closes-Bug #1790577
Change-Id: Ic96c1f1e1a80099d9dafa95a014fc47f05b88e42
This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I5e75f2ea7dd02065bc18793d974f56fef2daa2c4
Story: #2002586
Task: #24344
As openstack installation guides suggest to run mysql with root shell
user, mysql will not ask for password, so in
controller-install-*.rst "-u root -p" is useless.
Change-Id: I511f39d734702ab3d1a209f7d868f52fb184f1fc
Related-Bug: #1785025
This patch set refactors logs of workload stabilization
strategy to make them more readable and sensible.
Change-Id: I408988712bb7560728157f3b4e4f2b37572128c4
We just need to check decision engine service status
when Rescheduling continuous audits.
This is an update for 1
1:https://review.openstack.org/#/c/586033
Change-Id: I05a17f39b6ff80c6b9382248c72cac571191e395
We have replaced voluptuous with jsonschema in [1].
Now voluptuous can be removed.
[1]: https://review.openstack.org/#/c/561182/
Change-Id: I99c65ed79ef166839838559a808ee7607389e07a
This patch set adds background job that reschedules CONTINUOUS
audits from FAILED to ACTIVE decision engine nodes using round-robin
algorithm. It also contains fix for main[1] HA PS about filtering audits.
[1]: https://review.openstack.org/#/c/578102/
Partially-Implements: blueprint support-watcher-ha-active-active-mode
Change-Id: Ib248a6cd3adbd3927c47db6bb819300361492411
This patch set adds hostname field to Audit and Action Plan
objects to track services which execute these objects.
Change-Id: I786e419952925c380c969b12cc60f9a1004af96b
Partially-Implements: blueprint support-watcher-ha-active-active-mode
This commit allows to consume notifications via notifications pools[1].
Listeners in notification pools recieves a copy of notification. It
will let Watcher to sync Data Models of Decision Engines.
[1]: https://docs.openstack.org/oslo.messaging/ocata/notification_listener.html
Change-Id: Ie37528263181924f84510500fc1277b0237c1df8
Partially-Implements: blueprint support-watcher-ha-active-active-mode
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.
[1] https://etherpad.openstack.org/p/YVR-python-pti
Change-Id: I8f70f7d8a3d18301559c0eb47e6a64c8b5100d39
Non-voting jobs should not be in gate queue at all, remove them.
They are wasting needlessly are resources.
To make clearer which jobs are non-voting, move voting:false
from base job to individual jobs.
Make watcherclient-tempest-functional as voting job and explicitly set
it non-voting in project stanza so that it can be easier to make it
voting in other repos.
Change-Id: Id94e49347006bca850f72a0400794da8c2a67144
The legacy-rally-dsvm-watcher-rally job does not exist but it is listed
in the .zuul.yaml config. This is a zuul configuration error. Remove
this job which does not exist to fix zuul.
Change-Id: I1bbfd373ad12b98696ab2ddb78e56e6503cc4c4d
This patch fixes watcher.tests.common.test_clients.TestClients.
test_clients_ironic unit test error due to python-ironicclient 2.4.0.
Change-Id: I0aaa2047cc8fcf09dee8bc7168e35f7a58c10125
This patch replaces voluptuous with JSON-schema to validate
efficacy indicator since in watcher we want to remove voluptuous
and use JSONSchema as our only JSON validation tool to keep consistency.
Change-Id: Iaa77566f1cdfdac03ce8e7d5a75406274c7d5298
Implements: blueprint replace-voplutuous-with-jsonschema
url in action_plan policy file is:
'path': '/v1/action_plans/{action_plan_uuid}/action'
whereas it shouls be :
'path': '/v1/action_plans/{action_plan_uuid}/start'
Related-Bug: #1756274
Change-Id: Ic15fed9af739b59efb2777b70514697747b2af7f
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: Ia481330b9a889b113b585fca0d4ddb86df9f74d3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The python 3 version of the linter does not allow Windows-style line
endings (\r\n) so replace them with UNIX-style endings (\n).
Change-Id: Ifb97491323d3df92bb1520e373552aeb5e1919a4
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
oslo_messaging's rpc_backend setting, which is set by
ConfFixture.transport_driver has been deprecated since Newton. To allow
oslo_messaging to remove it, switch to setting transport_url instead.
Change-Id: Ie37b20d1be6c177f2f1c26cc473b632d7d934c05
Partial-Bug: #1712399
This patch set adds API Reference along with some
fixes to documentation. It partially fixes bug #1757423.
Change-Id: I107b4fd5daf40aad63fc13864debbbbc82a9826c
Storage CDM can not be build for some environment such as
the one using VMwareVcVmdkDriver, since some attributes of
Storage CDM'S pool element can be 'unknown'.
This patch updates storage CDM collector to raise watcher
specific exception if some attributes of storage CDM'S pool
element is 'unknown'
Change-Id: If75a909025c8d764e4de6e20f058b84e23123c1a
Closes-Bug: #1751206
Now that the v2.0 API has been removed, we don't have a reason to
include deployment instructions for two separate applications on
different ports.
Related-bug: #1754104
Change-Id: I98fae626d39cb62ad51c86435c1a2c60be5c1fb9
maintain one compute node without having the user's application
been interruptted.
It will firstly migrate all instances from the maintenance node
to one backup node. If not, it will migrate all instances,
relying on nova-schduler.
Change-Id: I29ecb65745d5e6ecab41508e9a91b29b39a3f0a8
Implements:blueprint cluster-maintaining
Currently RabbitMQ connection gradually increases by CONTINUOUS audit
with auto-trigger option.
This patch fixes watcher to reuse RabbitMQ connection.
Change-Id: I818fc1ce982f67bac08c815821f1ad67f8f3c893
Currently the REST API to start action plan in watcher
is which is same as for update action plan.
PATCH /v1/action_plans
https://docs.openstack.org/watcher/latest/api/v1.html
we need to make it easy to understand like :
POST /v1/action_plans/{action_plan_uuid}/start
the action should be start in above case.
Change-Id: I5353e4aa58d1675d8afb94bea35d9b953514129a
Closes-Bug: #1756274
Beginning with the Queens release, the keystone install guide
recommends running all interfaces on the same port. This patch
updates the install guide to reflect that change.
Change-Id: Ice155d0b80d2f2ed6c1a9a9738be2184b6e9e76c
Closes-bug: #1754104
In I4d2f44fa149aee564c62a69822c6ad79de5bba8a we introduced new
_get_model_list method that introduces unify way for retrieving models
from db. This commit adds tests that do checks on bug 1761956, when
selecting with filter() method could return deleted entites.
Change-Id: I12df4af70bcc25654a0fb276ea7145d772d891e2
Related-Bug: 1761956
Both Continuous and Oneshot audits made same action in
do_execute, so it's a good idea to move it to the base
class
TrivialFix
Change-Id: Ic0353f010509ce45f94126e4db0e629417128ded
This patch adds project_id in compute CDM, It also adds logic for
excluding project_id in audit scope.
Change-Id: Ife228e3d1855b65abee637516470e463ba8a2815
Implements: blueprint audit-scope-exclude-project
Since Nova API v2.56, Nova migrate Server(migrate Action) API
has host option.
This patch replaces cold migration implementation to use the API.
Change-Id: Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1
Implements: blueprint replace-cold-migrate-to-use-nova-migration-api
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I2ef330d7f9b632e9a81d22a8edec3c88eb532ff5
When we call audittemplate list without filters, it returns all Audit
Templates that are not deleted, as expected. If we add any filter to
query and context.show_deleted is None (we request only current AT),
query.filter_by adds filter to joined table (for example, goals, results
in a query like JOIN goals ... WHERE ... goals.deleted_at IS NULL) not
to model's table (AuditTemplate in our case).
We change call for filter_by to filter, explicitly point to model that
we want to filter.
Also, we moved query generating code to new method _get_model_list(). As
a result we applied same fix to all of the other models.
Change-Id: I4d2f44fa149aee564c62a69822c6ad79de5bba8a
Closes-bug: 1761956
New releases of oslo.config support a 'mutable' parameter to Opts.
Configuration options are mutable if their oslo.config Opt's
mutable=True is set. This mutable setting is respected when the oslo
method mutate_config_files is called instead of reload_config_files.
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e allows services to tell
oslo.service they want mutate_config_files to be called by specifying
the 'restart_method=mutate' parameter, what this patch does.
The default mutable configuration options (set by oslo.config Opts'
mutable=True) are:
- [DEFAULT]/pin_release_version
- [DEFAULT]/debug
- [DEFAULT]/log_config_append
Concrete params, that made mutable in Watcher:
* watcher_decision_engine.action_plan_expiry
* watcher_decision_engine.check_periodic_interval
* watcher_decision_engine.continuous_audit_interval
* gnocchi_client.query_max_retries
* gnocchi_client.query_timeout
* DEFAULT.periodic_interval
Change-Id: If28f2de094d99471a3ab756c947e29ae3d8a28a2
Implements: bp mutable-config
Removed duplicates of strategies descriptions, added references to
that descriptions instead of module descriptions.
Change-Id: Ife396ddce5c3cc926cc111f1ff1abd3a42c22561
Added creation of [DEFAULT]/transport_url value
in devstack.
Also, fixed same topic in docs.
Change-Id: I9ad9475c4fccf023daac40c0b1e841eeeb22f040
Closes-Bug: 1738329
This patch set removes legacy-* jobs and migrates
tempest functional job to ZuulV3 syntax.
Change-Id: I87771737cc713eae20b4d6aaaefefc5e40875666
Implements: blueprint migrate-to-zuulv3
While sorting output of list command ("audittemplate list",
"strategy list", etc) by sort-key that is not belongs
to specific model, this sort-key was passed to db what
caused error (HTTP 500). We added check on such keys and now,
if got one of them, then we make sort on API side
instead of db side.
We removed excess sort and changed all sorting routines
to unified way.
Also added sort tests on every model.
Change-Id: I41faea1622605ee4fa8dc48cd572876d75be8383
Closes-Bug: 1662887
This patch removes the unnecessary maintenance of a date and version
from the CLI documentation.
NOTE: Cinder/Nova teams also did the same removal with
the commit Idf78bbed44f942bb6976ccf4da67c748d9283ed9
and the commit I0a9dd49e68f2d47c58a46b107c77975e7b2aeaf7
Change-Id: I6a0faeb596f1ee3a3b67d1d37a14e1507aa40eba
With current URL [1], default driver will be used.
In order to ensure the compatibility, it is better to include the exact driver [2].
[1] connection = mysql://
[2] connection = mysql+pymysql://
Change-Id: I4f7b3ccbecfb2f1e2b3d125179dbd5c6fbf5e6b9
The change is wrong. We link on purpose to the unversioned version and update that one once rocky is released.
This reverts commit e771ae9e95.
Change-Id: I0f981a8473a47d18ce20be74a8e2d12d22f40061
NotImplementedError are reported in decision-engine log file
when we activate storage data model and see a Guru Meditation Report.
This patch fixes by adding default values.
Change-Id: I06386f8295f7758cbb633612eee8b19225905c92
Closes-Bug: #1750300
The string %(action) is not valid, it misses the conversion specified,
add s for string.
Note that this leads to an untranslatable string, since our translation
tools check for valid formats and fail. In this case the failure comes
from a source code fail.
Change-Id: I2e630928dc32542a8a7c02657a9f0ab1eaab62ff
Replace the old http url links with the lastest https ones according
to the official OpenStack website.
Change-Id: I1abd79bb80dae44ee2ba5946b8a375c7096b39d6
This patch set completes schema by adding restrictions
to different types of schema properties.
It also makes workload_stabilization strategy more
user friendly by setting cpu_util as default metric.
Change-Id: If34cf4b7ee2f70dc9a86309cb94a90b19e3d9bec
Replace 'a instance' with 'an instance' in
watcher/decision_engine/model/collector/nova.py
watcher/decision_engine/model/element/instance.py
Change-Id: I39020f3e7b460dea768f7e38fef9ae9e2a4b7357
Zuul no longer requires the project-name for in-repo configuration.
Omitting it makes forking or renaming projects easier.
Change-Id: Ib3be82015be1d6853c44cf53faacb238237ad701
This patch set fixes issues that have appeared after merging
watcher-multi-datasource and strategy-requirements patches.
It is final commit in watcher-multi-datasource blueprint.
Partially-Implements: blueprint watcher-multi-datasource
Change-Id: I25b4cb0e1b85379ff0c4da9d0c1474380d75ce3a
If hostname is different from uuid in Compute CDM,
get_compute_node_by_hostname method returns empty.
This patch set fixes to return a compute node even if hostname
is different from uuid.
Change-Id: I6cbc0be1a79cc238f480caed9adb8dc31256754a
Closes-Bug: #1746162
This patch remove redundant import aliases and add pep8 hacking function
to check no redundant import aliases.
Co-Authored-By: Dao Cong Tien <tiendc@vn.fujitsu.com>
Change-Id: I3207cb9f0eb4b4a029b7e822b9c59cf48d1e0f9d
Closes-Bug: #1745527
This patch set adds validation of baremetal model.
It also fixes PEP issues with storage capacity balance
strategy.
Change-Id: I53e37d91fa6c65f7c3d290747169007809100304
Depends-On: I177b443648301eb50da0da63271ecbfd9408bd4f
This patch set adds /state resource to strategy API
which allows to retrieve strategy requirements.
Partially-Implements: blueprint check-strategy-requirements
Change-Id: I177b443648301eb50da0da63271ecbfd9408bd4f
This patch set excludes nodes and instances from auditing
if appropriate metrics aren't available.
Change-Id: I87c6c249e3962f45d082f92d7e6e0be04e101799
Closes-Bug: #1736982
This patch set:
1. Removes nova, ceilometer and gnocchi properties.
2. Adds using of datasource_backend properties along with
statistic_aggregation method.
3. Changes type of datasource config.
Change-Id: I09d2dce00378f0ee5381d7c85006752aea6975d2
Partially-Implements: blueprint watcher-multi-datasource
This patch adds Storage Capacity Balance Strategy to balance the
storage capacity through volume migration.
Change-Id: I52ea7ce00deb609a2f668db330f1fbc1c9932613
Implements: blueprint storage-workload-balance
This patch set:
1. Removes nova, ceilometer and gnocchi properties.
2. Adds using of datasource_backend properties along with
statistic_aggregation method.
3. Changes type of datasource config.
Change-Id: I4a2f05772248fddd97a41e27be4094eb59ee0bdb
Partially-Implements: blueprint watcher-multi-datasource
If it uses a specific value(usually 1 or 2) as the min_sd when starting
to simulate migrations. The first simulate_migration case will always be
less than the min_sd and come into the solution, even though the migration
will increase the weighted sd. This is unreasonable, and make migrations
among hosts back and forth
Change-Id: I7813c4c92c380c489c349444b85187c5611d9c92
Closes-Bug: #1739723
This patch set adds data migration version that fills noname audits
with name like strategy.name + '-' + audit.created_at.
Closes-Bug: #1738758
Change-Id: I1d65b3110166e9f64ce5b80a34672d24d629807d
when we update a existing audit's interval, the interval of
'execute_audit' job is still the old value.
We need to update the interval of 'execute_audit' job.
Change-Id: I402efaa6b2fd3a454717c3df9746c827927ffa91
Closes-Bug: #1738140
Now that CDM-scoping was implemented, Getting scoped storage model
have to be updated.
This patch updates getting storage cluster data model.
Change-Id: Iefc22b54995aa8d2f3a7b3698575f6eb800d4289
This commit registers all policies formally kept in policy.json as
defaults in code. This is an effort to make policy management easier
for operators. More information on this initiative can be found
below:
https://governance.openstack.org/tc/goals/queens/policy-in-code.html
bp policy-and-docs-in-code
Change-Id: Ibab08f8e1c95b86e08737c67a39c293566dbabc7
In one integrated cloud env, there would be many solutions, which would
make the compute resource strongly relocated. Watcher should listen to
all the notifications which represent the compute resource changes, to
update compute CDM. If not, the compute CDM will be stale, Watcher
couldn't work steadily and harmoniously.
Change-Id: I793131dd8f24f1ac5f5a6a070bb4fe7980c8dfb2
Implements:blueprint listen-all-necessary-notifications
Migration action 'cold' is not intuitive for the developers and users,
so this patch replaces it with ‘migrate’ and 'retype'.
Change-Id: I58acac741499f47e79630a6031d44088681e038a
Closes-Bug: #1733247
Migration action with migration_type 'cold' does not work.
This patch fixes nova_helper to follow Pike release python-novaclient.
Change-Id: I0637cefad68cd4b7077a5d3e6b5db5562e7f2917
Closes-Bug: #1736624
This patch set follows deprecation steps
and changes default datasource to gnocchi
Closes-Bug: #1735180
Change-Id: I3b7ec47af7707359373a424a6469d28cbf9ce177
This patch removes 'audit_scope' from __init__ of
BaseClusterDataModelCollector class, as it is a singleton class
and cannot be instantiate more than once.
A new method is defined in BaseClusterDataModelCollector in place
of property audit_scope_handler, which takes audit_scope as
argument.
Change-Id: I0664c151d71a711c118d43c180d8b0760b1c81fa
Closes-Bug: #1732849
In one integrated cloud env, there would be many solutions, which would
make the compute resource strongly relocated. Watcher should listen to
all the notifications which represent the compute resource changes, to
update compute CDM. If not, the compute CDM will be stale, Watcher
couldn't work steadily and harmoniously.
Change-Id: I57173f0cce0717aa36c5ff758d972d38013e3ef8
Implements:blueprint listen-all-necessary-notifications
This patch adds notfication changes after changing the type
of global_efficacy in ActionPlan objects.
Change-Id: I1116cf1d19e4b19e19789f943a631d0400871766
Partially-Implements: blueprint multiple-global-efficacy-indicator
Storage pool can have many volume types,
'get_volume_type_by_backendname' should return a list of types.
Closes-Bug: #1733257
Change-Id: I877d5886259e482089ed0f9944d97bb99f375824
The app.wsgi file should be adjusted to meet pep8 standards and
apply pep8 check to app.wsgi automatically.
Change-Id: I34ec9ce56a329ede76d3d69f6e973d66350d85d1
This patch does step 1 in the docs: Move Legacy Jobs to Projects.
Partial-Implements: blueprint migrate-to-zuulv3
Change-Id: I0c0713260c2dfa2ba64a5746c342db01c1bc3d75
The os-testr 1.0.0 release had a couple of required config changes due
to it's internal usage of stestr. This commit fixes those by adding a
.stestr.conf (to remove a warning) sets the fixture env variables in
the tox.ini instead of being hidden in .testr.conf and removing the
quotes around posargs to make passing args to ostestr actually work.
Also to keep the expected dev experience stestr is used directly for
places where ostestr was called directly.
Change-Id: I785b0dc1720328b119f0978aa573ad0b89e54c75
Closes-Bug: #1731155
This patch update the global efficacy for server consolidation
strategy, test cases and general format.
Change-Id: I62af1e4be415998669f938b3d587c1ccf4293419
Implements: blueprint multiple-global-efficacy-indicator
According to "code conventions" [1], do not use "-y" option.
Instead, use apt-get install package, yum install package,
or zypper install package.
[1] https://docs.openstack.org/doc-contrib-guide/writing-style/
code-conventions.html
Change-Id: I174c8a0c653e322bbf951d72445b1a43251baa7e
If server migration fails, the while loop will not break
until retry to 0,
we can check the task_state to avoid this situation.
Closes-Bug: #1728476
Change-Id: I07e1048eb736263a261456ee78c96fee9db13cb5
Use https instead of http to ensure the safety without containing our
account/password information
Change-Id: I2e4f120f1d57c6c666d65a9a279eddefd112bf23
If scheduler uses db job store, the self.scheduler is None
beacuse execute_audit is class method.
We get [] from self.scheduler.get_jobs()
and can't remove the cancelled audit.
Change-Id: I1768c847465bbefbc9794281328edb07fea83b03
Closes-Bug: #1726345
This patch updates document and add release note for cdm-scoping.
Change-Id: I8e9099aa5caf6bf17ea912e7b1a600af3b6d519d
Partially-Implements: blueprint cdm-scoping
Use https instead of http to ensure the safety without containing our
account/password information
Change-Id: I16bdf6067fba4ea841071100d68edae7750946de
1. Update the instance migration link
2. remove the unnecessary install-guide link
configure and install Ceilometer by
https://docs.openstack.org/ceilometer/latest is enough
remove the other link.
Change-Id: I2bf408de1023750a3d1f2c9e25293649d99ac428
There are more than one 'migrating' status in the volume migration.
Others include starting, completing and so on.
So we should check the final status 'success' and 'error'.
Change-Id: I8ee9330aa32ec4516f2bf9e046ea68a72bc8a53d
https://github.com/openstack/python-novaclient/blob/
master/novaclient/v2/servers.py#L468
From nova api version 2.25(Mitaka release), the default value of
block_migration is None which is mapped to 'auto'.
So we can remove 'block_migration' from live_migrate_instance.
Change-Id: I75a581d3146824b1146e2acf5a7fbe4f8368a4e8
There is already define version_string in version.py
So we don't need generate version in other files, just call it.
Change-Id: I7d8294860523eedad92e213ad00569829e120c39
As of Nova API microversion 2.47, response of GET /servers/detail has flavor
which contains a subset of the actual flavor information used to create the
server instance, represented as a nested dictionary.
Since current watcher nova default api version is 2.53(Pike), This patch
follows the API response change.
Change-Id: Ia575950f0702afa1d093f03ca8ddedd3c410b7de
Closes-Bug: #1722462
This patch removes, explicitly enabled neutron from local.conf
as devstack now default uses neutron
Change-Id: Icf6bd944dd2262ff23cbcceb762a9ba80f471dbb
This patch fixes TypeError of range() in function chunkify, as
range() integer step argument expected, but got str.
Change-Id: I2acde859e014baa4c4c59caa6f4ea938c7c4c3bf
Since default workflow engine action container do_execute method
does not raise exception when action failed, workflow engine action
container execute method never raise exception and
action plan state becomes always SUCCEEDED.
This patch fixes default workflow engine action container do_execute
method to raise exception when action does not return True.
Change-Id: I7eeef69dbdfb5d40e3cf0b1004cbfe199a16bf7b
Closes-Bug: #1719793
Since packages for openSUSE and SLES are not provided,
this patch removes installation guide for openSUSE and SLES.
Change-Id: Ic15d8c4b262e935c7acaef41e18960d0b259d5c9
Closes-Bug: #1715032
This patch adds Notifications for cancel action plan
operation.
Change-Id: I5a89a80729349e3db43ca35ff9fbe8579e86b3b1
Implements: blueprint notifications-actionplan-cancel
disk_over_commit flag was removed in Nova API microversion 2.25(Mitaka).
Since current watcher nova default api version is 2.53(Pike),
this patch removes disk_over_commit flag.
Change-Id: Ib141505b9e8cb41997b29c1762e387b1f84f5143
Closes-Bug: #1720054
As of now we can not know what was happend when migrate action
failed critically.
This patch adds exception log when migrate action failed critically.
Change-Id: I54d0bc54ee1df6f13754771775c58255f53f5008
At various places in watcher code, we are using property getters
to set property, in this way the property setters defined are
never used, this patch fixes to use property setters to set
property.
Change-Id: Idb274887f383523cea39277b166ec9b46ebcda85
python-subunit is not used directly anywhere
and it is dependency of both testrepository
and os-testr
(probably was used by some tox wrapper script before)
Change-Id: I89279430554bc522817c4e2685afef0d95c641dd
add 'disabled_reason' filed into 'ComputeNode' resource, to distinguish
which nodes are disabled by Watcher and which are not by Watcher.
Implements:blueprint extend-node-status
Change-Id: I7175f14870834a4582e45309529d7e8d9fbb2e6f
This patch set replaces create_schema with upgrade to fix
apscheduler creation issue. It also fixes pep8 warnings to
d09a5945e4a0_add_action_description_table.py
Change-Id: Ica842d585ee3a9cd67e45eb1d7bb1916573d7c9c
The return value of ironic_client.node.set_power_state is None, so it's
useless to return the result.
We should check the node state until it's changed or timeout.
Change-Id: I31f75a2c4a721ce4481e6ae7fb83d154a443dad9
Closes-Bug: #1713655
This Patch fixes gate failure, encountered in recent version
of oslo_messaging.
Change-Id: I6d8ab882a7c157dcf4f78c805a4ce2d9b1fa3f14
Closes-Bug: #1716476
iso8601.UTC is correct datetime UTC field object.
iso8601 >= 0.1.12 includes only iso8601.UTC for python3
while both UTC and Utc() for python2. Less then 0.1.12
included both UTC and Utc() for both python2/3.
Change-Id: I0f8796fba6725eea013b3f8d9ad33c10a402c524
Closes-Bug: #1715486
AverageCpuLoad and MigrationEfficacy efficacy indicators are not used.
This patch removes unused indicators.
Change-Id: I2b21defd442c135d26f8fd45f6faf9f67c770bde
Gnocchiclient uses keystoneauth1.adapter so that adapter_options
need to be given.
This patch fixes gnocchiclient creation.
Change-Id: I6b5d8ee775929f4b3fd30be3321b378d19085547
Closes-Bug: #1714871
Audit scope JSON schema should restrict key of host_aggregates
to "id" or "name", but that is not working now.
This patch fixes DEFAULT_SCHEMA to validate host_aggregates.
Change-Id: Iea42da41d61435780e247736599a56c026f47914
Closes-Bug: #1714448
Diffrent stratege has diffrent default scope, restrict them to their
default scope will avoid usage problems.
1)workload_balancing/thermal_optimization/airflow_optimization goals
react on enabled nodes, so restrict default scope to compute nodes
with up state and enabled status.
2)server_consolidation goal react on enabled or disabled nodes, So
restrict default scope to compute nodes with up state and
enabled/disabled status.
Change-Id: I7437dee699ee2d3dd227a047196d4d8db811b81e
Closes-Bug: #1714002
Services are now identified by uuid instead of database id to ensure
uniqueness across cells.
GET /os-services returns a uuid in the id field of the response
from API microversion 2.53(maximum in Pike)
This patch set updates default Nova API version to 2.53.
Change-Id: Ib9fefb794eda3c9e75c6a2f5cfdb0e682b8955f3
Closes-Bug: #1709544
Since iter(dict) is equivalent to iter(dict.keys()), it is unnecessary
to call the keys() method of a dict, the dictionary itself is enough
to be referenced. The shorter form is also considered to be more
Pythonic.
This patch removes the unnecessary dict.keys() method calls in api.
This is a part of a larger patch series that removes dict.keys()
method calls.
TrivialFix
Change-Id: I29000f1f05b90d70109fa01393e97e1ebf450c63
During the strategy sync process,
if goal_id can't be found in the goals table,
will throw a KeyError exception.
Change-Id: I62800ac5c69f4f5c7820908f2e777094a51a5541
Closes-Bug: #1711086
Only when True is returned, the action state is set to SUCCEEDED
some actions(such as migrate) will return None if exception raised
Change-Id: I52e7a1ffb68f54594f2b00d9843e8e0a4c985667
By the input parameter "metrics", it makes decision to migrate a VM
base on cpu or memory utilization.
Change-Id: I35cce3495c8dacad64ea6c6ee71082a85e9e0a83
outlet_temperature strategy relies on a datasource config parameter,
which can be either "ceilometer" or "gnocchi". This patch overrides
get_config_opts method of base class to allow specify datasource.
Change-Id: I551401039e26816568a04c7f2151d5b3c7ed269a
Closes-Bug: #1709024
List comprehensions and generator expressions are considered to be more
Pythonic (and usually more readable) than map and filter with lambda.
This patch replaces four usages of [map|filter](lambda ...) with the
appropriate list comprehension or generator expression.
TrivialFix
Change-Id: Ifda9030bb8aa196cb7a5977a57ef46dfefd70fa6
Metric 'memory.usage' depends on balloon driver,
and shows the memory usage inside guest-os,
while 'memory.resident' represents volume of RAM
used by the instance on the physical machine
So, it is more reasonable to use 'memory.resident'
when calculating node utilization
by gathering instances utilization of the node.
Change-Id: I12dff5176bcf6cb103aa44cafd54f9ecd7170864
Watcher fails to load storage plugin in case there is no installed
Cinder in OpenStack services.
This patch set adds collector_plugins parameter under collector
section in watcher.conf. If plugin name is in collector_plugins,
The plugin is loaded.
Change-Id: Ie3c3543216c925d49b772bf5fe3773ca7d5ae437
Closes-Bug: #1707603
compute_model.get_node_by_uuid(node_uuid) will raise a exception
when the compute node isn't in the compute model.
Closes-Bug: #1709004
Change-Id: I667a9dbfcf67f9f895976aadd5300bbea2ffe6f0
monascaclient was recently migrated to
use 'osc' library. Due to that, exception
classes has been changed. This commit adjusts
the exception class to recently released
monascaclient==1.7.0
Depends-On: Ie647221a3ab055e7b150d65ffb9287b44ef834cb
Change-Id: Icfef345c4269ac4cb682049f22a43fdab3d39845
This patch set updates gnocchi repository URL in local.conf.controller
bacause it moved from under openstack to their own repository.
Change-Id: I53c6efcb40b26f83bc1867564b9067ae5f50938d
Currently compute CDM excludes disabled compute node.
This patch set fixes compute CDM to include disabled compute node.
Change-Id: I8236bb73e0d9bb242251c2abfb59ad5693087afa
Closes-Bug: #1685787
This patch updates the state machine diagram for action plan, It
includes new state "cancelling" which is introduced by actionplan
cancel operation
Change-Id: I0af59f2164922c56d59fbad8018e2aecfef97098
In this patch feching resource_id by resource's original_id was added to
statistic_aggregation method.
Closes-Bug: #1707653
Change-Id: I70b9346146f810e2236ccdb31de4c3fedf200568
This strategy now allow us to create action plans with an explicit
set of actions.
Co-Authored-By: Mikhail Kizilov <kizilov.mikhail@gmail.com>
Change-Id: I7b04b9936ce5f3b5b38f319da7f8737e0f3eea88
Closes-Bug: #1659243
1. Update python version from 3.4 to 3.5
2. Update Ubuntu version from 14.04 to 16.04
3. Update Fedora version from 19+ to 24+
Change-Id: Ic5e9bbd126e10697300c6ffd51ff55d0b815d5ca
Currently continuous audit fails once it fails
because continous audit tries to remove job
even if job is not exists.
This patch set fixes it.
Change-Id: Ic461408c97d71e14c57e368f8436b26fe355fa4e
Closes-Bug: #1706857
This patch fixes the hardcoded value of availability zone
in nova-helper, Now nova api is used to get the availability zone
of destination node
Change-Id: I4c5a34946ed404df5bbfe34ce99873d32772dbf4
Add a new table to save the mapping
Add logic to update the table when action loading
Add logic to show the action description
Change-Id: Ia008a8715bcc666ab0fefe444ef612394c775e91
Implements: blueprint dynamic-action-description
Cinder data model was added in Pike cycle and that needs
configuration in cinder.conf for refreshing the model in
real time.
This patch set adds Add Configure Cinder Notifications section
for explaining the configuration.
Change-Id: I41cc870e2d47c56fd7c9fcdd6f03c95fa939c3f2
Now that we replaced voluptuous with JSONSchema in all actions,
this patch replaces voluptuous with JSONSchema in BaseAction
and removes validate_parameters method in each action.
Partially Implements: blueprint jsonschema-validation
Change-Id: I07c907ddfa4a568d7fff42776df02218330d56a0
This patch set fixes devstack plugin by removing head
keyword from watcher-db-manage.
It is related to https://review.openstack.org/#/c/483825/
Change-Id: I3dcea6ae799c94a882e68d66920c5cd87d83d85e
This patch removes the requirement to specify destination node,
for migrate action. There can be some strategies that wants to rely
on nova-scheduler for destination node. It will not impact any current
strategy behavior.
Change-Id: Ia12b2f0b8369e632a7959b28e485d86b6cff83e9
Closes-Bug: #1691056
There are many uselees logging in the decision log file:
Arguments dropped when creating context: {u'global_request_id': None}
Change-Id: I7583b2ff6ea1cc15173536ca1cf08cc9f0ecb20d
Closes-Bug: #1704736
This patch adds abort operation for live migration
to support abort in cancel action plan.
Change-Id: I458e93d9bd09dc4cf80cc941104129fc7600a6b1
Partially-Implements: blueprint cancel-action-plan
wrong parameter passing of function "query_sample"
the default parameter will be recognized as "user_id"
Change-Id: I293b130c3f709dc93cd4b1b7382ae8895a54765d
Closes-Bug: #1703297
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the change_node_power_state action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: If9ffe5e0b107e0da5673247e4d5ec9917790827f
This patch set updates documentation link in README
because of doc migrations.
Depends-On: Iecb4f60efb015a56b9b37331859848b287112842
Change-Id: I5b8a5ec8a328ae275fe6271965ea0077ae301814
This patch set makes the following changes:
* Add index file to each subdirectory of doc/source
* Update doc/source/index.rst with new links
* Move content of install-guide to the doc/source/install
* Minor changes
Depends-On: Ifc5512c0e2373cf3387e0e0498268eab092e52bb
Change-Id: Iecb4f60efb015a56b9b37331859848b287112842
The default neutron endpoint type is publicURL in neutronclient.
This patch replaces default neutron endpoint type from
internalURL to publicURL.
Change-Id: I8162b6178051df5f086488ecfb5d0bdc569ef9cd
Related-Bug: #1686298
The default glance endpoint type is publicURL in glanceclient.
This patch replaces default glance endpoint type from
internalURL to publicURL.
Change-Id: I39451ba89f191693475a694bd5c9045bf3bf539a
Related-Bug: #1686298
There is no action plan if creating audit was cancelled.
In this case, we just return none.
Change-Id: Ia7a93dab5978d181557d7dd7499e83655aec6f40
Closes-Bug: #1700901
In tempest, alias 'manager' has been moved to 'os_primary'
in version Pike, and it will be removed in version Queens.
This patch is to replace the usage of 'manager' with 'os_primary'.
For other details, please check [1] and [2]
[1] https://review.openstack.org/#/c/468036/
[2] https://review.openstack.org/#/c/463484/
Change-Id: I582758c42f61a85a2fd4aac5f7a97cd6021ecf68
When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.
This patch fix this problem. Refer patch set [1]
[1] https://review.openstack.org/#/c/189569
Change-Id: I6c0b896a6de1b7193dd4b77b6bc4433d0c75732d
Some of the available checks are disabled by default, like:
[H106] Don’t put vim configuration in source files
[H203] Use assertIs(Not)None to check for None
Change-Id: I369cff1c0f7f3cd3f5bcf3785b6904c9326c6759
When using 'query.filter_by(deleted_at=None)'
Will generate the incorrect SQL 'audits.deleted_at', as below:
SELECT * FROM action_plans INNER JOIN audits ON audits.id =
action_plans.audit_id WHERE audits.uuid = '' AND
audits.deleted_at IS NULL ORDER BY action_plans.id ASC
The correct filter field is 'action_plans.deleted_at'
Change-Id: Ie05f35233f78e82bc7af2c26c9effd62ea5f86ab
Closes-Bug: #1698720
The primary usage of "node.state" is wrong, it should be 'node.status'.
So correct it and refactor the method 'get_state_str'.
Change-Id: I9004c85bfb09f9b41fc56ecb5a56122d523a661f
Added strategy to identify and migrate a Noisy Neighbor - a low priority VM
that negatively affects peformance of a high priority VM by over utilizing
Last Level Cache.
Partially Implements: blueprint noisy-neighbor-strategy
Change-Id: Ia13f7e91a695024410b8f3b3d3d1646a0dd687d4
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the migration action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I02bff5db9bd06567bcc33b61a316c42c805bb20e
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the change_nova_service_state action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I09a03fff96d9555024a74ba255c6951affc39de8
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use JSONSchema
as our only JSON validation tool to keep consistence and also
to make it easier to expose the validation schema through
our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the resize action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I0ee4ba010a9f437658af81d5c971449aefc7f9c4
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the sleep action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: I3032490236536a11e7045a56ad0bd40ef979407e
Now in watcher,both JSONSchema and voluptuous are used to validate
JSON payloads. We want to remove voluptuous and Use
JSONSchema as our only JSON validation tool to keep consistence
and also to make it easier to expose the validation schema
through our API in future work.
In this patch, we replace voluptuous with JSONSchema to validate
the nop action in watcher applier.
Partially Implements: blueprint jsonschema-validation
Change-Id: Idf42b3359c36ac9480bd1f1bdd31e756214628ef
DeprecationWarning:
Using function/method 'oslo_messaging.transport.get_transport()'
is deprecated: use get_rpc_transport or get_notification_transport.
Replace get_transport with get_rpc_transport.
Change-Id: I13455842235a16463e61e5500c9e250a5cc9f86e
1.TypeError: watcher_non_live_migrate_instance() got an unexpected keyword
argument 'dest_hostname'
2.Conflict: Cannot 'stop' instance ****** while it is in vm_state stopped
(HTTP 409)
Closes-Bug: #1693434
Change-Id: I7293dd5d08e33c2e534d072da8592172bc438c9e
2017-05-25 07:27:26 +00:00
657 changed files with 37177 additions and 10909 deletions
The list of efficacy indicators associated to this action plan.
in:body
required:false
type:array
actionplan_global_efficacy:
description:|
The global efficacy of this action plan.
in:body
required:false
type:array
actionplan_hostname:
description:|
Hostname the actionplan is running on
in:body
required:false
type:string
actionplan_state:
description:|
State of this action plan. To get more information about states and
action plan's lifecycle, visit `Action Plan State Machine page <https://docs.openstack.org/watcher/latest/architecture.html#action-plan-state-machine>`_.
in:body
required:false
type:string
# Audit
audit_autotrigger:
description:|
Auto execute action plan once audit is succeeded.
in:body
required:false
type:boolean
audit_endtime_req:
description:|
The local time after which audit can't be executed.
It will be converted to UTC time by Watcher.
in:body
required:false
type:string
min_version:1.1
audit_endtime_resp:
description:|
The UTC time after which audit can't be executed.
in:body
required:false
type:string
min_version:1.1
audit_force:
description:|
Launch audit even if action plan is ongoing.
in:body
required:false
type:boolean
min_version:1.2
audit_goal:
description:|
The UUID or name of the Goal.
in:body
required:false
type:string
audit_hostname:
description:|
Hostname the audit is running on
in:body
required:false
type:string
audit_interval:
description:|
Time interval between audit's execution.
Can be set either in seconds or cron syntax.
Should be defined only for CONTINUOUS audits.
in:body
required:false
type:string
audit_name:
description:|
Name of this audit.
in:body
required:false
type:string
audit_next_run_time:
description:|
The next time audit launch. Defined only for CONTINUOUS audits.
in:body
required:false
type:string
audit_parameters:
description:|
The strategy parameters for this audit.
in:body
required:false
type:JSON
audit_starttime_req:
description:|
The local time after which audit can be executed in accordance
with interval. It will be converted to UTC time by Watcher.
in:body
required:false
type:string
min_version:1.1
audit_starttime_resp:
description:|
The UTC time after which audit can be executed in accordance with interval.
in:body
required:false
type:string
min_version:1.1
audit_state:
description:|
State of this audit. To get more information about states and
audit's lifecycle, visit `Audit State Machine page <https://docs.openstack.org/watcher/latest/architecture.html#audit-state-machine>`_.
in:body
required:true
type:string
audit_strategy:
description:|
The UUID or name of the Strategy.
in:body
required:false
type:string
audit_type:
description:|
Type of this audit. Can only be either ONESHOT or CONTINUOUS.
in:body
required:true
type:string
# Audit Template
audittemplate_description:
description:|
Short description of the Audit Template.
in:body
required:false
type:string
audittemplate_goal:
description:|
The UUID or name of the Goal.
in:body
required:true
type:string
audittemplate_name:
description:|
The name of the Audit template.
in:body
required:true
type:string
audittemplate_scope:
description:|
Audit Scope.
in:body
required:false
type:JSON
audittemplate_strategy:
description:|
The UUID or name of the Strategy.
in:body
required:false
type:string
audittemplate_uuid:
description:|
The UUID of the Audit template.
in:body
required:true
type:string
created_at:
description:|
The date and time when the resource was created. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
deleted_at:
description:|
The date and time when the resource was deleted. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
# Goal
goal_display_name:
description:|
Localized name of the goal.
in:body
required:true
type:string
goal_efficacy_specification:
description:|
Efficacy specifications as result of stategy's execution.
in:body
required:true
type:array
goal_name:
description:|
Name of the goal.
in:body
required:true
type:string
goal_uuid:
description:|
Unique UUID for this goal.
in:body
required:true
type:string
links:
description:|
A list of relative links. Includes the self and bookmark links.
in:body
required:true
type:array
# Data Model Node
node_disk:
description:|
The Disk of the node(in GiB).
in:body
required:true
type:integer
node_disk_ratio:
description:|
The Disk Ratio of the node.
in:body
required:true
type:float
node_hostname:
description:|
The Host Name of the node.
in:body
required:true
type:string
node_memory:
description:|
The Memory of the node(in MiB).
in:body
required:true
type:integer
node_memory_ratio:
description:|
The Memory Ratio of the node.
in:body
required:true
type:float
node_state:
description:|
The State of the node. The value is up or down.
in:body
required:true
type:string
node_uuid:
description:|
The Unique UUID of the node.
in:body
required:true
type:string
node_vcpu_ratio:
description:|
The Vcpu ratio of the node.
in:body
required:true
type:float
node_vcpus:
description:|
The Vcpu of the node.
in:body
required:true
type:integer
# Scoring Engine
scoring_engine_description:
description:|
A human readable description of the Scoring Engine.
in:body
required:true
type:string
scoring_engine_metainfo:
description:|
A metadata associated with the scoring engine
in:body
required:true
type:string
scoring_engine_name:
description:|
The name of the scoring engine.
in:body
required:true
type:string
# Data Model Server
server_disk:
description:|
The Disk of the server.
in:body
required:true
type:integer
server_memory:
description:|
The Memory of server.
in:body
required:true
type:integer
server_name:
description:|
The Name of the server.
in:body
required:true
type:string
server_state:
description:|
The State of the server.
in:body
required:true
type:string
server_uuid:
description:|
The Unique UUID of the server.
in:body
required:true
type:string
server_vcpus:
description:|
The Vcpu of the server.
in:body
required:true
type:integer
# Service
service_host:
description:|
The Name of host where service is placed on.
in:body
required:true
type:string
service_id:
description:|
The ID of service.
in:body
required:true
type:integer
service_last_seen_up:
description:|
The Time when Watcher service sent latest heartbeat.
in:body
required:true
type:string
service_name:
description:|
The Name of service like ``watcher-applier``.
in:body
required:true
type:string
service_status:
description:|
The State of service. It can be either in ACTIVE or FAILED state.
in:body
required:true
type:string
# Strategy
strategy_check_comment:
description:|
Requirement comment.
in:body
required:true
type:string
strategy_check_mandatory:
description:|
Whether this requirement mandatory or not.
in:body
required:true
type:boolean
strategy_check_state:
description:|
State of requirement for Strategy.
in:body
required:true
type:string or JSON
strategy_check_type:
description:|
Type of requirement for Strategy.
in:body
required:true
type:string
strategy_display_name:
description:|
Localized name of the strategy.
in:body
required:true
type:string
strategy_name:
description:|
Name of the strategy.
in:body
required:true
type:string
strategy_parameters_spec:
description:|
Parameters specifications for this strategy.
in:body
required:true
type:JSON
strategy_uuid:
description:|
Unique UUID for this strategy.
in:body
required:true
type:string
updated_at:
description:|
The date and time when the resource was updated. The date and time
stamp format is `ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
in:body
required:true
type:string
uuid:
description:|
The UUID for the resource.
in:body
required:true
type:string
# Version
version:
description:|
Versioning of this API response, eg. "1.1".
in:body
required:true
type:string
version_description:
description:|
Descriptive text about the Watcher service.
in:body
required:true
type:string
version_id:
description:|
Major API version, eg, "v1"
in:body
required:true
type:string
versions:
description:|
Array of information about currently supported versions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
