The apscheduler background scheduler spawns a native thread
which is not monkey patched which interacts with shared module
level objects like the module level LOG instances and sqlachmey
engine facades.
This is unsafe and leads to mixing patched and unpatched
code in the same thread.
This manifests in 2 ways:
1.) https://paste.opendev.org/show/bGPgfURx1cZYOsgmtDyw/
sqlalchmey calls can fail due to a time.sleep(0) in oslo.db being invoked
using the unpatched time modules in an eventlet greenthrad.
2.) https://paste.opendev.org/show/b5C2Zz4A4BFIGbKLKrQU/
over time that caused the sqlalchmy connection queuepool to fill up preventing
backgound tasks form running like reconsiling audits.
This change adresses this by overloading the background scheduler _main_loop
to monkey patch the main loop if the calling thread was monkey patched.
Closes-Bug: #2086710
Change-Id: I672c183274b0a17cb40d7b5ab8c313197760b5a0
This commit removes the execute bit from several files
and remove the shebang lines from the devstack plugin.
While the devstack plugin is written in bash, it is not an executable
script. The devstack plugin is sourced by devstack as needed,
as such it is not executed in a subshell and the #!/bin/bash
lines are not used even when present.
Change-Id: I82ca22b7a47bf267fe6cf11f3e3519510108c146
This chanage enabled codespell in precommit and
fixes the existing typos.
A followup commit will enable this in tox and ci.
Change-Id: I0a11bcd5a88247a48d3437525fc8a3cb3cdd4e58
At the moment, Watcher can use a single bare metal provisioning
service: Openstack Ironic.
We're now adding support for Canonical's MAAS service [1], which
is commonly used along with Juju [2] to deploy Openstack.
In order to do so, we're building a metal client abstraction, with
concrete implementations for Ironic and MAAS. We'll pick the MAAS
client if the MAAS url is provided, otherwise defaulting to Ironic.
For now, we aren't updating the baremetal model collector since it
doesn't seem to be used by any of the existing Watcher strategy
implementations.
[1] https://maas.io/docs
[2] https://juju.is/docs
Implements: blueprint maas-support
Change-Id: I6861995598f6c542fa9c006131f10203f358e0a6
Python introduced http.HTTPStatus since version 3.5,
and Wallaby has targeted a minimum version of python 3.6.
Change-Id: I45f732f0f59b8fae831bb6c07f4fdd98cdd7409a
There is a commonly shared and proven rpc pattern used
across most OpenStack services that is already implemented
in watcher, but the functions are not used.
This patch basically makes use of the existing
rpc classes and removes some unnecessary code.
Change-Id: I57424561e0675a836d10b712ef1579a334f72018
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('watcher/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Co-Authored-By: Sean McGinnis <sean.mcginnis@gmail.com>
Change-Id: Icf35d3a6c10c529e07d1a4edaa36f504e5bf553a
We want to set the value of uuid field of Watcher ComputeNode
to hypversion id(as uuid). So we need to get hypervisor
information by uuid.
Change-Id: I752fbfa560313e28e87d83e46431c283b4db4f23
Related-Bug: #1835192
openstack hypervisor list contains ironic nodes. we should
filter out baremetal nodes when get compute node list.
Change-Id: I4ab3e1a63dc6f61cdc3e99fa2cae749a711459cc
Closes-Bug: #1835183
This patch added Placement to Watcher
We plan to improve the data model and strategies in
the future specs.
Change-Id: I7141459eef66557cd5d525b5887bd2a381cdac3f
Implements: blueprint support-placement-api
The get_compute_node_by_hostname method is given a
compute service hostname and then does two queries to
find the matching hypervisor (compute node) with details:
1. List hypervisors with details and find the one that
matches the given compute service hostname.
2. Using that node, search for hypervisors with the
matching hypervisor_hostname.
There are two issues here:
1. The first query is inefficient in that it has to list
all hypervisors in the deployment to try and match the
one with the compute service hostname client side.
2. The second query is a fuzzy match on the server side [1]
so even though we have matched on the node we want,
get_compute_node_by_name can still return more than
one hypervisor which will result in the helper method
raising ComputeNodeNotFound. Consider having compute
hosts with names compute1, compute10, compute11, compute100,
and so on. The fuzzy match on compute1 would return all of
those hypervisors.
For non-ironic nodes in nova, the compute service host and
hypervisor should be 1:1, meaning the hypervisor.service['host']
should be the same as hypervisor.hypervisor_hostname. Knowing
this, we can simplify the code to search just on the given
compute service hostname and if we get more than one result, it
is because of the fuzzy match and we can then do our client-side
filtering on the compute service hostname.
[1] https://github.com/openstack/nova/blob/d4f58f5eb/nova/db/sqlalchemy/api.py#L676
Change-Id: I84f387982f665d7cc11bffe8ec390cc7e7ed5278
The nova CDM builder code and notification handling
code had some inefficiencies when it came to looking
up a hypevisor to get details. The general pattern
used before was:
1. get the minimal hypervisor information by hypervisor_hostname
2. make another query to get the hypervisor details by id
In the notifications case, it was actually three calls because
the first is listing hyprvisors to filter client-side by service
host.
This change collapses 1 and 2 above into a single API call
to get the hypervisor by hypervisor_hostname with details
which will include the service (compute) host information
which is what get_compute_node_by_id() was being used for.
Now that nothing is using get_compute_node_by_id it is removed.
There is more work we could do in get_compute_node_by_hostname
if the compute API allowed filtering hypervisors by service
host so a TODO is left for that.
One final thing: the TODO in get_compute_node_by_hostname about
there being more than one hypervisor per compute service host
for vmware vcenter is not accurate - nova's vcenter driver
hasn't supported a host:node 1:M topology like that since the
Liberty release [1]. The only in-tree driver in nova that supports
1:M is the ironic baremetal driver, so the comment is updated.
[1] Ifc17c5049e3ed29c8dd130339207907b00433960
Depends-On: https://review.opendev.org/661785/
Change-Id: I5e0e88d7b2dd1a69117ab03e0e66851c687606da
This does two things:
1. Rather than make an API call per server on the host,
get all of the servers in a single API call by
filtering on the host. The os-hypervisors API results
to use make this require a bit of refactoring since
get_compute_node_by_name does not have the service
entry in it and get_compute_node_by_id does not have the
servers entry in it. A TODO is added to clean that up
with a single call to os-hypervisors once we have the
support in python-novaclient.
2. Pulls get_node_by_uuid() out of the loop.
A test is added for the nova_helper get_instance_list method
since one did not exist before.
The fake compute node mocks in test_nova_cdmc_execute are
also cleaned up since, as noted above, get_compute_node_by_name
and get_compute_node_by_id don't both return all the details.
Change-Id: Ifd9f83c2f399d4c1765b0c520f4d5a62ad0f5fbd
The [nova_client]/api_version defaults to 2.56 since
change Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1. There
is compatibility code for that change but if 2.56 is
not available watcher_non_live_migrate_instance will
still fail if a destination host is used.
Since 2.56 has been available since the Queens version of
nova it should be reasonable to require at least that
version of nova is running for using Watcher.
This adds code which enforces the minimum version along
with a release note and "watcher-status upgrade check"
check method.
Note that it's kind of weird for watcher to have a config
option like nova_client.api_version since compute API
microversions are per API request even though novaclient
is constructed with the single configured version. It should
really be something the client (watcher in this case) determines
using version discovery and gracefully enables features if
the required nova API version is available, but that's a bigger
change.
Change-Id: Id34938c7bb8a5ca934d997e52cac3b365414c006
When use tox-elower-constraint, we will meet these errors:
1. line 417, in test_clients_ironic:
b"AssertionError: Expected call: Client('1', endpoint_override...
b"Actual call: Client('1', 'http://localhost:6385/'...
2. line 39, in test_wrong_major_version:
b"KeyError: 'HTTP_ACCEPT'"
3. RUN END RESULT_TIMED_OUT:
[untrusted : git.openstack.org/openstack-infra/
zuul-jobs/playbooks/tox/run.yaml@master]
For the first error, The reason is that the unittest for the
ironicclient is too strict and must be adapted to the latest code.
In fact, the watcher can use the previous ironicclient version.
Therefore, we modified the unittest so that the watcher does not
have to rely on the latest ironicclient version.
For the second error, The reason is that we need to update the minimum
version of pecan and webOb.
For the third error, the reason is that the version of the oslo_messaging
is too low.
Change-Id: Icb3eda3d27fa4452e13e2dcd3c016cc76fc2c7c7
With these brackets, the statement "raise Exception("Live migration
execution.....") in watcher/common/nova_helper.py, line 379 will never
be executed. So remove it and let the statement make sense.
Change-Id: I42a2fa0c8ffa9c84a918d432c5093470dbd80f82
This patch fixes watcher.tests.common.test_clients.TestClients.
test_clients_ironic unit test error due to python-ironicclient 2.4.0.
Change-Id: I0aaa2047cc8fcf09dee8bc7168e35f7a58c10125
Now that the v2.0 API has been removed, we don't have a reason to
include deployment instructions for two separate applications on
different ports.
Related-bug: #1754104
Change-Id: I98fae626d39cb62ad51c86435c1a2c60be5c1fb9
Since Nova API v2.56, Nova migrate Server(migrate Action) API
has host option.
This patch replaces cold migration implementation to use the API.
Change-Id: Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1
Implements: blueprint replace-cold-migrate-to-use-nova-migration-api
Migration action 'cold' is not intuitive for the developers and users,
so this patch replaces it with ‘migrate’ and 'retype'.
Change-Id: I58acac741499f47e79630a6031d44088681e038a
Closes-Bug: #1733247
Storage pool can have many volume types,
'get_volume_type_by_backendname' should return a list of types.
Closes-Bug: #1733257
Change-Id: I877d5886259e482089ed0f9944d97bb99f375824
If server migration fails, the while loop will not break
until retry to 0,
we can check the task_state to avoid this situation.
Closes-Bug: #1728476
Change-Id: I07e1048eb736263a261456ee78c96fee9db13cb5
add 'disabled_reason' filed into 'ComputeNode' resource, to distinguish
which nodes are disabled by Watcher and which are not by Watcher.
Implements:blueprint extend-node-status
Change-Id: I7175f14870834a4582e45309529d7e8d9fbb2e6f
This Patch fixes gate failure, encountered in recent version
of oslo_messaging.
Change-Id: I6d8ab882a7c157dcf4f78c805a4ce2d9b1fa3f14
Closes-Bug: #1716476
Gnocchiclient uses keystoneauth1.adapter so that adapter_options
need to be given.
This patch fixes gnocchiclient creation.
Change-Id: I6b5d8ee775929f4b3fd30be3321b378d19085547
Closes-Bug: #1714871
