When retrieving the list of instances and volumes to propose a
solution, the zone migration strategy can raise an exception for
instance or volume not found, which will make the audit goes to a
failure state. This fix maintains the logic of listing all elements
directly from the client (nova) but now checks if the instance
is already in the model. The storage model check was already fixed
in another patch[1].
[1] cb6fb16097
Closes-Bug: #2098984
Assisted-By: Cursor (claude-3.5-sonnet)
Change-Id: I4c8993f051b797104172047eaae1fe1523eaf7eb
Signed-off-by: Douglas Viroel <viroel@gmail.com>
When only running volume migrations, a zone migration
strategy audit without setting compute_nodes should work.
Before this change, an audit with defined storage_pools,
no compute_nodes parameters, and with_attached_volume is set to True
would trigger the migration of the instances attached to the volumes
being migrated.
This patch decouples instance and volume migrations unless the user
explicitely asks for both. When migrating attached volumes, the zone
migration strategy will check for which instances should be migrated
according to the audit parameters, and if the instance the volume is
attached to can be migrated, it will be just after the volume.
On the other hand, when the attached instances should not be migrated
according to user input, only the volumes will be migrated.
In an audit that migrates instnaces but not volumes, the
with_attached_volume parameter will continue doing nothing.
Closes-Bug: 2111429
Change-Id: If641af77ba368946398f9860c537a639d1053f69
Signed-off-by: jgilaber <jgilaber@redhat.com>
Currently, when an audit with strategy zone_migration has added at least
one volume_migration action, it will not process the instances
migrations according to the definition of the `compute_nodes` parameter.
This behavior is unexpected according to the documentation of the
strategy.
This patch is fixing that behavior and making sure that not duplicated
actions are added to the solution, to handle the case where instances
migration actions are created when analyzing the volumes if the
`with_attached_volume` parameter is enabled. The patch is also removing
the method `instances_no_attached` which is not longer used.
Finally, it's adding some unit tests for the new method and fixing the
ones to cover the mixed instances and volumes migration situation.
Closes-Bug: #2109722
Change-Id: Ief7386ab448c2711d0d8a94a77fa9ba189c8b7d2
Signed-off-by: jgilaber <jgilaber@redhat.com>
Despite having the src_type paremeter for the storage_pool dictionary as
a mandatory parameter, the value is not being used to filter the volumes
to migrate, using only 'src_pool'.
This change makes 'src_type' optional, since it was ignored until this
point, making it optional keeps the same behaviour by default. If
'src_type' is in the audit parameters, the strategy uses both 'src_pool' and
'src_type' to filter the volumes to migrate.
Closes-Bug: 2111507
Change-Id: Id83a96de85ada1ae6c0e25f8b7fcf54034604911
Signed-off-by: jgilaber <jgilaber@redhat.com>
Add file to the reno documentation build to show release notes for
stable/2025.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2025.2.
Sem-Ver: feature
Change-Id: I21fd5f9a613e5e2ee81ae4fe34165f3f4a6ae479
Signed-off-by: OpenStack Release Bot <infra-root@openstack.org>
Generated-By: openstack/project-config:roles/copy-release-tools-scripts/files/release-tools/add_release_note_page.sh
CORS middleware needs to be added to api pipeline to support
Cross-Origin Resource Sharing(CORS). CORS is supported globally by
multiple OpenStack services but is not by watcher, due to lack of
CORS middleware and no mechanism to inject it into api pipeline.
Closes-Bug: #2122347
Change-Id: I6b47abe4f08dc257e9156b254fa60005b82898d7
Signed-off-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
In case standalone watcher-api runs behind forwarders (like load
balancers), it should parse specific request headers to determine
the endpoint url clients actually use.
Add http_proxy_to_wsgi middleware to api pipeline to handle this.
Closes-Bug: #2122353
Change-Id: I27ade17f7ce1649295f92f3ea1af620df63ba1bc
Signed-off-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
Request ID is essential in operating OpenStack services, especially
when troubleshooting some API problems. It allows us to find out
the log lines actually related to a specific request.
However watcher api hasn't returned it properly, so operators had no
way to determine the exact ID they should search.
Add RequestID middleware to return the id in X-OpenStack-Request-Id
header, which is globally used.
Closes-Bug: #2122350
Change-Id: Ie4a8307e8e7e981cedbeaf5fe731dbd47a50bade
Signed-off-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
Unlike Nova, Cinder does not support calling the 'os-migrate_volume'[1]
action without a host or a cluster. For volume migrations of type
'migrate' in watcher the dst_pool is required, but for other migrations
that migrate the volumes to different types is not needed. This
change checks if the dst_pool is defined and prevents some migrations
when it's misssing information.
Adds testing for creating audits with the Zone Migration status,
validating the schema changes.
[1] https://docs.openstack.org/api-ref/block-storage/v3/index.html#migrate-a-volume
Closes-Bug: 2108988
Change-Id: I305c58e47093c4a884e86f1d91fdc15ef2a1cfba
Signed-off-by: jgilaber <jgilaber@redhat.com>
By default Watcher enables only the compute model collector [1]. This
change enables the storage one as well, since otherwise when doing
volume migration the model quickly becomes obsolete if there are new
volumes created while an audit is running. The storage model is only
enabled if a cinder service is registered in keystone.
[1] https://docs.openstack.org/watcher/latest/configuration/watcher.html#collector.collector_plugins
Assisted-By: Cursor
Closes-Bug: 2111785
Change-Id: I864d3fc12d6364f1932cf5d2348a6b68169641e9
Signed-off-by: jgilaber <jgilaber@redhat.com>
The prelude provides a high-level overview of the
security improvements, operational enhancements,
and new monitoring capabilities for operators.
Assisted-By: claude-code
Change-Id: Ia2c1409d26aca0eddfb1685e9009305215c2405a
Signed-off-by: Sean Mooney <work@seanmooney.info>
Currently, when there is a volume_migrate action and migration_type is
`retype`, watcher assumes that the retype always triggers a migration
and checks the result of the retype based on the fields related to
the migration action (actually, it uses the same function to check the
result when `migration_type` is `retype` or `migrate`. This creates
problem in different scenarios:
- Actions keep in ONGOING status forever for volumes which have never
being migrated as the migration fields of the volume are empty.
- Actions which were migrated anytime before, still have the old values
so it may report the status of te retype actions wrongly.
This patch is implementing an entirely new function to check the result
of a retype action based on the final type and the status field of the
volume. This should be valid for any kind of retype action, with or
without migration. The criteria for successfull retype is that the type
for the volume is the destination one in the action and the status is
available or in-use.
Closes-Bug: #2112100
Change-Id: I76e91ed99e7a814a43a6dd906b6bcc150d471624
Signed-off-by: jgilaber <jgilaber@redhat.com>
Monasca is deprecated for removal. This change makes the Monasca client
an optional dependency and ensures it is only imported and instantiated
when the Monasca datasource is explicitly selected. This reduces the
default footprint while preserving functionality for deployments that
still rely on Monasca.
What changed
============
- requirements.txt: remove python-monascaclient from hard deps
- setup.cfg: add [options.extras_require] monasca extra
- watcher/common/clients.py: lazy import with clear UnsupportedError
- watcher/decision_engine/datasources/monasca.py: lazy client property
and deferred import of monascaclient.exc; reset on Unauthorized
- watcher/decision_engine/datasources/manager.py: unconditionally
import Monasca helper and include in metric_map; helper is lazy
- tests: conditionally include Monasca based on availability; adjust
expectations instead of skipping by default; avoid over-mocking
- tox.ini: enable optional extras via WATCHER_EXTRAS env var
- docs: datasources index notes Monasca is deprecated and optional
- releasenotes: upgrade note with install example and behavior
Why
===
- Allow deployments not using Monasca to run without the client
- Keep Monasca functional when explicitly installed via extras
- Provide clear operator guidance and smooth upgrades
Compatibility
=============
- No change for deployments that do not use Monasca
- Deployments using Monasca must install the optional extra:
pip install watcher[monasca]
Testing
=======
- Default: tox -e py3
- With Monasca: WATCHER_EXTRAS=monasca tox -e py3
Assisted-By: GPT-5 (Cursor)
Closes-Bug: #2120192
Change-Id: I7c02b74e83d656083ce612727e6da58761200ae4
Signed-off-by: Sean Mooney <work@seanmooney.info>
Fixed action status_message update restrictions to allow updates when
action is already in SKIPPED state. Previously, users could only update
the status_message when initially transitioning to SKIPPED state.
Changes include:
- Modified validation logic to allow status_message updates for SKIPPED actions
- Changed exception type from PatchError to Conflict for better semantics
- Added comprehensive test coverage for the new behavior
- Updated API documentation and samples
- Added release note documenting the fix
This enables administrators to fix typos, provide more detailed
explanations, or expand on reasons in action status messages after
the action has been skipped.
Generated-By: claude-code
Closes-Bug: #2121601
Change-Id: I64def708389a8ecd32080fba1638a4499ead349d
Signed-off-by: Sean Mooney <work@seanmooney.info>
These do not actually define timeout but interval. Rename the options
to reflect what they actually define. The existing deprecated options
in the [gnocchi_client] are also removed, because these have been kept
for 6 years.
In addition, fix inconsistent name (query vs call).
Change-Id: Ib29115746a25b45bdff1c3da8df9d7167c2db662
Signed-off-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
This patch extends compute model attributes by
adding new fields to Instance element. Values are
populated by nova the collector, using the same
nova list call, but requires a more recent compute
API microversion.
A new config option was added to allow users to
enable or disable the extended attributes and it is
disable by default.
Configure prometheus-based jobs to run on newer version
of nova api (2.96) and enables the extended attributes
collection.
Implements: bp/extend-compute-model-attributes
Assisted-By: Cursor (claude-4-sonnet)
Change-Id: Ibf31105d780dce510a59fc74241fa04e28529ade
Signed-off-by: Douglas Viroel <viroel@gmail.com>
Bug #2113862 details a number of suggested
corrections and additions to the Workload
Stabilization doc. This patch adds those
suggested changes.
Closes-Bug: #2113862
Assisted-By: Cursor (claude-3.5-sonnet)
Change-Id: I4131a304c064d2ea397b2447025c7edf69a56e2a
Signed-off-by: Ronelle Landy <rlandy@redhat.com>
This change enhances the Host Maintenance strategy by introducing
two new input parameters: `disable_live_migration` and
`disable_cold_migration`. These parameters allow cloud
administrators to control whether live or cold migration should be
considered during host maintenance operations.
If `disable_live_migration` is set, active instances will be cold
migrated if `disable_cold_migration` is not set, otherwise
active instances will be stopped. If `disable_cold_migration` is set,
inactive instances will not be cold migrated.
If both are set, only stop actions will be performed on instances.
The strategy logic and action plan generation have been updated to
reflect these behaviors. A new "stop" action is introduced and
registered, and the weight planner is updated to handle new action.
Documentation for the Host Maintenance strategy is updated to
describe the new parameters and their effects.
Test Plan:
- Unit tests for HostMaintenance strategy with new parameters
- Integration tests for action plan generation with stop action
This implements the specification:
Spec: https://review.opendev.org/c/openstack/watcher-specs/+/943873
Change-Id: I201b8e5c52e1bc1a74f3886a0e301e3c0fa5d351
Signed-off-by: Quang Ngo <quang.ngo@canonical.com>
Fixes the microversion comparison in both enable and
disable nova-compute service methods in NovaHelper.
The previous implementation was incorrect and started to
fail for microversion greather than 2.99.
Closes-Bug: #2120586
Assisted-By: Cursor (claude-4-sonnet)
Change-Id: I69da7f10cd5b42f7d4613d8947bca3e382815c3f
Signed-off-by: Douglas Viroel <viroel@gmail.com>
This patch implements the changes in the API required for the
skipped action blueprint. It includes:
- New field `status_message` is visible in API get calls for Audits,
ActionPlans and Audits.
- New Patch call is added to `/actions/{action_id}` which allows to
manually move actions in PENDING state to SKIPPED for ActionPlans
which have not been started.
- A new API microversion 1.5 is added for these changes.
It also adds requried tests and documentation.
Implements: blueprint add-skip-actions
Assisted-By: Cursor (claude-4-sonnet)
Change-Id: I71fb9af76085e5941a7fd3e9e4c89d6f3a3ada47
Signed-off-by: Alfredo Moralejo <amoralej@redhat.com>
In order to test the different code paths for action execution
it is very useful to be able to make the actions fail in the different
execution stages.
This patch adds three new options `fail_pre_condition`, `fail_execute`
and `fail_post_condition`. Setting any of them to True makes the action
to fail in the specified step.
Change-Id: Ied8c0bb767d9bb6bdfb9209365857a3b4d606b40
Signed-off-by: Alfredo Moralejo <amoralej@redhat.com>
This change removes watchers in tree functionality
for swapping instance volumes and defines swap as an alias
of cinder volume migrate.
The watcher native implementation was missing error handling
which could lead to irretrievable data loss.
The removed code also forged project user credentials to
perform admin request as if it was done by a member of a project.
this was unsafe an posses a security risk due to how it was
implemented. This code has been removed without replacement.
While some effort has been made to allow existing
audits that were defined to work, any reduction of functionality
as a result of this security hardening is intentional.
Closes-Bug: #2112187
Change-Id: Ic3b6bfd164e272d70fe86d7b182478dd962f8ac0
Signed-off-by: Sean Mooney <work@seanmooney.info>
Implement the spec for multi-tenancy support for metrics. This adds
a new 'Aetos' datasource very similar to the current Prometheus
datasource. Because of that, the original PrometheusHelper class
was split into two classes and the base class is used for
PrometheusHelper and for AetosHelper. Except for the split, there
is one more change to the original PrometheusHelper class code, which
is the addition and use of the _get_fqdn_label() and
_get_instance_uuid_label() methods.
As part of the change, I refactored the current prometheus datasource
unit tests. Most of them are now used to test the PrometheusBase class
with minimal changes. Changes I've made to the original tests:
- the ones that can be be used to test the base class are moved into the
TestPrometheusBase class
- the _setup_prometheus_client, _get_instance_uuid_label and
_get_fqdn_label functions are mocked in the base class tests.
Their concrete implementations are tested in each datasource tests
separately.
- a self._create_helper() is used to instantiate the helper class with
correct mocking.
- all config value modification is the original tests got moved out and
instead of modifying the config values, the _get_* methods are mocked
to return the wanted values
- to keep similar test coverage, config retrieval is tested for each
concrete class by testing the _get_* methods.
New watcher-aetos-integration and watcher-aetos-integration-realdata
zuul jobs are added to test the new datasource. These use the same set
of tempest tests as the current watcher-prometheus-integration jobs.
The only difference is the environment setup and the Watcher config,
so that the job deploys Aetos and Watcher uses it instead of accessing
Prometheus directly.
At first this was generated by asking cursor to implement the linked spec
with some additional prompts for some smaller changes. Afterwards I manually
went through the code doing some cleanups, ensuring it complies with
PEP8 and hacking and so on. Later on I manually adjusted the code to use
the latest observabilityclient changes.
The zuul job was also mostly generated by cursor.
Implements: https://blueprints.launchpad.net/watcher/+spec/prometheus-multitenancy-support
Generated-By: Cursor with claude-4-sonnet model
Change-Id: I72c2171f72819bbde6c9cbbf565ee895e5d2bd53
Signed-off-by: Jaromir Wysoglad <jwysogla@redhat.com>
Some response parameters from GET /infra-optim/v1/data_model
endpoint are missing from api-ref documentation. This patch
updates the doc to include them.
For more details see, LP #2117726
Closes-Bug: #2117726
Change-Id: Iaa775f56bb8167d9c6b458cd07f1ec3cefaf70fe
Signed-off-by: Douglas Viroel <viroel@gmail.com>
With the events of eventlet removal, Watcher will need
to be adapted to support both modes, eventlet and threading, for
a couple of releases before removing all eventlet code.
This patch adds methods and classes that allow decision engine
modules to create futurist thread pools instead of green thread pools,
based on a environment variable that can be enabled by service.
It moves continuous audit handler instance to decison engine service,
so it can be started together with the main decision engine service.
Adds an environment variable that allows the user to disable
eventlet monkey patching and to use oslo.service threading backend.
Change-Id: I8a8be0a7cebdc44005fd77ec960543828c7da318
Signed-off-by: Douglas Viroel <viroel@gmail.com>
This cr fixes:
* Replaced ``dateutil.tz.tzlocal()`` and ``dateutil.tz.tzutc()`` with
``datetime.timezone`` built-in classes in audit controllers and
continuous audit scheduling.
* Replaced ``dateutil.parser.parse()`` with
``oslo_utils.timeutils.parse_isotime()`` in the zone migration
strategy for parsing datetime strings.
Closes-Bug: #2118404
Change-Id: I6d8a345fa4339a688769b147413dcdf3016bf4a0
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
The last release of openstack to support python 3.9
was 2025.1 (epoxy), with this change watcher now requires
3.10, testing of 3.9 was removed in previous commits.
Change-Id: Ida53740293e93b0c20dec2e175b390fa18bed852
Signed-off-by: Sean Mooney <work@seanmooney.info>
The following exception was added in initial import of watcher
code base[1].
In each of the controller REST APIs, it was called with a flag
stating request was coming from top level resources apis.
But this exception and code was not used anywhere in the
rest api. It seems to be a dead code. So, it needs to be
cleaned up.
Note: In audit_template, under patchapi, this exception
was used for not removal goal from audit template.
Since this cr drops this exception, It replace the same
with NotAuthorized exception keeping status code same.
Links:
[1]. d14e057da1 (diff-6d510a275605e20ba8b435157062da2b749265a88a3cfd6d90abb7e8e5feac2aR235)
Closes-Bug: #2115968
Change-Id: I82a5e4a7a51726b3a89257c84a75157fbfcb82eb
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
These apis are not implemented with in the watcher code base and
was marked as a forbidden to use.
It does not make sense to keep these api as they are not implemented.
This cr drops the code around that to make the action apis cleaner.
Closes-Bug: #2110895
Change-Id: I0f465157e6cd481b27665ca6016db68c198cebeb
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
