Set the default interface for keystone_client to public in the watcher
conf instead of admin.
Closes-Bug: 2109494
Change-Id: I9e0289249981ca965190df6dbdc37e09fd0951d7
In order to support vm_workload_consolidation, workload_balance and
workload_stabilization strategis some instance metrics are required.
This patch is adding support for them.
Implementation is based on a prometheus store populated using sg-core
from ceilometer metrics with Pollster source.
- instance_ram_usage: rely on ceilometer_memory_usage metrics created from
ceilometer memory.usage meter.
- instance_ram_allocated: rely on the memory value provided by the
inventory created from nova and placement APIs.
- instance_cpu_usage: rely on ceilometer_cpu metric created from
ceilometer cpu meter. A max value of 100 is set in the query.
- instance_root_disk_size: rely on the `disk` value provided by the
inventory created from nova and placement APIs.
A new parameterer `instance_uuid_label` has been added to the prometheus
datasource configuration to identify the label used to store the value of the
OpenStack instance uuid for eache instance metric in prometheus. Default
value is `resource`.
Change-Id: I2f2b56aa002014e511a5e48398ef1da43fc4f5e2
This adds a new data source for the Watcher decision engine that
implements the watcher.decision_engine.datasources.DataSourceBase.
related spec was merged at [1].
Implements: blueprint prometheus-datasource
[1] https://review.opendev.org/c/openstack/watcher-specs/+/933300
Change-Id: I6a70c4acc70a864c418cf347f5f6951cb92ec906
This datasource requires Ceilometer API which was already removed some
years ago. The implementation should have been removed when dependency
on ceilometerclient was removed by [1].
Also remove some job definitions which are not actually used.
[1] 01d74d0a87
Change-Id: I29c3865dc1207f1bbbb266e4217cf8888afebfb6
This commit removes the execute bit from several files
and remove the shebang lines from the devstack plugin.
While the devstack plugin is written in bash, it is not an executable
script. The devstack plugin is sourced by devstack as needed,
as such it is not executed in a subshell and the #!/bin/bash
lines are not used even when present.
Change-Id: I82ca22b7a47bf267fe6cf11f3e3519510108c146
This chanage enabled codespell in precommit and
fixes the existing typos.
A followup commit will enable this in tox and ci.
Change-Id: I0a11bcd5a88247a48d3437525fc8a3cb3cdd4e58
At the moment, Watcher can use a single bare metal provisioning
service: Openstack Ironic.
We're now adding support for Canonical's MAAS service [1], which
is commonly used along with Juju [2] to deploy Openstack.
In order to do so, we're building a metal client abstraction, with
concrete implementations for Ironic and MAAS. We'll pick the MAAS
client if the MAAS url is provided, otherwise defaulting to Ironic.
For now, we aren't updating the baremetal model collector since it
doesn't seem to be used by any of the existing Watcher strategy
implementations.
[1] https://maas.io/docs
[2] https://juju.is/docs
Implements: blueprint maas-support
Change-Id: I6861995598f6c542fa9c006131f10203f358e0a6
It has costs when rollback action_plan.
So give users an option whether to rollback it
when the action_plan fails.
Change-Id: I20c0afded795eda7fb1b57ffdd2ae1ca36c45301
Implements the singleton general purpose threadpool for the decision
engine and associated tests.
A threadpool is a collection of one or more threads typically called
'workers' to which tasks can be submitted. These submitted tasks will
be scheduled by the threadpool and subsequently executed. How many
tasks will be executed concurrently is managed by the underlying
threadpool and its configuration. In Python the submission of tasks
to a threadpool returns an object called a 'future'. Futures provide
a method to interface with the task being executed that allows to
retrieve information about its state. Such as if it currently is being
executed, if it is waiting on a condition and if it has completed
succesfully. Finally, futures allow to retrieve what has been returned
by the submitted task.
In the case of most OpenStack projects instead of interfacing with native
Python concurrency the futurist library is used. This library provides
very similar interfaces to native concurrency with some extras such as
the wait_for_any method.
For more information about futurist or Python concurrency the following
references can be consulted:
https://docs.python.org/3/library/concurrent.futures.htmlhttps://docs.openstack.org/futurist/latest/reference/index.html#executors
Partially Implements: blueprint general-purpose-decision-engine-threadpool
Change-Id: I94bd9a17290967f011762f2b9c787ee7c46ff930
Add call_retry method for ModelBuilder classes along with configuration
options. This allows ModelBuilder classes to reattempt any failed calls
to external services such as Nova or Ironic.
Change-Id: Ided697adebed957e5ff13b4c6b5b06c816f81c4a
Actually list_opts() return a list like[1], So we don't need to
convert list to dict and then convert to list[2].
The reason why we need to convert it before is to put together
the same group of configuration objects, but we don't need it
actually.
Now, the list_opts()'s result like this[3].
Reference:
[1]. [(Group1,[cfgObj1,cfgObj2....]),(Group2,[cfgObj3,cfgObj3....])..]
[2]. 375ae32fad/watcher/conf/opts.py (L51-L52)
[3]. [(Group1,[cfgObj1]),(Group1,[cfgObj2]),(Group2,[cfgObj3,cfgObj3....])..]
Change-Id: I50fcc5f812be42038852662639fb10c6dd2f6f72
The datasources are only used by the decision_engine, however, they
are placed in a directory one level higher. This patch moves the
datasources code into the decision_engine folder.
Change-Id: Ia54531fb899b79a59bb77adea079ff27c0d518fa
This is a follow-up to: https://review.opendev.org/#/c/666897/
and makes sure titles and help information get rendered in
the configuration documentation and configuration samples.
The options for the placement_client group are already changed
and left untouched as a result. The changes to grafana_client
are already done in another patch and also untouched.
Change-Id: Ia33cd4576e4b55e651f3f3779a01f2867126138d
This improves the documentation on configuration parameters for the
Grafana datasource.
Follow-up: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Depends-on: I5d1d3129b5d225f0f2fc86d149c046f9aab94d47
Change-Id: Ifd8be7491669c429482d880fdf0219be5ef03163
This implements the configuration parameters to implement
Grafana as a datasource including the influxdb translator
Change-Id: If1f27dc01e853c5b24bdb21f1e810f64eaee2e5c
Partially-implements: blueprint grafana-proxy-datasource
Moves the query_retry method into the baseclass and makes the query
retry and timeout options part of the watcher_datasources config group.
This makes the query_retry behavior uniform across all datasources.
A new baseclass method named query_retry_reset is added so datasources
can define operations to perform when recovering from a query error.
Test cases are added to verify the behavior of query_retry.
The query_max_retries and query_timeout config parameters are
deprecated in the gnocchi_client group and will be removed in a future
release.
Change-Id: I33e9dc2d1f5ba8f83fcf1488ff583ca5be5529cc
This patch added Placement to Watcher
We plan to improve the data model and strategies in
the future specs.
Change-Id: I7141459eef66557cd5d525b5887bd2a381cdac3f
Implements: blueprint support-placement-api
The [nova_client]/api_version defaults to 2.56 since
change Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1. There
is compatibility code for that change but if 2.56 is
not available watcher_non_live_migrate_instance will
still fail if a destination host is used.
Since 2.56 has been available since the Queens version of
nova it should be reasonable to require at least that
version of nova is running for using Watcher.
This adds code which enforces the minimum version along
with a release note and "watcher-status upgrade check"
check method.
Note that it's kind of weird for watcher to have a config
option like nova_client.api_version since compute API
microversions are per API request even though novaclient
is constructed with the single configured version. It should
really be something the client (watcher in this case) determines
using version discovery and gracefully enables features if
the required nova API version is available, but that's a bigger
change.
Change-Id: Id34938c7bb8a5ca934d997e52cac3b365414c006
Override the metric map of each datasource as soon as it is created by
the manager. This override comes from a file whose path is provided by
a setting in config file.
Loading at creation time allows the correct datasource be used when
get_backend is called, this allows loading a datasource whose metric
names get updated outside the watcher's codebase.
The function 'load_metric_map' returns empty-dict in any error case.
Also in case the file is empty where safe_load is unable finds any
yaml documents, it will return None. [1]
Some minor refactoring in the test_manager file for readability and
added tests for file load and metric override.
1 - https://pyyaml.org/wiki/PyYAMLDocumentation
Change-Id: I1df16245f4c7dfd34066f3ab0553cd67154faa58
Implements: blueprint file-based-metric-map
Some users may want to create keystoneclient by specifying the
type of endpoint and region name, so we need to supply the option
for user to choose.
Implements: blueprint support-keystoneclient-option
Change-Id: I49b33a69ec99d2a91568ce27ef89dc80b75e7091
This enhances the [collector]/collector_plugins
config option help text to mention the storage
and baremetal in-tree collectors and the ability
to load out-of-tree collectors via extension point.
While doing this, the help text is formatted for
prettier rst rendering in the docs.
Change-Id: Ifd32c95c664c4e9586c250e6bceaeaba2e2df417
Allows to define a global preference for metric datasources with the
ability for strategy specific overrides. In addition, strategies which
do not require datasources have the config options removed this is
done to prevent confusion.
Some documentation that details the inner workings of selecting
datasources is updated.
Imports for some files in watcher/common have been changed to resolve
circular dependencies and now match the overall method to import
configuration.
Addtional datasources will be retrieved by the manager if the
datasource throws an error.
Implements: blueprint global-datasource-preference
Change-Id: I6fc455b288e338c20d2c4cfec5a0c95350bebc36
Ceilometer Datasource has been deprecated since its API has been
deprecated in Ocata cycle. Watcher has supported Ceilometer for some
releases after Ocata to let users migrate to Gnocchi/Monasca datasources.
Since U-release, Ceilometer support will be removed.
Change-Id: I944a5a562ab09a36961eb9b75e9a5144ba0b9ca4
This commit allows to consume notifications via notifications pools[1].
Listeners in notification pools recieves a copy of notification. It
will let Watcher to sync Data Models of Decision Engines.
[1]: https://docs.openstack.org/oslo.messaging/ocata/notification_listener.html
Change-Id: Ie37528263181924f84510500fc1277b0237c1df8
Partially-Implements: blueprint support-watcher-ha-active-active-mode
Since Nova API v2.56, Nova migrate Server(migrate Action) API
has host option.
This patch replaces cold migration implementation to use the API.
Change-Id: Idd6ebc94f81ad5d65256c80885f2addc1aaeaae1
Implements: blueprint replace-cold-migrate-to-use-nova-migration-api
New releases of oslo.config support a 'mutable' parameter to Opts.
Configuration options are mutable if their oslo.config Opt's
mutable=True is set. This mutable setting is respected when the oslo
method mutate_config_files is called instead of reload_config_files.
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e allows services to tell
oslo.service they want mutate_config_files to be called by specifying
the 'restart_method=mutate' parameter, what this patch does.
The default mutable configuration options (set by oslo.config Opts'
mutable=True) are:
- [DEFAULT]/pin_release_version
- [DEFAULT]/debug
- [DEFAULT]/log_config_append
Concrete params, that made mutable in Watcher:
* watcher_decision_engine.action_plan_expiry
* watcher_decision_engine.check_periodic_interval
* watcher_decision_engine.continuous_audit_interval
* gnocchi_client.query_max_retries
* gnocchi_client.query_timeout
* DEFAULT.periodic_interval
Change-Id: If28f2de094d99471a3ab756c947e29ae3d8a28a2
Implements: bp mutable-config
