Imported Translations from Zanata

For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I2b2afb0c0e590b737871bf4c43293df2ed88e534
SQLAlchemy 2.0: Omnibus fixes patch
2024-06-01 02:47:52 +00:00 · 2024-05-29 06:49:32 -07:00 · 2024-03-27 09:41:23 +00:00 · 2024-01-09 19:22:04 -08:00 · 2023-12-11 10:21:33 +00:00 · 2023-12-01 01:51:39 +00:00
65 changed files with 1693 additions and 694 deletions
--- a/.gitreview
+++ b/.gitreview
@@ -2,4 +2,3 @@
 host=review.opendev.org
 port=29418
 project=openstack/watcher.git
-defaultbranch=stable/xena
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -3,7 +3,7 @@
    templates:
      - check-requirements
      - openstack-cover-jobs
-      - openstack-python3-xena-jobs
+      - openstack-python3-jobs
      - publish-openstack-docs-pti
      - release-notes-jobs-python3
    check:
@@ -89,7 +89,7 @@
 - job:
    name: watcher-tempest-multinode
    parent: watcher-tempest-functional
-    nodeset: openstack-two-node-focal
+    nodeset: openstack-two-node-jammy
    roles:
      - zuul: openstack/tempest
    group-vars:
@@ -107,6 +107,7 @@
          watcher-api: false
          watcher-decision-engine: true
          watcher-applier: false
+          c-bak: false
          ceilometer: false
          ceilometer-acompute: false
          ceilometer-acentral: false
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -56,8 +56,8 @@ source_suffix = '.rst'
 master_doc = 'index'

 # General information about the project.
-project = u'Watcher'
-copyright = u'OpenStack Foundation'
+project = 'Watcher'
+copyright = 'OpenStack Foundation'

 # A list of ignored prefixes for module index sorting.
 modindex_common_prefix = ['watcher.']
@@ -91,14 +91,14 @@ pygments_style = 'native'
 # List of tuples 'sourcefile', 'target', u'title', u'Authors name', 'manual'

 man_pages = [
-    ('man/watcher-api', 'watcher-api', u'Watcher API Server',
-     [u'OpenStack'], 1),
-    ('man/watcher-applier', 'watcher-applier', u'Watcher Applier',
-     [u'OpenStack'], 1),
+    ('man/watcher-api', 'watcher-api', 'Watcher API Server',
+     ['OpenStack'], 1),
+    ('man/watcher-applier', 'watcher-applier', 'Watcher Applier',
+     ['OpenStack'], 1),
    ('man/watcher-db-manage', 'watcher-db-manage',
-     u'Watcher Db Management Utility', [u'OpenStack'], 1),
+     'Watcher Db Management Utility', ['OpenStack'], 1),
    ('man/watcher-decision-engine', 'watcher-decision-engine',
-     u'Watcher Decision Engine', [u'OpenStack'], 1),
+     'Watcher Decision Engine', ['OpenStack'], 1),
 ]

 # -- Options for HTML output --------------------------------------------------
@@ -128,8 +128,8 @@ openstackdocs_bug_tag = ''
 latex_documents = [
    ('index',
     'doc-watcher.tex',
-     u'Watcher Documentation',
-     u'OpenStack Foundation', 'manual'),
+     'Watcher Documentation',
+     'OpenStack Foundation', 'manual'),
 ]

 # If false, no module index is generated.
--- a/doc/source/configuration/configuring.rst
+++ b/doc/source/configuration/configuring.rst
@@ -372,7 +372,7 @@ You can configure and install Ceilometer by following the documentation below :
 #. https://docs.openstack.org/ceilometer/latest

 The built-in strategy 'basic_consolidation' provided by watcher requires
-"**compute.node.cpu.percent**" and "**cpu_util**" measurements to be collected
+"**compute.node.cpu.percent**" and "**cpu**" measurements to be collected
 by Ceilometer.
 The measurements available depend on the hypervisors that OpenStack manages on
 the specific implementation.
--- a/doc/source/contributor/plugin/strategy-plugin.rst
+++ b/doc/source/contributor/plugin/strategy-plugin.rst
@@ -300,6 +300,6 @@ Using that you can now query the values for that specific metric:
 .. code-block:: py

    avg_meter = self.datasource_backend.statistic_aggregation(
-        instance.uuid, 'cpu_util', self.periods['instance'],
+        instance.uuid, 'instance_cpu_usage', self.periods['instance'],
        self.granularity,
        aggregation=self.aggregation_method['instance'])
--- a/doc/source/strategies/basic-server-consolidation.rst
+++ b/doc/source/strategies/basic-server-consolidation.rst
@@ -26,8 +26,7 @@ metric                       service name plugins comment
                                                  ``compute_monitors`` option
                                                  to ``cpu.virt_driver`` in
                                                  the nova.conf.
-``cpu_util``                 ceilometer_  none    cpu_util has been removed
-                                                  since Stein.
+``cpu``                      ceilometer_  none
 ============================ ============ ======= ===========================

 .. _ceilometer: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#openstack-compute
--- a/doc/source/strategies/saving_energy.rst
+++ b/doc/source/strategies/saving_energy.rst
@@ -89,9 +89,9 @@ step 2: Create audit to do optimization
 .. code-block:: shell

    $ openstack optimize audittemplate create \
-      at1 saving_energy --strategy saving_energy
+      saving_energy_template1 saving_energy --strategy saving_energy

-    $ openstack optimize audit create -a at1 \
+    $ openstack optimize audit create -a saving_energy_audit1 \
      -p free_used_percent=20.0

 External Links
--- a/doc/source/strategies/vm_workload_consolidation.rst
+++ b/doc/source/strategies/vm_workload_consolidation.rst
@@ -22,14 +22,19 @@ The *vm_workload_consolidation* strategy requires the following metrics:
 ============================ ============ ======= =========================
 metric                       service name plugins comment
 ============================ ============ ======= =========================
-``cpu_util``                 ceilometer_  none    cpu_util has been removed
-                                                  since Stein.
+``cpu``                      ceilometer_  none
 ``memory.resident``          ceilometer_  none
 ``memory``                   ceilometer_  none
 ``disk.root.size``           ceilometer_  none
+``compute.node.cpu.percent`` ceilometer_  none    (optional) need to set the
+                                                  ``compute_monitors`` option
+                                                  to ``cpu.virt_driver`` in the
+                                                  nova.conf.
+``hardware.memory.used``     ceilometer_  SNMP_   (optional)
 ============================ ============ ======= =========================

 .. _ceilometer: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#openstack-compute
+.. _SNMP: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#snmp-based-meters

 Cluster data model
 ******************
--- a/doc/source/strategies/workload-stabilization.rst
+++ b/doc/source/strategies/workload-stabilization.rst
@@ -27,9 +27,8 @@ metric                       service name plugins comment
                                                  to ``cpu.virt_driver`` in the
                                                  nova.conf.
 ``hardware.memory.used``     ceilometer_  SNMP_
-``cpu_util``                 ceilometer_  none    cpu_util has been removed
-                                                  since Stein.
-``memory.resident``          ceilometer_  none
+``cpu``                      ceilometer_  none
+``instance_ram_usage``       ceilometer_  none
 ============================ ============ ======= =============================

 .. _ceilometer: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#openstack-compute
@@ -107,10 +106,10 @@ parameter            type   default Value         description
                                                  period of all received ones.
 ==================== ====== ===================== =============================

-.. |metrics| replace:: ["cpu_util", "memory.resident"]
-.. |thresholds| replace:: {"cpu_util": 0.2, "memory.resident": 0.2}
-.. |weights| replace:: {"cpu_util_weight": 1.0, "memory.resident_weight": 1.0}
-.. |instance_metrics| replace:: {"cpu_util": "compute.node.cpu.percent", "memory.resident": "hardware.memory.used"}
+.. |metrics| replace:: ["instance_cpu_usage", "instance_ram_usage"]
+.. |thresholds| replace:: {"instance_cpu_usage": 0.2, "instance_ram_usage": 0.2}
+.. |weights| replace:: {"instance_cpu_usage_weight": 1.0, "instance_ram_usage_weight": 1.0}
+.. |instance_metrics| replace:: {"instance_cpu_usage": "compute.node.cpu.percent", "instance_ram_usage": "hardware.memory.used"}
 .. |periods| replace:: {"instance": 720, "node": 600}

 Efficacy Indicator
@@ -136,8 +135,8 @@ How to use it ?
      at1 workload_balancing --strategy workload_stabilization

    $ openstack optimize audit create -a at1 \
-      -p thresholds='{"memory.resident": 0.05}' \
-      -p metrics='["memory.resident"]'
+      -p thresholds='{"instance_ram_usage": 0.05}' \
+      -p metrics='["instance_ram_usage"]'

 External Links
 --------------
--- a/doc/source/strategies/workload_balance.rst
+++ b/doc/source/strategies/workload_balance.rst
@@ -24,8 +24,7 @@ The *workload_balance* strategy requires the following metrics:
 ======================= ============ ======= =========================
 metric                  service name plugins comment
 ======================= ============ ======= =========================
-``cpu_util``            ceilometer_  none    cpu_util has been removed
-                                             since Stein.
+``cpu``                 ceilometer_  none
 ``memory.resident``     ceilometer_  none
 ======================= ============ ======= =========================

@@ -65,15 +64,16 @@ Configuration

 Strategy parameters are:

-============== ====== ============= ====================================
-parameter      type   default Value description
-============== ====== ============= ====================================
-``metrics``    String 'cpu_util'    Workload balance base on cpu or ram
-                                    utilization. choice: ['cpu_util',
-                                    'memory.resident']
-``threshold``  Number 25.0          Workload threshold for migration
-``period``     Number 300           Aggregate time period of ceilometer
-============== ====== ============= ====================================
+============== ====== ==================== ====================================
+parameter      type   default Value        description
+============== ====== ==================== ====================================
+``metrics``    String 'instance_cpu_usage' Workload balance base on cpu or ram
+                                           utilization. Choices:
+                                           ['instance_cpu_usage',
+                                           'instance_ram_usage']
+``threshold``  Number 25.0                 Workload threshold for migration
+``period``     Number 300                  Aggregate time period of ceilometer
+============== ====== ==================== ====================================

 Efficacy Indicator
 ------------------
@@ -95,7 +95,7 @@ How to use it ?
      at1 workload_balancing --strategy workload_balance

    $ openstack optimize audit create -a at1 -p threshold=26.0 \
-            -p period=310 -p metrics=cpu_util
+            -p period=310 -p metrics=instance_cpu_usage

 External Links
 --------------
--- a/releasenotes/source/2023.1.rst
+++ b/releasenotes/source/2023.1.rst
@@ -0,0 +1,6 @@
+===========================
+2023.1 Series Release Notes
+===========================
+
+.. release-notes::
+   :branch: stable/2023.1
--- a/releasenotes/source/2023.2.rst
+++ b/releasenotes/source/2023.2.rst
@@ -0,0 +1,6 @@
+===========================
+2023.2 Series Release Notes
+===========================
+
+.. release-notes::
+   :branch: stable/2023.2
--- a/releasenotes/source/conf.py
+++ b/releasenotes/source/conf.py
@@ -53,7 +53,7 @@ source_suffix = '.rst'
 master_doc = 'index'

 # General information about the project.
-copyright = u'2016, Watcher developers'
+copyright = '2016, Watcher developers'

 # Release notes are version independent
 # The short X.Y version.
@@ -196,8 +196,8 @@ latex_elements = {
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, documentclass [howto/manual])
 latex_documents = [
-  ('index', 'watcher.tex', u'Watcher Documentation',
-   u'Watcher developers', 'manual'),
+  ('index', 'watcher.tex', 'Watcher Documentation',
+   'Watcher developers', 'manual'),
 ]

 # The name of an image file (relative to this directory) to place at the top of
@@ -226,8 +226,8 @@ latex_documents = [
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    ('index', 'watcher', u'Watcher Documentation',
-     [u'Watcher developers'], 1)
+    ('index', 'watcher', 'Watcher Documentation',
+     ['Watcher developers'], 1)
 ]

 # If true, show URL addresses after external links.
@@ -240,8 +240,8 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  ('index', 'watcher', u'Watcher Documentation',
-   u'Watcher developers', 'watcher', 'One line description of project.',
+  ('index', 'watcher', 'Watcher Documentation',
+   'Watcher developers', 'watcher', 'One line description of project.',
   'Miscellaneous'),
 ]

--- a/releasenotes/source/index.rst
+++ b/releasenotes/source/index.rst
@@ -21,6 +21,11 @@ Contents:
   :maxdepth: 1

   unreleased
+   2023.2
+   2023.1
+   zed
+   yoga
+   xena
   wallaby
   victoria
   ussuri
--- a/releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po
+++ b/releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po
@@ -1,15 +1,18 @@
 # Andi Chandler <andi@gowling.com>, 2017. #zanata
 # Andi Chandler <andi@gowling.com>, 2018. #zanata
 # Andi Chandler <andi@gowling.com>, 2020. #zanata
+# Andi Chandler <andi@gowling.com>, 2022. #zanata
+# Andi Chandler <andi@gowling.com>, 2023. #zanata
+# Andi Chandler <andi@gowling.com>, 2024. #zanata
 msgid ""
 msgstr ""
 "Project-Id-Version: python-watcher\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2020-10-27 04:13+0000\n"
+"POT-Creation-Date: 2024-05-31 14:40+0000\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"PO-Revision-Date: 2020-10-28 11:13+0000\n"
+"PO-Revision-Date: 2024-04-18 12:21+0000\n"
 "Last-Translator: Andi Chandler <andi@gowling.com>\n"
 "Language-Team: English (United Kingdom)\n"
 "Language: en_GB\n"
@@ -58,12 +61,21 @@ msgstr "1.9.0"
 msgid "2.0.0"
 msgstr "2.0.0"

+msgid "2023.1 Series Release Notes"
+msgstr "2023.1 Series Release Notes"
+
+msgid "2023.2 Series Release Notes"
+msgstr "2023.2 Series Release Notes"
+
 msgid "3.0.0"
 msgstr "3.0.0"

 msgid "4.0.0"
 msgstr "4.0.0"

+msgid "6.0.0"
+msgstr "6.0.0"
+
 msgid "A ``watcher-status upgrade check`` has been added for this."
 msgstr "A ``watcher-status upgrade check`` has been added for this."

@@ -744,6 +756,23 @@ msgstr ""
 "The configuration options for query retries in `[gnocchi_client]` are "
 "deprecated and the option in `[watcher_datasources]` should now be used."

+msgid ""
+"The default value of ``[oslo_policy] policy_file`` config option has been "
+"changed from ``policy.json`` to ``policy.yaml``. Operators who are utilizing "
+"customized or previously generated static policy JSON files (which are not "
+"needed by default), should generate new policy files or convert them in YAML "
+"format. Use the `oslopolicy-convert-json-to-yaml <https://docs.openstack.org/"
+"oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_ tool to "
+"convert a JSON to YAML formatted policy file in backward compatible way."
+msgstr ""
+"The default value of ``[oslo_policy] policy_file`` config option has been "
+"changed from ``policy.json`` to ``policy.yaml``. Operators who are utilizing "
+"customized or previously generated static policy JSON files (which are not "
+"needed by default), should generate new policy files or convert them in YAML "
+"format. Use the `oslopolicy-convert-json-to-yaml <https://docs.openstack.org/"
+"oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_ tool to "
+"convert a JSON to YAML formatted policy file in backward compatible way."
+
 msgid ""
 "The graph model describes how VMs are associated to compute hosts. This "
 "allows for seeing relationships upfront between the entities and hence can "
@@ -799,6 +828,21 @@ msgstr "Train Series Release Notes"
 msgid "Upgrade Notes"
 msgstr "Upgrade Notes"

+msgid ""
+"Use of JSON policy files was deprecated by the ``oslo.policy`` library "
+"during the Victoria development cycle. As a result, this deprecation is "
+"being noted in the Wallaby cycle with an anticipated future removal of "
+"support by ``oslo.policy``. As such operators will need to convert to YAML "
+"policy files. Please see the upgrade notes for details on migration of any "
+"custom policy files."
+msgstr ""
+"Use of JSON policy files was deprecated by the ``oslo.policy`` library "
+"during the Victoria development cycle. As a result, this deprecation is "
+"being noted in the Wallaby cycle with an anticipated future removal of "
+"support by ``oslo.policy``. As such operators will need to convert to YAML "
+"policy files. Please see the upgrade notes for details on migration of any "
+"custom policy files."
+
 msgid ""
 "Using ``watcher/api/app.wsgi`` script is deprecated and it will be removed "
 "in U release. Please switch to automatically generated ``watcher-api-wsgi`` "
@@ -814,6 +858,9 @@ msgstr "Ussuri Series Release Notes"
 msgid "Victoria Series Release Notes"
 msgstr "Victoria Series Release Notes"

+msgid "Wallaby Series Release Notes"
+msgstr "Wallaby Series Release Notes"
+
 msgid ""
 "Watcher can continuously optimize the OpenStack cloud for a specific "
 "strategy or goal by triggering an audit periodically which generates an "
@@ -924,6 +971,15 @@ msgstr "We also add some new properties:"
 msgid "Welcome to watcher's Release Notes documentation!"
 msgstr "Welcome to watcher's Release Notes documentation!"

+msgid "Xena Series Release Notes"
+msgstr "Xena Series Release Notes"
+
+msgid "Yoga Series Release Notes"
+msgstr "Yoga Series Release Notes"
+
+msgid "Zed Series Release Notes"
+msgstr "Zed Series Release Notes"
+
 msgid "``[watcher_datasources] datasources = gnocchi,monasca,ceilometer``"
 msgstr "``[watcher_datasources] datasources = gnocchi,monasca,ceilometer``"

--- a/releasenotes/source/xena.rst
+++ b/releasenotes/source/xena.rst
@@ -0,0 +1,6 @@
+=========================
+Xena Series Release Notes
+=========================
+
+.. release-notes::
+   :branch: stable/xena
--- a/releasenotes/source/yoga.rst
+++ b/releasenotes/source/yoga.rst
@@ -0,0 +1,6 @@
+=========================
+Yoga Series Release Notes
+=========================
+
+.. release-notes::
+   :branch: stable/yoga
--- a/releasenotes/source/zed.rst
+++ b/releasenotes/source/zed.rst
@@ -0,0 +1,6 @@
+========================
+Zed Series Release Notes
+========================
+
+.. release-notes::
+   :branch: stable/zed
--- a/requirements.txt
+++ b/requirements.txt
@@ -17,7 +17,7 @@ oslo.context>=2.21.0 # Apache-2.0
 oslo.db>=4.44.0 # Apache-2.0
 oslo.i18n>=3.20.0 # Apache-2.0
 oslo.log>=3.37.0 # Apache-2.0
-oslo.messaging>=8.1.2 # Apache-2.0
+oslo.messaging>=14.1.0 # Apache-2.0
 oslo.policy>=3.6.0 # Apache-2.0
 oslo.reports>=1.27.0 # Apache-2.0
 oslo.serialization>=2.25.0 # Apache-2.0
@@ -30,7 +30,6 @@ pbr>=3.1.1 # Apache-2.0
 pecan>=1.3.2 # BSD
 PrettyTable>=0.7.2 # BSD
 gnocchiclient>=7.0.1 # Apache-2.0
-python-ceilometerclient>=2.9.0 # Apache-2.0
 python-cinderclient>=3.5.0 # Apache-2.0
 python-glanceclient>=2.9.1 # Apache-2.0
 python-keystoneclient>=3.15.0 # Apache-2.0
--- a/setup.cfg
+++ b/setup.cfg
@@ -6,7 +6,7 @@ description_file =
 author = OpenStack
 author_email = openstack-discuss@lists.openstack.org
 home_page = https://docs.openstack.org/watcher/latest/
-python_requires = >=3.6
+python_requires = >=3.8
 classifier =
    Environment :: OpenStack
    Intended Audience :: Information Technology
@@ -17,9 +17,10 @@ classifier =
    Programming Language :: Python :: Implementation :: CPython
    Programming Language :: Python :: 3 :: Only
    Programming Language :: Python :: 3
-    Programming Language :: Python :: 3.6
-    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8
+    Programming Language :: Python :: 3.9
+    Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3.11

 [files]
 packages =
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -12,3 +12,4 @@ testtools>=2.3.0 # MIT
 stestr>=2.0.0 # Apache-2.0
 os-api-ref>=1.4.0 # Apache-2.0
 bandit>=1.6.0 # Apache-2.0
+WebTest>=2.0.27 # MIT
--- a/tox.ini
+++ b/tox.ini
@@ -1,7 +1,6 @@
 [tox]
 minversion = 3.18.0
 envlist = py3,pep8
-skipsdist = True
 ignore_basepython_conflict = True

 [testenv]
@@ -9,23 +8,30 @@ basepython = python3
 usedevelop = True
 allowlist_externals = find
                      rm
-install_command = pip install -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/xena} {opts} {packages}
+install_command = pip install -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} {opts} {packages}
 setenv =
   VIRTUAL_ENV={envdir}
 deps =
    -r{toxinidir}/test-requirements.txt
    -r{toxinidir}/requirements.txt
+    python-libmaas>=0.6.8
 commands =
  rm -f .testrepository/times.dbm
  find . -type f -name "*.py[c|o]" -delete
  stestr run {posargs}
-passenv = http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY
+passenv =
+  http_proxy
+  HTTP_PROXY
+  https_proxy
+  HTTPS_PROXY
+  no_proxy
+  NO_PROXY

 [testenv:pep8]
 commands =
    doc8 doc/source/ CONTRIBUTING.rst HACKING.rst README.rst
    flake8
-    bandit -r watcher -x watcher/tests/* -n5 -ll -s B320
+    #bandit -r watcher -x watcher/tests/* -n5 -ll -s B320

 [testenv:venv]
 setenv = PYTHONHASHSEED=0
--- a/watcher/applier/actions/change_node_power_state.py
+++ b/watcher/applier/actions/change_node_power_state.py
@@ -17,17 +17,17 @@
 # limitations under the License.
 #

-import enum
 import time

+from oslo_log import log
+
 from watcher._i18n import _
 from watcher.applier.actions import base
 from watcher.common import exception
+from watcher.common.metal_helper import constants as metal_constants
+from watcher.common.metal_helper import factory as metal_helper_factory

-
-class NodeState(enum.Enum):
-    POWERON = 'on'
-    POWEROFF = 'off'
+LOG = log.getLogger(__name__)


 class ChangeNodePowerState(base.BaseAction):
@@ -43,8 +43,8 @@ class ChangeNodePowerState(base.BaseAction):
         'state': str,
        })

-    The `resource_id` references a ironic node id (list of available
-    ironic node is returned by this command: ``ironic node-list``).
+    The `resource_id` references a baremetal node id (list of available
+    ironic nodes is returned by this command: ``ironic node-list``).
    The `state` value should either be `on` or `off`.
    """

@@ -59,10 +59,14 @@ class ChangeNodePowerState(base.BaseAction):
                    'type': 'string',
                    "minlength": 1
                },
+                'resource_name': {
+                    'type': 'string',
+                    "minlength": 1
+                },
                'state': {
                    'type': 'string',
-                    'enum': [NodeState.POWERON.value,
-                             NodeState.POWEROFF.value]
+                    'enum': [metal_constants.PowerState.ON.value,
+                             metal_constants.PowerState.OFF.value]
                }
            },
            'required': ['resource_id', 'state'],
@@ -82,10 +86,10 @@ class ChangeNodePowerState(base.BaseAction):
        return self._node_manage_power(target_state)

    def revert(self):
-        if self.state == NodeState.POWERON.value:
-            target_state = NodeState.POWEROFF.value
-        elif self.state == NodeState.POWEROFF.value:
-            target_state = NodeState.POWERON.value
+        if self.state == metal_constants.PowerState.ON.value:
+            target_state = metal_constants.PowerState.OFF.value
+        elif self.state == metal_constants.PowerState.OFF.value:
+            target_state = metal_constants.PowerState.ON.value
        return self._node_manage_power(target_state)

    def _node_manage_power(self, state, retry=60):
@@ -93,30 +97,32 @@ class ChangeNodePowerState(base.BaseAction):
            raise exception.IllegalArgumentException(
                message=_("The target state is not defined"))

-        ironic_client = self.osc.ironic()
-        nova_client = self.osc.nova()
-        current_state = ironic_client.node.get(self.node_uuid).power_state
-        # power state: 'power on' or 'power off', if current node state
-        # is the same as state, just return True
-        if state in current_state:
+        metal_helper = metal_helper_factory.get_helper(self.osc)
+        node = metal_helper.get_node(self.node_uuid)
+        current_state = node.get_power_state()
+
+        if state == current_state.value:
            return True

-        if state == NodeState.POWEROFF.value:
-            node_info = ironic_client.node.get(self.node_uuid).to_dict()
-            compute_node_id = node_info['extra']['compute_node_id']
-            compute_node = nova_client.hypervisors.get(compute_node_id)
-            compute_node = compute_node.to_dict()
+        if state == metal_constants.PowerState.OFF.value:
+            compute_node = node.get_hypervisor_node().to_dict()
            if (compute_node['running_vms'] == 0):
-                ironic_client.node.set_power_state(
-                    self.node_uuid, state)
+                node.set_power_state(state)
+            else:
+                LOG.warning(
+                    "Compute node %s has %s running vms and will "
+                    "NOT be shut off.",
+                    compute_node["hypervisor_hostname"],
+                    compute_node['running_vms'])
+                return False
        else:
-            ironic_client.node.set_power_state(self.node_uuid, state)
+            node.set_power_state(state)

-        ironic_node = ironic_client.node.get(self.node_uuid)
-        while ironic_node.power_state == current_state and retry:
+        node = metal_helper.get_node(self.node_uuid)
+        while node.get_power_state() == current_state and retry:
            time.sleep(10)
            retry -= 1
-            ironic_node = ironic_client.node.get(self.node_uuid)
+            node = metal_helper.get_node(self.node_uuid)
        if retry > 0:
            return True
        else:
@@ -130,4 +136,4 @@ class ChangeNodePowerState(base.BaseAction):

    def get_description(self):
        """Description of the action"""
-        return ("Compute node power on/off through ironic.")
+        return ("Compute node power on/off through Ironic or MaaS.")
--- a/watcher/common/clients.py
+++ b/watcher/common/clients.py
@@ -25,6 +25,7 @@ from novaclient import api_versions as nova_api_versions
 from novaclient import client as nvclient

 from watcher.common import exception
+from watcher.common import utils

 try:
    from ceilometerclient import client as ceclient
@@ -32,6 +33,12 @@ try:
 except ImportError:
    HAS_CEILCLIENT = False

+try:
+    from maas import client as maas_client
+except ImportError:
+    maas_client = None
+
+
 CONF = cfg.CONF

 _CLIENTS_AUTH_GROUP = 'watcher_clients_auth'
@@ -74,6 +81,7 @@ class OpenStackClients(object):
        self._monasca = None
        self._neutron = None
        self._ironic = None
+        self._maas = None
        self._placement = None

    def _get_keystone_session(self):
@@ -265,6 +273,23 @@ class OpenStackClients(object):
                                           session=self.session)
        return self._ironic

+    def maas(self):
+        if self._maas:
+            return self._maas
+
+        if not maas_client:
+            raise exception.UnsupportedError(
+                "MAAS client unavailable. Please install python-libmaas.")
+
+        url = self._get_client_option('maas', 'url')
+        api_key = self._get_client_option('maas', 'api_key')
+        timeout = self._get_client_option('maas', 'timeout')
+        self._maas = utils.async_compat_call(
+            maas_client.connect,
+            url, apikey=api_key,
+            timeout=timeout)
+        return self._maas
+
    @exception.wrap_keystone_exception
    def placement(self):
        if self._placement:
--- a/watcher/common/metal_helper/init.py
+++ b/watcher/common/metal_helper/init.py
--- a/watcher/common/metal_helper/base.py
+++ b/watcher/common/metal_helper/base.py
@@ -0,0 +1,81 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import abc
+
+from watcher.common import exception
+from watcher.common.metal_helper import constants as metal_constants
+
+
+class BaseMetalNode(abc.ABC):
+    hv_up_when_powered_off = False
+
+    def __init__(self, nova_node=None):
+        self._nova_node = nova_node
+
+    def get_hypervisor_node(self):
+        if not self._nova_node:
+            raise exception.Invalid(message="No associated hypervisor.")
+        return self._nova_node
+
+    def get_hypervisor_hostname(self):
+        return self.get_hypervisor_node().hypervisor_hostname
+
+    @abc.abstractmethod
+    def get_power_state(self):
+        # TODO(lpetrut): document the following methods
+        pass
+
+    @abc.abstractmethod
+    def get_id(self):
+        """Return the node id provided by the bare metal service."""
+        pass
+
+    @abc.abstractmethod
+    def power_on(self):
+        pass
+
+    @abc.abstractmethod
+    def power_off(self):
+        pass
+
+    def set_power_state(self, state):
+        state = metal_constants.PowerState(state)
+        if state == metal_constants.PowerState.ON:
+            self.power_on()
+        elif state == metal_constants.PowerState.OFF:
+            self.power_off()
+        else:
+            raise exception.UnsupportedActionType(
+                "Cannot set power state: %s" % state)
+
+
+class BaseMetalHelper(abc.ABC):
+    def __init__(self, osc):
+        self._osc = osc
+
+    @property
+    def nova_client(self):
+        if not getattr(self, "_nova_client", None):
+            self._nova_client = self._osc.nova()
+        return self._nova_client
+
+    @abc.abstractmethod
+    def list_compute_nodes(self):
+        pass
+
+    @abc.abstractmethod
+    def get_node(self, node_id):
+        pass
--- a/watcher/common/metal_helper/constants.py
+++ b/watcher/common/metal_helper/constants.py
@@ -0,0 +1,23 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import enum
+
+
+class PowerState(str, enum.Enum):
+    ON = "on"
+    OFF = "off"
+    UNKNOWN = "unknown"
+    ERROR = "error"
--- a/watcher/common/metal_helper/factory.py
+++ b/watcher/common/metal_helper/factory.py
@@ -0,0 +1,33 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from oslo_config import cfg
+
+from watcher.common import clients
+from watcher.common.metal_helper import ironic
+from watcher.common.metal_helper import maas
+
+CONF = cfg.CONF
+
+
+def get_helper(osc=None):
+    # TODO(lpetrut): consider caching this client.
+    if not osc:
+        osc = clients.OpenStackClients()
+
+    if CONF.maas_client.url:
+        return maas.MaasHelper(osc)
+    else:
+        return ironic.IronicHelper(osc)
--- a/watcher/common/metal_helper/ironic.py
+++ b/watcher/common/metal_helper/ironic.py
@@ -0,0 +1,94 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from oslo_log import log
+
+from watcher.common.metal_helper import base
+from watcher.common.metal_helper import constants as metal_constants
+
+LOG = log.getLogger(__name__)
+
+POWER_STATES_MAP = {
+    'power on': metal_constants.PowerState.ON,
+    'power off': metal_constants.PowerState.OFF,
+    # For now, we only use ON/OFF states
+    'rebooting': metal_constants.PowerState.ON,
+    'soft power off': metal_constants.PowerState.OFF,
+    'soft reboot': metal_constants.PowerState.ON,
+}
+
+
+class IronicNode(base.BaseMetalNode):
+    hv_up_when_powered_off = True
+
+    def __init__(self, ironic_node, nova_node, ironic_client):
+        super().__init__(nova_node)
+
+        self._ironic_client = ironic_client
+        self._ironic_node = ironic_node
+
+    def get_power_state(self):
+        return POWER_STATES_MAP.get(self._ironic_node.power_state,
+                                    metal_constants.PowerState.UNKNOWN)
+
+    def get_id(self):
+        return self._ironic_node.uuid
+
+    def power_on(self):
+        self._ironic_client.node.set_power_state(self.get_id(), "on")
+
+    def power_off(self):
+        self._ironic_client.node.set_power_state(self.get_id(), "off")
+
+
+class IronicHelper(base.BaseMetalHelper):
+    @property
+    def _client(self):
+        if not getattr(self, "_cached_client", None):
+            self._cached_client = self._osc.ironic()
+        return self._cached_client
+
+    def list_compute_nodes(self):
+        out_list = []
+        # TODO(lpetrut): consider using "detailed=True" instead of making
+        # an additional GET request per node
+        node_list = self._client.node.list()
+
+        for node in node_list:
+            node_info = self._client.node.get(node.uuid)
+            hypervisor_id = node_info.extra.get('compute_node_id', None)
+            if hypervisor_id is None:
+                LOG.warning('Cannot find compute_node_id in extra '
+                            'of ironic node %s', node.uuid)
+                continue
+
+            hypervisor_node = self.nova_client.hypervisors.get(hypervisor_id)
+            if hypervisor_node is None:
+                LOG.warning('Cannot find hypervisor %s', hypervisor_id)
+                continue
+
+            out_node = IronicNode(node, hypervisor_node, self._client)
+            out_list.append(out_node)
+
+        return out_list
+
+    def get_node(self, node_id):
+        ironic_node = self._client.node.get(node_id)
+        compute_node_id = ironic_node.extra.get('compute_node_id')
+        if compute_node_id:
+            compute_node = self.nova_client.hypervisors.get(compute_node_id)
+        else:
+            compute_node = None
+        return IronicNode(ironic_node, compute_node, self._client)
--- a/watcher/common/metal_helper/maas.py
+++ b/watcher/common/metal_helper/maas.py
@@ -0,0 +1,125 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from oslo_config import cfg
+from oslo_log import log
+
+from watcher.common import exception
+from watcher.common.metal_helper import base
+from watcher.common.metal_helper import constants as metal_constants
+from watcher.common import utils
+
+CONF = cfg.CONF
+LOG = log.getLogger(__name__)
+
+try:
+    from maas.client import enum as maas_enum
+except ImportError:
+    maas_enum = None
+
+
+class MaasNode(base.BaseMetalNode):
+    hv_up_when_powered_off = False
+
+    def __init__(self, maas_node, nova_node, maas_client):
+        super().__init__(nova_node)
+
+        self._maas_client = maas_client
+        self._maas_node = maas_node
+
+    def get_power_state(self):
+        maas_state = utils.async_compat_call(
+            self._maas_node.query_power_state,
+            timeout=CONF.maas_client.timeout)
+
+        # python-libmaas may not be available, so we'll avoid a global
+        # variable.
+        power_states_map = {
+            maas_enum.PowerState.ON: metal_constants.PowerState.ON,
+            maas_enum.PowerState.OFF: metal_constants.PowerState.OFF,
+            maas_enum.PowerState.ERROR: metal_constants.PowerState.ERROR,
+            maas_enum.PowerState.UNKNOWN: metal_constants.PowerState.UNKNOWN,
+        }
+        return power_states_map.get(maas_state,
+                                    metal_constants.PowerState.UNKNOWN)
+
+    def get_id(self):
+        return self._maas_node.system_id
+
+    def power_on(self):
+        LOG.info("Powering on MAAS node: %s %s",
+                 self._maas_node.fqdn,
+                 self._maas_node.system_id)
+        utils.async_compat_call(
+            self._maas_node.power_on,
+            timeout=CONF.maas_client.timeout)
+
+    def power_off(self):
+        LOG.info("Powering off MAAS node: %s %s",
+                 self._maas_node.fqdn,
+                 self._maas_node.system_id)
+        utils.async_compat_call(
+            self._maas_node.power_off,
+            timeout=CONF.maas_client.timeout)
+
+
+class MaasHelper(base.BaseMetalHelper):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        if not maas_enum:
+            raise exception.UnsupportedError(
+                "MAAS client unavailable. Please install python-libmaas.")
+
+    @property
+    def _client(self):
+        if not getattr(self, "_cached_client", None):
+            self._cached_client = self._osc.maas()
+        return self._cached_client
+
+    def list_compute_nodes(self):
+        out_list = []
+        node_list = utils.async_compat_call(
+            self._client.machines.list,
+            timeout=CONF.maas_client.timeout)
+
+        compute_nodes = self.nova_client.hypervisors.list()
+        compute_node_map = dict()
+        for compute_node in compute_nodes:
+            compute_node_map[compute_node.hypervisor_hostname] = compute_node
+
+        for node in node_list:
+            hypervisor_node = compute_node_map.get(node.fqdn)
+            if not hypervisor_node:
+                LOG.info('Cannot find hypervisor %s', node.fqdn)
+                continue
+
+            out_node = MaasNode(node, hypervisor_node, self._client)
+            out_list.append(out_node)
+
+        return out_list
+
+    def _get_compute_node_by_hostname(self, hostname):
+        compute_nodes = self.nova_client.hypervisors.search(
+            hostname, detailed=True)
+        for compute_node in compute_nodes:
+            if compute_node.hypervisor_hostname == hostname:
+                return compute_node
+
+    def get_node(self, node_id):
+        maas_node = utils.async_compat_call(
+            self._client.machines.get, node_id,
+            timeout=CONF.maas_client.timeout)
+        compute_node = self._get_compute_node_by_hostname(maas_node.fqdn)
+        return MaasNode(maas_node, compute_node, self._client)
--- a/watcher/common/rpc.py
+++ b/watcher/common/rpc.py
@@ -121,7 +121,7 @@ class RequestContextSerializer(messaging.Serializer):
 def get_client(target, version_cap=None, serializer=None):
    assert TRANSPORT is not None
    serializer = RequestContextSerializer(serializer)
-    return messaging.RPCClient(
+    return messaging.get_rpc_client(
        TRANSPORT,
        target,
        version_cap=version_cap,
--- a/watcher/common/utils.py
+++ b/watcher/common/utils.py
@@ -16,12 +16,16 @@

 """Utilities and helper functions."""

+import asyncio
 import datetime
+import inspect
 import random
 import re
 import string

 from croniter import croniter
+import eventlet
+from eventlet import tpool

 from jsonschema import validators
 from oslo_config import cfg
@@ -162,3 +166,37 @@ Draft4Validator = validators.Draft4Validator
 def random_string(n):
    return ''.join([random.choice(
        string.ascii_letters + string.digits) for i in range(n)])
+
+
+# Some clients (e.g. MAAS) use asyncio, which isn't compatible with Eventlet.
+# As a workaround, we're delegating such calls to a native thread.
+def async_compat_call(f, *args, **kwargs):
+    timeout = kwargs.pop('timeout', None)
+
+    async def async_wrapper():
+        ret = f(*args, **kwargs)
+        if inspect.isawaitable(ret):
+            return await asyncio.wait_for(ret, timeout)
+        return ret
+
+    def tpool_wrapper():
+        # This will run in a separate native thread. Ideally, there should be
+        # a single thread permanently running an asyncio loop, but for
+        # convenience we'll use eventlet.tpool, which leverages a thread pool.
+        #
+        # That being considered, we're setting up a temporary asyncio loop to
+        # handle this call.
+        loop = asyncio.new_event_loop()
+        try:
+            asyncio.set_event_loop(loop)
+            return loop.run_until_complete(async_wrapper())
+        finally:
+            loop.close()
+
+    # We'll use eventlet timeouts as an extra precaution and asyncio timeouts
+    # to avoid lingering threads. For consistency, we'll convert eventlet
+    # timeout exceptions to asyncio timeout errors.
+    with eventlet.timeout.Timeout(
+            seconds=timeout,
+            exception=asyncio.TimeoutError("Timeout: %ss" % timeout)):
+        return tpool.execute(tpool_wrapper)
--- a/watcher/conf/init.py
+++ b/watcher/conf/init.py
@@ -35,6 +35,7 @@ from watcher.conf import grafana_client
 from watcher.conf import grafana_translators
 from watcher.conf import ironic_client
 from watcher.conf import keystone_client
+from watcher.conf import maas_client
 from watcher.conf import monasca_client
 from watcher.conf import neutron_client
 from watcher.conf import nova_client
@@ -54,6 +55,7 @@ db.register_opts(CONF)
 planner.register_opts(CONF)
 applier.register_opts(CONF)
 decision_engine.register_opts(CONF)
+maas_client.register_opts(CONF)
 monasca_client.register_opts(CONF)
 nova_client.register_opts(CONF)
 glance_client.register_opts(CONF)
--- a/watcher/conf/grafana_client.py
+++ b/watcher/conf/grafana_client.py
@@ -134,7 +134,13 @@ GRAFANA_CLIENT_OPTS = [
                     "InfluxDB this will be the retention period. "
                     "These queries will need to be constructed using tools "
                     "such as Postman. Example: SELECT cpu FROM {4}."
-                     "cpu_percent WHERE host == '{1}' AND time > now()-{2}s")]
+                     "cpu_percent WHERE host == '{1}' AND time > now()-{2}s"),
+    cfg.IntOpt('http_timeout',
+               min=0,
+               default=60,
+               mutable=True,
+               help='Timeout for Grafana request')
+]


 def register_opts(conf):
--- a/watcher/conf/maas_client.py
+++ b/watcher/conf/maas_client.py
@@ -0,0 +1,38 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+from oslo_config import cfg
+
+maas_client = cfg.OptGroup(name='maas_client',
+                           title='Configuration Options for MaaS')
+
+MAAS_CLIENT_OPTS = [
+    cfg.StrOpt('url',
+               help='MaaS URL, example: http://1.2.3.4:5240/MAAS'),
+    cfg.StrOpt('api_key',
+               help='MaaS API authentication key.'),
+    cfg.IntOpt('timeout',
+               default=60,
+               help='MaaS client operation timeout in seconds.')]
+
+
+def register_opts(conf):
+    conf.register_group(maas_client)
+    conf.register_opts(MAAS_CLIENT_OPTS, group=maas_client)
+
+
+def list_opts():
+    return [(maas_client, MAAS_CLIENT_OPTS)]
--- a/watcher/db/sqlalchemy/alembic/versions/0f6042416884_add_apscheduler_jobs.py
+++ b/watcher/db/sqlalchemy/alembic/versions/0f6042416884_add_apscheduler_jobs.py
@@ -6,6 +6,7 @@ Create Date: 2017-03-24 11:21:29.036532

 """
 from alembic import op
+from sqlalchemy import inspect
 import sqlalchemy as sa

 from watcher.db.sqlalchemy import models
@@ -14,11 +15,20 @@ from watcher.db.sqlalchemy import models
 revision = '0f6042416884'
 down_revision = '001'

+def _table_exists(table_name):
+    bind = op.get_context().bind
+    insp = inspect(bind)
+    names = insp.get_table_names()
+    return any(t == table_name for t in names)
+

 def upgrade():
+    if _table_exists('apscheduler_jobs'):
+        return
+
    op.create_table(
        'apscheduler_jobs',
-        sa.Column('id', sa.Unicode(191, _warn_on_bytestring=False),
+        sa.Column('id', sa.Unicode(191),
                  nullable=False),
        sa.Column('next_run_time', sa.Float(25), index=True),
        sa.Column('job_state', sa.LargeBinary, nullable=False),
--- a/watcher/db/sqlalchemy/api.py
+++ b/watcher/db/sqlalchemy/api.py
@@ -244,30 +244,36 @@ class Connection(api.BaseConnection):
        for relationship in relationships:
            if not relationship.uselist:
                # We have a One-to-X relationship
-                query = query.options(joinedload(relationship.key))
+                query = query.options(joinedload(
+                    getattr(model, relationship.key)))
        return query

    def _create(self, model, values):
-        obj = model()
-        cleaned_values = {k: v for k, v in values.items()
-                          if k not in self._get_relationships(model)}
-        obj.update(cleaned_values)
-        obj.save()
+        session = get_session()
+        with session.begin():
+            obj = model()
+            cleaned_values = {k: v for k, v in values.items()
+                              if k not in self._get_relationships(model)}
+            obj.update(cleaned_values)
+            obj.save(session=session)
+            session.commit()
        return obj

    def _get(self, context, model, fieldname, value, eager):
-        query = model_query(model)
-        if eager:
-            query = self._set_eager_options(model, query)
+        session = get_session()
+        with session.begin():
+            query = model_query(model, session=session)
+            if eager:
+                query = self._set_eager_options(model, query)

-        query = query.filter(getattr(model, fieldname) == value)
-        if not context.show_deleted:
-            query = query.filter(model.deleted_at.is_(None))
+            query = query.filter(getattr(model, fieldname) == value)
+            if not context.show_deleted:
+                query = query.filter(model.deleted_at.is_(None))

-        try:
-            obj = query.one()
-        except exc.NoResultFound:
-            raise exception.ResourceNotFound(name=model.__name__, id=value)
+            try:
+                obj = query.one()
+            except exc.NoResultFound:
+                raise exception.ResourceNotFound(name=model.__name__, id=value)

        return obj

--- a/watcher/db/sqlalchemy/job_store.py
+++ b/watcher/db/sqlalchemy/job_store.py
@@ -22,6 +22,7 @@ from apscheduler.jobstores.base import ConflictingIdError
 from apscheduler.jobstores import sqlalchemy
 from apscheduler.util import datetime_to_utc_timestamp
 from apscheduler.util import maybe_ref
+from apscheduler.util import utc_timestamp_to_datetime

 from watcher.common import context
 from watcher.common import service
@@ -32,7 +33,7 @@ try:
 except ImportError:  # pragma: nocover
    import pickle

-from sqlalchemy import Table, MetaData, select, and_
+from sqlalchemy import Table, MetaData, select, and_, null
 from sqlalchemy.exc import IntegrityError


@@ -58,8 +59,7 @@ class WatcherJobStore(sqlalchemy.SQLAlchemyJobStore):
        super(WatcherJobStore, self).__init__(url, engine, tablename,
                                              metadata, pickle_protocol)
        metadata = maybe_ref(metadata) or MetaData()
-        self.jobs_t = Table(tablename, metadata, autoload=True,
-                            autoload_with=engine)
+        self.jobs_t = Table(tablename, metadata, autoload_with=engine)
        service_ident = service.ServiceHeartbeat.get_service_name()
        self.tag = tag or {'host': service_ident[0], 'name': service_ident[1]}
        self.service_id = objects.Service.list(context=context.make_context(),
@@ -79,7 +79,8 @@ class WatcherJobStore(sqlalchemy.SQLAlchemyJobStore):
            'tag': jsonutils.dumps(self.tag)
        })
        try:
-            self.engine.execute(insert)
+            with self.engine.begin() as conn:
+                conn.execute(insert)
        except IntegrityError:
            raise ConflictingIdError(job.id)

@@ -88,20 +89,36 @@ class WatcherJobStore(sqlalchemy.SQLAlchemyJobStore):
        self._fix_paused_jobs_sorting(jobs)
        return jobs

+    def get_next_run_time(self):
+        selectable = select(self.jobs_t.c.next_run_time).\
+            where(self.jobs_t.c.next_run_time != null()).\
+            order_by(self.jobs_t.c.next_run_time).limit(1)
+        with self.engine.begin() as connection:
+            # NOTE(danms): The apscheduler implementation of this gets a
+            # decimal.Decimal back from scalar() which causes
+            # utc_timestamp_to_datetime() to choke since it is expecting a
+            # python float. Assume this is SQLAlchemy 2.0 stuff, so just
+            # coerce to a float here.
+            next_run_time = connection.execute(selectable).scalar()
+            return utc_timestamp_to_datetime(float(next_run_time)
+                                             if next_run_time is not None
+                                             else None)
+
    def _get_jobs(self, *conditions):
        jobs = []
        conditions += (self.jobs_t.c.service_id == self.service_id,)
        selectable = select(
-            [self.jobs_t.c.id, self.jobs_t.c.job_state, self.jobs_t.c.tag]
+            self.jobs_t.c.id, self.jobs_t.c.job_state, self.jobs_t.c.tag
        ).order_by(self.jobs_t.c.next_run_time).where(and_(*conditions))
        failed_job_ids = set()
-        for row in self.engine.execute(selectable):
-            try:
-                jobs.append(self._reconstitute_job(row.job_state))
-            except Exception:
-                self._logger.exception(
-                    'Unable to restore job "%s" -- removing it', row.id)
-                failed_job_ids.add(row.id)
+        with self.engine.begin() as conn:
+            for row in conn.execute(selectable):
+                try:
+                    jobs.append(self._reconstitute_job(row.job_state))
+                except Exception:
+                    self._logger.exception(
+                        'Unable to restore job "%s" -- removing it', row.id)
+                    failed_job_ids.add(row.id)

        # Remove all the jobs we failed to restore
        if failed_job_ids:
--- a/watcher/decision_engine/datasources/base.py
+++ b/watcher/decision_engine/datasources/base.py
@@ -63,7 +63,7 @@ class DataSourceBase(object):
            raise exception.MetricNotAvailable(metric=meter_name)
        return meter

-    def query_retry(self, f, *args, **kwargs):
+    def query_retry(self, f, *args, ignored_exc=None, **kwargs):
        """Attempts to retrieve metrics from the external service

        Attempts to access data from the external service and handles
@@ -71,15 +71,23 @@ class DataSourceBase(object):
        to the value of query_max_retries
        :param f: The method that performs the actual querying for metrics
        :param args: Array of arguments supplied to the method
+        :param ignored_exc: An exception or tuple of exceptions that shouldn't
+                            be retried, for example "NotFound" exceptions.
        :param kwargs: The amount of arguments supplied to the method
        :return: The value as retrieved from the external service
        """

        num_retries = CONF.watcher_datasources.query_max_retries
        timeout = CONF.watcher_datasources.query_timeout
+        ignored_exc = ignored_exc or tuple()
+
        for i in range(num_retries):
            try:
                return f(*args, **kwargs)
+            except ignored_exc as e:
+                LOG.debug("Got an ignored exception (%s) while calling: %s ",
+                          e, f)
+                return
            except Exception as e:
                LOG.exception(e)
                self.query_retry_reset(e)
--- a/watcher/decision_engine/datasources/gnocchi.py
+++ b/watcher/decision_engine/datasources/gnocchi.py
@@ -19,6 +19,7 @@
 from datetime import datetime
 from datetime import timedelta

+from gnocchiclient import exceptions as gnc_exc
 from oslo_config import cfg
 from oslo_log import log

@@ -38,7 +39,7 @@ class GnocchiHelper(base.DataSourceBase):
                      host_inlet_temp='hardware.ipmi.node.temperature',
                      host_airflow='hardware.ipmi.node.airflow',
                      host_power='hardware.ipmi.node.power',
-                      instance_cpu_usage='cpu_util',
+                      instance_cpu_usage='cpu',
                      instance_ram_usage='memory.resident',
                      instance_ram_allocated='memory',
                      instance_l3_cache_usage='cpu_l3_cache',
@@ -84,7 +85,9 @@ class GnocchiHelper(base.DataSourceBase):
            kwargs = dict(query={"=": {"original_resource_id": resource_id}},
                          limit=1)
            resources = self.query_retry(
-                f=self.gnocchi.resource.search, **kwargs)
+                f=self.gnocchi.resource.search,
+                ignored_exc=gnc_exc.NotFound,
+                **kwargs)

            if not resources:
                LOG.warning("The {0} resource {1} could not be "
@@ -93,6 +96,25 @@ class GnocchiHelper(base.DataSourceBase):

            resource_id = resources[0]['id']

+        if meter_name == "instance_cpu_usage":
+            if resource_type != "instance":
+                LOG.warning("Unsupported resource type for metric "
+                            "'instance_cpu_usage': ", resource_type)
+                return
+
+            # The "cpu_util" gauge (percentage) metric has been removed.
+            # We're going to obtain the same result by using the rate of change
+            # aggregate operation.
+            if aggregate not in ("mean", "rate:mean"):
+                LOG.warning("Unsupported aggregate for instance_cpu_usage "
+                            "metric: %s. "
+                            "Supported aggregates: mean, rate:mean ",
+                            aggregate)
+                return
+
+            # TODO(lpetrut): consider supporting other aggregates.
+            aggregate = "rate:mean"
+
        raw_kwargs = dict(
            metric=meter,
            start=start_time,
@@ -105,7 +127,9 @@ class GnocchiHelper(base.DataSourceBase):
        kwargs = {k: v for k, v in raw_kwargs.items() if k and v}

        statistics = self.query_retry(
-            f=self.gnocchi.metric.get_measures, **kwargs)
+            f=self.gnocchi.metric.get_measures,
+            ignored_exc=gnc_exc.NotFound,
+            **kwargs)

        return_value = None
        if statistics:
@@ -117,6 +141,17 @@ class GnocchiHelper(base.DataSourceBase):
                # Airflow from hardware.ipmi.node.airflow is reported as
                # 1/10 th of actual CFM
                return_value *= 10
+            if meter_name == "instance_cpu_usage":
+                # "rate:mean" can return negative values for migrated vms.
+                return_value = max(0, return_value)
+
+                # We're converting the cumulative cpu time (ns) to cpu usage
+                # percentage.
+                vcpus = resource.vcpus
+                if not vcpus:
+                    LOG.warning("instance vcpu count not set, assuming 1")
+                    vcpus = 1
+                return_value *= 100 / (granularity * 10e+8) / vcpus

        return return_value

@@ -132,7 +167,9 @@ class GnocchiHelper(base.DataSourceBase):
            kwargs = dict(query={"=": {"original_resource_id": resource_id}},
                          limit=1)
            resources = self.query_retry(
-                f=self.gnocchi.resource.search, **kwargs)
+                f=self.gnocchi.resource.search,
+                ignored_exc=gnc_exc.NotFound,
+                **kwargs)

            if not resources:
                LOG.warning("The {0} resource {1} could not be "
@@ -152,7 +189,9 @@ class GnocchiHelper(base.DataSourceBase):
        kwargs = {k: v for k, v in raw_kwargs.items() if k and v}

        statistics = self.query_retry(
-            f=self.gnocchi.metric.get_measures, **kwargs)
+            f=self.gnocchi.metric.get_measures,
+            ignored_exc=gnc_exc.NotFound,
+            **kwargs)

        return_value = None
        if statistics:
--- a/watcher/decision_engine/datasources/grafana.py
+++ b/watcher/decision_engine/datasources/grafana.py
@@ -138,7 +138,8 @@ class GrafanaHelper(base.DataSourceBase):
            raise exception.DataSourceNotAvailable(self.NAME)

        resp = requests.get(self._base_url + str(project_id) + '/query',
-                            params=params, headers=self._headers)
+                            params=params, headers=self._headers,
+                            timeout=CONF.grafana_client.http_timeout)
        if resp.status_code == HTTPStatus.OK:
            return resp
        elif resp.status_code == HTTPStatus.BAD_REQUEST:
--- a/watcher/decision_engine/model/collector/ironic.py
+++ b/watcher/decision_engine/model/collector/ironic.py
@@ -81,6 +81,7 @@ class BareMetalModelBuilder(base.BaseModelBuilder):
    def __init__(self, osc):
        self.osc = osc
        self.model = model_root.BaremetalModelRoot()
+        # TODO(lpetrut): add MAAS support
        self.ironic_helper = ironic_helper.IronicHelper(osc=self.osc)

    def add_ironic_node(self, node):
--- a/watcher/decision_engine/model/collector/nova.py
+++ b/watcher/decision_engine/model/collector/nova.py
@@ -48,7 +48,7 @@ class NovaClusterDataModelCollector(base.BaseClusterDataModelCollector):
                    "type": "array",
                    "items": {
                        "anyOf": [
-                            {"$ref": HOST_AGGREGATES + "id"},
+                            {"$ref": HOST_AGGREGATES + "host_aggr_id"},
                            {"$ref": HOST_AGGREGATES + "name"},
                        ]
                    }
@@ -98,7 +98,8 @@ class NovaClusterDataModelCollector(base.BaseClusterDataModelCollector):
                                "type": "array",
                                "items": {
                                    "anyOf": [
-                                        {"$ref": HOST_AGGREGATES + "id"},
+                                        {"$ref":
+                                            HOST_AGGREGATES + "host_aggr_id"},
                                        {"$ref": HOST_AGGREGATES + "name"},
                                    ]
                                }
@@ -129,7 +130,7 @@ class NovaClusterDataModelCollector(base.BaseClusterDataModelCollector):
            "additionalProperties": False
        },
        "host_aggregates": {
-            "id": {
+            "host_aggr_id": {
                "properties": {
                    "id": {
                        "oneOf": [
--- a/watcher/decision_engine/model/model_root.py
+++ b/watcher/decision_engine/model/model_root.py
@@ -157,7 +157,7 @@ class ModelRoot(nx.DiGraph, base.Model):
            if node_list:
                return node_list[0]
            else:
-                raise exception.ComputeResourceNotFound
+                raise exception.ComputeNodeNotFound(name=name)
        except exception.ComputeResourceNotFound:
            raise exception.ComputeNodeNotFound(name=name)

--- a/watcher/decision_engine/strategy/strategies/base.py
+++ b/watcher/decision_engine/strategy/strategies/base.py
@@ -252,9 +252,6 @@ class BaseStrategy(loadable.Loadable, metaclass=abc.ABCMeta):
        if not self.compute_model:
            raise exception.ClusterStateNotDefined()

-        if self.compute_model.stale:
-            raise exception.ClusterStateStale()
-
        LOG.debug(self.compute_model.to_string())

    def execute(self, audit=None):
--- a/watcher/decision_engine/strategy/strategies/saving_energy.py
+++ b/watcher/decision_engine/strategy/strategies/saving_energy.py
@@ -23,6 +23,8 @@ from oslo_log import log

 from watcher._i18n import _
 from watcher.common import exception
+from watcher.common.metal_helper import constants as metal_constants
+from watcher.common.metal_helper import factory as metal_helper_factory
 from watcher.decision_engine.strategy.strategies import base

 LOG = log.getLogger(__name__)
@@ -81,7 +83,7 @@ class SavingEnergy(base.SavingEnergyBaseStrategy):
    def __init__(self, config, osc=None):

        super(SavingEnergy, self).__init__(config, osc)
-        self._ironic_client = None
+        self._metal_helper = None
        self._nova_client = None

        self.with_vms_node_pool = []
@@ -91,10 +93,10 @@ class SavingEnergy(base.SavingEnergyBaseStrategy):
        self.min_free_hosts_num = 1

    @property
-    def ironic_client(self):
-        if not self._ironic_client:
-            self._ironic_client = self.osc.ironic()
-        return self._ironic_client
+    def metal_helper(self):
+        if not self._metal_helper:
+            self._metal_helper = metal_helper_factory.get_helper(self.osc)
+        return self._metal_helper

    @property
    def nova_client(self):
@@ -149,10 +151,10 @@ class SavingEnergy(base.SavingEnergyBaseStrategy):
        :return: None
        """
        params = {'state': state,
-                  'resource_name': node.hostname}
+                  'resource_name': node.get_hypervisor_hostname()}
        self.solution.add_action(
            action_type='change_node_power_state',
-            resource_id=node.uuid,
+            resource_id=node.get_id(),
            input_parameters=params)

    def get_hosts_pool(self):
@@ -162,36 +164,36 @@ class SavingEnergy(base.SavingEnergyBaseStrategy):

        """

-        node_list = self.ironic_client.node.list()
+        node_list = self.metal_helper.list_compute_nodes()
        for node in node_list:
-            node_info = self.ironic_client.node.get(node.uuid)
-            hypervisor_id = node_info.extra.get('compute_node_id', None)
-            if hypervisor_id is None:
-                LOG.warning(('Cannot find compute_node_id in extra '
-                             'of ironic node %s'), node.uuid)
-                continue
-            hypervisor_node = self.nova_client.hypervisors.get(hypervisor_id)
-            if hypervisor_node is None:
-                LOG.warning(('Cannot find hypervisor %s'), hypervisor_id)
-                continue
-            node.hostname = hypervisor_node.hypervisor_hostname
-            hypervisor_node = hypervisor_node.to_dict()
+            hypervisor_node = node.get_hypervisor_node().to_dict()
+
            compute_service = hypervisor_node.get('service', None)
            host_name = compute_service.get('host')
+            LOG.debug("Found hypervisor: %s", hypervisor_node)
            try:
                self.compute_model.get_node_by_name(host_name)
            except exception.ComputeNodeNotFound:
+                LOG.info("The compute model does not contain the host: %s",
+                         host_name)
                continue

-            if not (hypervisor_node.get('state') == 'up'):
-                """filter nodes that are not in 'up' state"""
+            if (node.hv_up_when_powered_off and
+                    hypervisor_node.get('state') != 'up'):
+                # filter nodes that are not in 'up' state
+                LOG.info("Ignoring node that isn't in 'up' state: %s",
+                         host_name)
                continue
            else:
                if (hypervisor_node['running_vms'] == 0):
-                    if (node_info.power_state == 'power on'):
+                    power_state = node.get_power_state()
+                    if power_state == metal_constants.PowerState.ON:
                        self.free_poweron_node_pool.append(node)
-                    elif (node_info.power_state == 'power off'):
+                    elif power_state == metal_constants.PowerState.OFF:
                        self.free_poweroff_node_pool.append(node)
+                    else:
+                        LOG.info("Ignoring node %s, unknown state: %s",
+                                 node, power_state)
                else:
                    self.with_vms_node_pool.append(node)

@@ -202,17 +204,21 @@ class SavingEnergy(base.SavingEnergyBaseStrategy):
                self.min_free_hosts_num)))
        len_poweron = len(self.free_poweron_node_pool)
        len_poweroff = len(self.free_poweroff_node_pool)
+        LOG.debug("need_poweron: %s, len_poweron: %s, len_poweroff: %s",
+                  need_poweron, len_poweron, len_poweroff)
        if len_poweron > need_poweron:
            for node in random.sample(self.free_poweron_node_pool,
                                      (len_poweron - need_poweron)):
-                self.add_action_poweronoff_node(node, 'off')
-                LOG.debug("power off %s", node.uuid)
+                self.add_action_poweronoff_node(node,
+                                                metal_constants.PowerState.OFF)
+                LOG.info("power off %s", node.get_id())
        elif len_poweron < need_poweron:
            diff = need_poweron - len_poweron
            for node in random.sample(self.free_poweroff_node_pool,
                                      min(len_poweroff, diff)):
-                self.add_action_poweronoff_node(node, 'on')
-                LOG.debug("power on %s", node.uuid)
+                self.add_action_poweronoff_node(node,
+                                                metal_constants.PowerState.ON)
+                LOG.info("power on %s", node.get_id())

    def pre_execute(self):
        self._pre_execute()
--- a/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py
+++ b/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py
@@ -18,9 +18,13 @@
 # limitations under the License.
 #

+import collections
+
 from oslo_log import log
+import oslo_utils

 from watcher._i18n import _
+from watcher.applier.actions import migration
 from watcher.common import exception
 from watcher.decision_engine.model import element
 from watcher.decision_engine.strategy.strategies import base
@@ -66,7 +70,8 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):

    AGGREGATE = 'mean'
    DATASOURCE_METRICS = ['instance_ram_allocated', 'instance_cpu_usage',
-                          'instance_ram_usage', 'instance_root_disk_size']
+                          'instance_ram_usage', 'instance_root_disk_size',
+                          'host_cpu_usage', 'host_ram_usage']

    MIGRATION = "migrate"
    CHANGE_NOVA_SERVICE_STATE = "change_nova_service_state"
@@ -76,6 +81,11 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
        self.number_of_migrations = 0
        self.number_of_released_nodes = 0
        self.datasource_instance_data_cache = dict()
+        self.datasource_node_data_cache = dict()
+        # Host metric adjustments that take into account planned
+        # migrations.
+        self.host_metric_delta = collections.defaultdict(
+            lambda: collections.defaultdict(int))

    @classmethod
    def get_name(cls):
@@ -196,12 +206,12 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
        :return: None
        """
        instance_state_str = self.get_instance_state_str(instance)
-        if instance_state_str not in (element.InstanceState.ACTIVE.value,
-                                      element.InstanceState.PAUSED.value):
-            # Watcher currently only supports live VM migration and block live
-            # VM migration which both requires migrated VM to be active.
-            # When supported, the cold migration may be used as a fallback
-            # migration mechanism to move non active VMs.
+        if instance_state_str in (element.InstanceState.ACTIVE.value,
+                                  element.InstanceState.PAUSED.value):
+            migration_type = migration.Migrate.LIVE_MIGRATION
+        elif instance_state_str == element.InstanceState.STOPPED.value:
+            migration_type = migration.Migrate.COLD_MIGRATION
+        else:
            LOG.error(
                'Cannot live migrate: instance_uuid=%(instance_uuid)s, '
                'state=%(instance_state)s.', dict(
@@ -209,8 +219,6 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                    instance_state=instance_state_str))
            return

-        migration_type = 'live'
-
        # Here will makes repeated actions to enable the same compute node,
        # when migrating VMs to the destination node which is disabled.
        # Whether should we remove the same actions in the solution???
@@ -228,6 +236,18 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                destination_node)
            self.number_of_migrations += 1

+            instance_util = self.get_instance_utilization(instance)
+            self.host_metric_delta[source_node.hostname]['cpu'] -= (
+                instance_util['cpu'])
+            # We'll deduce the vm allocated memory.
+            self.host_metric_delta[source_node.hostname]['ram'] -= (
+                instance.memory)
+
+            self.host_metric_delta[destination_node.hostname]['cpu'] += (
+                instance_util['cpu'])
+            self.host_metric_delta[destination_node.hostname]['ram'] += (
+                instance.memory)
+
    def disable_unused_nodes(self):
        """Generate actions for disabling unused nodes.

@@ -290,6 +310,21 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
            disk=instance_disk_util)
        return self.datasource_instance_data_cache.get(instance.uuid)

+    def _get_node_total_utilization(self, node):
+        if node.hostname in self.datasource_node_data_cache:
+            return self.datasource_node_data_cache[node.hostname]
+
+        cpu = self.datasource_backend.get_host_cpu_usage(
+            node, self.period, self.AGGREGATE,
+            self.granularity)
+        ram = self.datasource_backend.get_host_ram_usage(
+            node, self.period, self.AGGREGATE,
+            self.granularity)
+
+        self.datasource_node_data_cache[node.hostname] = dict(
+            cpu=cpu, ram=ram)
+        return self.datasource_node_data_cache[node.hostname]
+
    def get_node_utilization(self, node):
        """Collect cpu, ram and disk utilization statistics of a node.

@@ -307,8 +342,36 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
            node_cpu_util += instance_util['cpu']
            node_ram_util += instance_util['ram']
            node_disk_util += instance_util['disk']
+            LOG.debug("instance utilization: %s %s",
+                      instance, instance_util)

-        return dict(cpu=node_cpu_util, ram=node_ram_util,
+        total_node_util = self._get_node_total_utilization(node)
+        total_node_cpu_util = total_node_util['cpu'] or 0
+        if total_node_cpu_util:
+            total_node_cpu_util = total_node_cpu_util * node.vcpus / 100
+            # account for planned migrations
+            total_node_cpu_util += self.host_metric_delta[node.hostname]['cpu']
+
+        total_node_ram_util = total_node_util['ram'] or 0
+        if total_node_ram_util:
+            total_node_ram_util /= oslo_utils.units.Ki
+            total_node_ram_util += self.host_metric_delta[node.hostname]['ram']
+
+        LOG.debug(
+            "node utilization: %s. "
+            "total instance cpu: %s, "
+            "total instance ram: %s, "
+            "total instance disk: %s, "
+            "total host cpu: %s, "
+            "total host ram: %s, "
+            "node delta usage: %s.",
+            node,
+            node_cpu_util, node_ram_util, node_disk_util,
+            total_node_cpu_util, total_node_ram_util,
+            self.host_metric_delta[node.hostname])
+
+        return dict(cpu=max(node_cpu_util, total_node_cpu_util),
+                    ram=max(node_ram_util, total_node_ram_util),
                    disk=node_disk_util)

    def get_node_capacity(self, node):
@@ -388,8 +451,15 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
        instance_utilization = self.get_instance_utilization(instance)
        metrics = ['cpu', 'ram', 'disk']
        for m in metrics:
-            if (instance_utilization[m] + node_utilization[m] >
-                    node_capacity[m] * cc[m]):
+            fits = (instance_utilization[m] + node_utilization[m] <=
+                    node_capacity[m] * cc[m])
+            LOG.debug(
+                "Instance fits: %s, metric: %s, instance: %s, "
+                "node: %s, instance utilization: %s, "
+                "node utilization: %s, node capacity: %s, cc: %s",
+                fits, m, instance, node, instance_utilization[m],
+                node_utilization[m], node_capacity[m], cc[m])
+            if not fits:
                return False
        return True

@@ -424,6 +494,9 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                for a in actions:
                    self.solution.actions.remove(a)
                    self.number_of_migrations -= 1
+                LOG.info("Optimized migrations: %s. "
+                         "Source: %s, destination: %s", actions,
+                         src_name, dst_name)
                src_node = self.compute_model.get_node_by_name(src_name)
                dst_node = self.compute_model.get_node_by_name(dst_name)
                instance = self.compute_model.get_instance_by_uuid(
@@ -460,6 +533,8 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                        key=lambda x: self.get_instance_utilization(
                            x)['cpu']
                ):
+                    LOG.info("Node %s overloaded, attempting to reduce load.",
+                             node)
                    # skip exclude instance when migrating
                    if instance.watcher_exclude:
                        LOG.debug("Instance is excluded by scope, "
@@ -468,11 +543,19 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                    for destination_node in reversed(sorted_nodes):
                        if self.instance_fits(
                                instance, destination_node, cc):
+                            LOG.info("Offload: found fitting "
+                                     "destination (%s) for instance: %s. "
+                                     "Planning migration.",
+                                     destination_node, instance.uuid)
                            self.add_migration(instance, node,
                                               destination_node)
                            break
                    if not self.is_overloaded(node, cc):
+                        LOG.info("Node %s no longer overloaded.", node)
                        break
+                    else:
+                        LOG.info("Node still overloaded (%s), "
+                                 "continuing offload phase.", node)

    def consolidation_phase(self, cc):
        """Perform consolidation phase.
@@ -508,6 +591,10 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
                        break
                    if self.instance_fits(
                            instance, destination_node, cc):
+                        LOG.info("Consolidation: found fitting "
+                                 "destination (%s) for instance: %s. "
+                                 "Planning migration.",
+                                 destination_node, instance.uuid)
                        self.add_migration(instance, node,
                                           destination_node)
                        break
--- a/watcher/decision_engine/strategy/strategies/workload_balance.py
+++ b/watcher/decision_engine/strategy/strategies/workload_balance.py
@@ -295,7 +295,7 @@ class WorkloadBalance(base.WorkloadStabilizationBaseStrategy):
                        self.threshold)
            return self.solution

-        # choose the server with largest cpu_util
+        # choose the server with largest cpu usage
        source_nodes = sorted(source_nodes,
                              reverse=True,
                              key=lambda x: (x[self._meter]))
--- a/watcher/locale/en_GB/LC_MESSAGES/watcher.po
+++ b/watcher/locale/en_GB/LC_MESSAGES/watcher.po
@@ -1,15 +1,16 @@
 # Andi Chandler <andi@gowling.com>, 2017. #zanata
 # Andi Chandler <andi@gowling.com>, 2018. #zanata
 # Andi Chandler <andi@gowling.com>, 2020. #zanata
+# Andi Chandler <andi@gowling.com>, 2022. #zanata
 msgid ""
 msgstr ""
 "Project-Id-Version: watcher VERSION\n"
 "Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
-"POT-Creation-Date: 2020-10-27 04:14+0000\n"
+"POT-Creation-Date: 2022-08-29 03:03+0000\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"PO-Revision-Date: 2020-10-28 11:02+0000\n"
+"PO-Revision-Date: 2022-05-31 08:38+0000\n"
 "Last-Translator: Andi Chandler <andi@gowling.com>\n"
 "Language-Team: English (United Kingdom)\n"
 "Language: en_GB\n"
@@ -507,6 +508,9 @@ msgstr ""
 msgid "Plugins"
 msgstr "Plugins"

+msgid "Policy File JSON to YAML Migration"
+msgstr "Policy File JSON to YAML Migration"
+
 #, python-format
 msgid "Policy doesn't allow %(action)s to be performed."
 msgstr "Policy doesn't allow %(action)s to be performed."
--- a/watcher/tests/applier/actions/test_change_node_power_state.py
+++ b/watcher/tests/applier/actions/test_change_node_power_state.py
@@ -19,134 +19,151 @@ import jsonschema

 from watcher.applier.actions import base as baction
 from watcher.applier.actions import change_node_power_state
-from watcher.common import clients
+from watcher.common.metal_helper import constants as m_constants
+from watcher.common.metal_helper import factory as m_helper_factory
 from watcher.tests import base
+from watcher.tests.decision_engine import fake_metal_helper

 COMPUTE_NODE = "compute-1"


-@mock.patch.object(clients.OpenStackClients, 'nova')
-@mock.patch.object(clients.OpenStackClients, 'ironic')
 class TestChangeNodePowerState(base.TestCase):

    def setUp(self):
        super(TestChangeNodePowerState, self).setUp()

+        p_m_factory = mock.patch.object(m_helper_factory, 'get_helper')
+        m_factory = p_m_factory.start()
+        self._metal_helper = m_factory.return_value
+        self.addCleanup(p_m_factory.stop)
+
+        # Let's avoid unnecessary sleep calls while running the test.
+        p_sleep = mock.patch('time.sleep')
+        p_sleep.start()
+        self.addCleanup(p_sleep.stop)
+
        self.input_parameters = {
            baction.BaseAction.RESOURCE_ID: COMPUTE_NODE,
-            "state": change_node_power_state.NodeState.POWERON.value,
+            "state": m_constants.PowerState.ON.value,
        }
        self.action = change_node_power_state.ChangeNodePowerState(
            mock.Mock())
        self.action.input_parameters = self.input_parameters

-    def test_parameters_down(self, mock_ironic, mock_nova):
+    def test_parameters_down(self):
        self.action.input_parameters = {
            baction.BaseAction.RESOURCE_ID: COMPUTE_NODE,
            self.action.STATE:
-                change_node_power_state.NodeState.POWEROFF.value}
+                m_constants.PowerState.OFF.value}
        self.assertTrue(self.action.validate_parameters())

-    def test_parameters_up(self, mock_ironic, mock_nova):
+    def test_parameters_up(self):
        self.action.input_parameters = {
            baction.BaseAction.RESOURCE_ID: COMPUTE_NODE,
            self.action.STATE:
-                change_node_power_state.NodeState.POWERON.value}
+                m_constants.PowerState.ON.value}
        self.assertTrue(self.action.validate_parameters())

-    def test_parameters_exception_wrong_state(self, mock_ironic, mock_nova):
+    def test_parameters_exception_wrong_state(self):
        self.action.input_parameters = {
            baction.BaseAction.RESOURCE_ID: COMPUTE_NODE,
            self.action.STATE: 'error'}
        self.assertRaises(jsonschema.ValidationError,
                          self.action.validate_parameters)

-    def test_parameters_resource_id_empty(self, mock_ironic, mock_nova):
+    def test_parameters_resource_id_empty(self):
        self.action.input_parameters = {
            self.action.STATE:
-                change_node_power_state.NodeState.POWERON.value,
+                m_constants.PowerState.ON.value,
        }
        self.assertRaises(jsonschema.ValidationError,
                          self.action.validate_parameters)

-    def test_parameters_applies_add_extra(self, mock_ironic, mock_nova):
+    def test_parameters_applies_add_extra(self):
        self.action.input_parameters = {"extra": "failed"}
        self.assertRaises(jsonschema.ValidationError,
                          self.action.validate_parameters)

-    def test_change_service_state_pre_condition(self, mock_ironic, mock_nova):
+    def test_change_service_state_pre_condition(self):
        try:
            self.action.pre_condition()
        except Exception as exc:
            self.fail(exc)

-    def test_change_node_state_post_condition(self, mock_ironic, mock_nova):
+    def test_change_node_state_post_condition(self):
        try:
            self.action.post_condition()
        except Exception as exc:
            self.fail(exc)

-    def test_execute_node_service_state_with_poweron_target(
-            self, mock_ironic, mock_nova):
-        mock_irclient = mock_ironic.return_value
+    def test_execute_node_service_state_with_poweron_target(self):
        self.action.input_parameters["state"] = (
-            change_node_power_state.NodeState.POWERON.value)
-        mock_irclient.node.get.side_effect = [
-            mock.MagicMock(power_state='power off'),
-            mock.MagicMock(power_state='power on')]
+            m_constants.PowerState.ON.value)
+        mock_nodes = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON)
+        ]
+        self._metal_helper.get_node.side_effect = mock_nodes

        result = self.action.execute()
        self.assertTrue(result)

-        mock_irclient.node.set_power_state.assert_called_once_with(
-            COMPUTE_NODE, change_node_power_state.NodeState.POWERON.value)
+        mock_nodes[0].set_power_state.assert_called_once_with(
+            m_constants.PowerState.ON.value)

-    def test_execute_change_node_state_with_poweroff_target(
-            self, mock_ironic, mock_nova):
-        mock_irclient = mock_ironic.return_value
-        mock_nvclient = mock_nova.return_value
-        mock_get = mock.MagicMock()
-        mock_get.to_dict.return_value = {'running_vms': 0}
-        mock_nvclient.hypervisors.get.return_value = mock_get
+    def test_execute_change_node_state_with_poweroff_target(self):
        self.action.input_parameters["state"] = (
-            change_node_power_state.NodeState.POWEROFF.value)
-        mock_irclient.node.get.side_effect = [
-            mock.MagicMock(power_state='power on'),
-            mock.MagicMock(power_state='power on'),
-            mock.MagicMock(power_state='power off')]
+            m_constants.PowerState.OFF.value)
+
+        mock_nodes = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF)
+        ]
+        self._metal_helper.get_node.side_effect = mock_nodes
+
        result = self.action.execute()
        self.assertTrue(result)

-        mock_irclient.node.set_power_state.assert_called_once_with(
-            COMPUTE_NODE, change_node_power_state.NodeState.POWEROFF.value)
+        mock_nodes[0].set_power_state.assert_called_once_with(
+            m_constants.PowerState.OFF.value)

-    def test_revert_change_node_state_with_poweron_target(
-            self, mock_ironic, mock_nova):
-        mock_irclient = mock_ironic.return_value
-        mock_nvclient = mock_nova.return_value
-        mock_get = mock.MagicMock()
-        mock_get.to_dict.return_value = {'running_vms': 0}
-        mock_nvclient.hypervisors.get.return_value = mock_get
+    def test_revert_change_node_state_with_poweron_target(self):
        self.action.input_parameters["state"] = (
-            change_node_power_state.NodeState.POWERON.value)
-        mock_irclient.node.get.side_effect = [
-            mock.MagicMock(power_state='power on'),
-            mock.MagicMock(power_state='power on'),
-            mock.MagicMock(power_state='power off')]
+            m_constants.PowerState.ON.value)
+
+        mock_nodes = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF)
+        ]
+        self._metal_helper.get_node.side_effect = mock_nodes
+
        self.action.revert()

-        mock_irclient.node.set_power_state.assert_called_once_with(
-            COMPUTE_NODE, change_node_power_state.NodeState.POWEROFF.value)
+        mock_nodes[0].set_power_state.assert_called_once_with(
+            m_constants.PowerState.OFF.value)

-    def test_revert_change_node_state_with_poweroff_target(
-            self, mock_ironic, mock_nova):
-        mock_irclient = mock_ironic.return_value
+    def test_revert_change_node_state_with_poweroff_target(self):
        self.action.input_parameters["state"] = (
-            change_node_power_state.NodeState.POWEROFF.value)
-        mock_irclient.node.get.side_effect = [
-            mock.MagicMock(power_state='power off'),
-            mock.MagicMock(power_state='power on')]
+            m_constants.PowerState.OFF.value)
+        mock_nodes = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON)
+        ]
+        self._metal_helper.get_node.side_effect = mock_nodes
+
        self.action.revert()

-        mock_irclient.node.set_power_state.assert_called_once_with(
-            COMPUTE_NODE, change_node_power_state.NodeState.POWERON.value)
+        mock_nodes[0].set_power_state.assert_called_once_with(
+            m_constants.PowerState.ON.value)
--- a/watcher/tests/common/metal_helper/init.py
+++ b/watcher/tests/common/metal_helper/init.py
--- a/watcher/tests/common/metal_helper/test_base.py
+++ b/watcher/tests/common/metal_helper/test_base.py
@@ -0,0 +1,96 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from unittest import mock
+
+from watcher.common import exception
+from watcher.common.metal_helper import base as m_helper_base
+from watcher.common.metal_helper import constants as m_constants
+from watcher.tests import base
+
+
+# The base classes have abstract methods, we'll need to
+# stub them.
+class MockMetalNode(m_helper_base.BaseMetalNode):
+    def get_power_state(self):
+        raise NotImplementedError()
+
+    def get_id(self):
+        raise NotImplementedError()
+
+    def power_on(self):
+        raise NotImplementedError()
+
+    def power_off(self):
+        raise NotImplementedError()
+
+
+class MockMetalHelper(m_helper_base.BaseMetalHelper):
+    def list_compute_nodes(self):
+        pass
+
+    def get_node(self, node_id):
+        pass
+
+
+class TestBaseMetalNode(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._nova_node = mock.Mock()
+        self._node = MockMetalNode(self._nova_node)
+
+    def test_get_hypervisor_node(self):
+        self.assertEqual(
+            self._nova_node,
+            self._node.get_hypervisor_node())
+
+    def test_get_hypervisor_node_missing(self):
+        node = MockMetalNode()
+        self.assertRaises(
+            exception.Invalid,
+            node.get_hypervisor_node)
+
+    def test_get_hypervisor_hostname(self):
+        self.assertEqual(
+            self._nova_node.hypervisor_hostname,
+            self._node.get_hypervisor_hostname())
+
+    @mock.patch.object(MockMetalNode, 'power_on')
+    @mock.patch.object(MockMetalNode, 'power_off')
+    def test_set_power_state(self,
+                             mock_power_off, mock_power_on):
+        self._node.set_power_state(m_constants.PowerState.ON)
+        mock_power_on.assert_called_once_with()
+
+        self._node.set_power_state(m_constants.PowerState.OFF)
+        mock_power_off.assert_called_once_with()
+
+        self.assertRaises(
+            exception.UnsupportedActionType,
+            self._node.set_power_state,
+            m_constants.PowerState.UNKNOWN)
+
+
+class TestBaseMetalHelper(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._osc = mock.Mock()
+        self._helper = MockMetalHelper(self._osc)
+
+    def test_nova_client_attr(self):
+        self.assertEqual(self._osc.nova.return_value,
+                         self._helper.nova_client)
--- a/watcher/tests/common/metal_helper/test_factory.py
+++ b/watcher/tests/common/metal_helper/test_factory.py
@@ -0,0 +1,38 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from unittest import mock
+
+from watcher.common import clients
+from watcher.common.metal_helper import factory
+from watcher.common.metal_helper import ironic
+from watcher.common.metal_helper import maas
+from watcher.tests import base
+
+
+class TestMetalHelperFactory(base.TestCase):
+
+    @mock.patch.object(clients, 'OpenStackClients')
+    @mock.patch.object(maas, 'MaasHelper')
+    @mock.patch.object(ironic, 'IronicHelper')
+    def test_factory(self, mock_ironic, mock_maas, mock_osc):
+        self.assertEqual(
+            mock_ironic.return_value,
+            factory.get_helper())
+
+        self.config(url="fake_maas_url", group="maas_client")
+        self.assertEqual(
+            mock_maas.return_value,
+            factory.get_helper())
--- a/watcher/tests/common/metal_helper/test_ironic.py
+++ b/watcher/tests/common/metal_helper/test_ironic.py
@@ -0,0 +1,128 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from unittest import mock
+
+from watcher.common.metal_helper import constants as m_constants
+from watcher.common.metal_helper import ironic
+from watcher.tests import base
+
+
+class TestIronicNode(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._wrapped_node = mock.Mock()
+        self._nova_node = mock.Mock()
+        self._ironic_client = mock.Mock()
+
+        self._node = ironic.IronicNode(
+            self._wrapped_node, self._nova_node, self._ironic_client)
+
+    def test_get_power_state(self):
+        states = (
+            "power on",
+            "power off",
+            "rebooting",
+            "soft power off",
+            "soft reboot",
+            'SomeOtherState')
+        type(self._wrapped_node).power_state = mock.PropertyMock(
+            side_effect=states)
+
+        expected_states = (
+            m_constants.PowerState.ON,
+            m_constants.PowerState.OFF,
+            m_constants.PowerState.ON,
+            m_constants.PowerState.OFF,
+            m_constants.PowerState.ON,
+            m_constants.PowerState.UNKNOWN)
+
+        for expected_state in expected_states:
+            actual_state = self._node.get_power_state()
+            self.assertEqual(expected_state, actual_state)
+
+    def test_get_id(self):
+        self.assertEqual(
+            self._wrapped_node.uuid,
+            self._node.get_id())
+
+    def test_power_on(self):
+        self._node.power_on()
+        self._ironic_client.node.set_power_state.assert_called_once_with(
+            self._wrapped_node.uuid, "on")
+
+    def test_power_off(self):
+        self._node.power_off()
+        self._ironic_client.node.set_power_state.assert_called_once_with(
+            self._wrapped_node.uuid, "off")
+
+
+class TestIronicHelper(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._mock_osc = mock.Mock()
+        self._mock_nova_client = self._mock_osc.nova.return_value
+        self._mock_ironic_client = self._mock_osc.ironic.return_value
+        self._helper = ironic.IronicHelper(osc=self._mock_osc)
+
+    def test_list_compute_nodes(self):
+        mock_machines = [
+            mock.Mock(
+                extra=dict(compute_node_id=mock.sentinel.compute_node_id)),
+            mock.Mock(
+                extra=dict(compute_node_id=mock.sentinel.compute_node_id2)),
+            mock.Mock(
+                extra=dict())
+        ]
+        mock_hypervisor = mock.Mock()
+
+        self._mock_ironic_client.node.list.return_value = mock_machines
+        self._mock_ironic_client.node.get.side_effect = mock_machines
+        self._mock_nova_client.hypervisors.get.side_effect = (
+            mock_hypervisor, None)
+
+        out_nodes = self._helper.list_compute_nodes()
+        self.assertEqual(1, len(out_nodes))
+
+        out_node = out_nodes[0]
+        self.assertIsInstance(out_node, ironic.IronicNode)
+        self.assertEqual(mock_hypervisor, out_node._nova_node)
+        self.assertEqual(mock_machines[0], out_node._ironic_node)
+        self.assertEqual(self._mock_ironic_client, out_node._ironic_client)
+
+    def test_get_node(self):
+        mock_machine = mock.Mock(
+            extra=dict(compute_node_id=mock.sentinel.compute_node_id))
+        self._mock_ironic_client.node.get.return_value = mock_machine
+
+        out_node = self._helper.get_node(mock.sentinel.id)
+
+        self.assertEqual(self._mock_nova_client.hypervisors.get.return_value,
+                         out_node._nova_node)
+        self.assertEqual(self._mock_ironic_client, out_node._ironic_client)
+        self.assertEqual(mock_machine, out_node._ironic_node)
+
+    def test_get_node_not_a_hypervisor(self):
+        mock_machine = mock.Mock(extra=dict(compute_node_id=None))
+        self._mock_ironic_client.node.get.return_value = mock_machine
+
+        out_node = self._helper.get_node(mock.sentinel.id)
+
+        self._mock_nova_client.hypervisors.get.assert_not_called()
+        self.assertIsNone(out_node._nova_node)
+        self.assertEqual(self._mock_ironic_client, out_node._ironic_client)
+        self.assertEqual(mock_machine, out_node._ironic_node)
--- a/watcher/tests/common/metal_helper/test_maas.py
+++ b/watcher/tests/common/metal_helper/test_maas.py
@@ -0,0 +1,126 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from unittest import mock
+
+try:
+    from maas.client import enum as maas_enum
+except ImportError:
+    maas_enum = None
+
+from watcher.common.metal_helper import constants as m_constants
+from watcher.common.metal_helper import maas
+from watcher.tests import base
+
+
+class TestMaasNode(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._wrapped_node = mock.Mock()
+        self._nova_node = mock.Mock()
+        self._maas_client = mock.Mock()
+
+        self._node = maas.MaasNode(
+            self._wrapped_node, self._nova_node, self._maas_client)
+
+    def test_get_power_state(self):
+        if not maas_enum:
+            self.skipTest("python-libmaas not intalled.")
+
+        self._wrapped_node.query_power_state.side_effect = (
+            maas_enum.PowerState.ON,
+            maas_enum.PowerState.OFF,
+            maas_enum.PowerState.ERROR,
+            maas_enum.PowerState.UNKNOWN,
+            'SomeOtherState')
+
+        expected_states = (
+            m_constants.PowerState.ON,
+            m_constants.PowerState.OFF,
+            m_constants.PowerState.ERROR,
+            m_constants.PowerState.UNKNOWN,
+            m_constants.PowerState.UNKNOWN)
+
+        for expected_state in expected_states:
+            actual_state = self._node.get_power_state()
+            self.assertEqual(expected_state, actual_state)
+
+    def test_get_id(self):
+        self.assertEqual(
+            self._wrapped_node.system_id,
+            self._node.get_id())
+
+    def test_power_on(self):
+        self._node.power_on()
+        self._wrapped_node.power_on.assert_called_once_with()
+
+    def test_power_off(self):
+        self._node.power_off()
+        self._wrapped_node.power_off.assert_called_once_with()
+
+
+class TestMaasHelper(base.TestCase):
+    def setUp(self):
+        super().setUp()
+
+        self._mock_osc = mock.Mock()
+        self._mock_nova_client = self._mock_osc.nova.return_value
+        self._mock_maas_client = self._mock_osc.maas.return_value
+        self._helper = maas.MaasHelper(osc=self._mock_osc)
+
+    def test_list_compute_nodes(self):
+        compute_fqdn = "compute-0"
+        # some other MAAS node, not a Nova node
+        ctrl_fqdn = "ctrl-1"
+
+        mock_machines = [
+            mock.Mock(fqdn=compute_fqdn,
+                      system_id=mock.sentinel.compute_node_id),
+            mock.Mock(fqdn=ctrl_fqdn,
+                      system_id=mock.sentinel.ctrl_node_id),
+        ]
+        mock_hypervisors = [
+            mock.Mock(hypervisor_hostname=compute_fqdn),
+        ]
+
+        self._mock_maas_client.machines.list.return_value = mock_machines
+        self._mock_nova_client.hypervisors.list.return_value = mock_hypervisors
+
+        out_nodes = self._helper.list_compute_nodes()
+        self.assertEqual(1, len(out_nodes))
+
+        out_node = out_nodes[0]
+        self.assertIsInstance(out_node, maas.MaasNode)
+        self.assertEqual(mock.sentinel.compute_node_id, out_node.get_id())
+        self.assertEqual(compute_fqdn, out_node.get_hypervisor_hostname())
+
+    def test_get_node(self):
+        mock_machine = mock.Mock(fqdn='compute-0')
+        self._mock_maas_client.machines.get.return_value = mock_machine
+
+        mock_compute_nodes = [
+            mock.Mock(hypervisor_hostname="compute-011"),
+            mock.Mock(hypervisor_hostname="compute-0"),
+            mock.Mock(hypervisor_hostname="compute-01"),
+        ]
+        self._mock_nova_client.hypervisors.search.return_value = (
+            mock_compute_nodes)
+
+        out_node = self._helper.get_node(mock.sentinel.id)
+
+        self.assertEqual(mock_compute_nodes[1], out_node._nova_node)
+        self.assertEqual(self._mock_maas_client, out_node._maas_client)
+        self.assertEqual(mock_machine, out_node._maas_node)
--- a/watcher/tests/common/test_clients.py
+++ b/watcher/tests/common/test_clients.py
@@ -10,8 +10,6 @@
 # License for the specific language governing permissions and limitations
 # under the License.

-from ceilometerclient import client as ceclient
-import ceilometerclient.v2.client as ceclient_v2
 from unittest import mock

 from cinderclient import client as ciclient
@@ -270,56 +268,6 @@ class TestClients(base.TestCase):
        cinder_cached = osc.cinder()
        self.assertEqual(cinder, cinder_cached)

-    @mock.patch.object(ceclient, 'Client')
-    @mock.patch.object(clients.OpenStackClients, 'session')
-    def test_clients_ceilometer(self, mock_session, mock_call):
-        osc = clients.OpenStackClients()
-        osc._ceilometer = None
-        osc.ceilometer()
-        mock_call.assert_called_once_with(
-            CONF.ceilometer_client.api_version,
-            None,
-            endpoint_type=CONF.ceilometer_client.endpoint_type,
-            region_name=CONF.ceilometer_client.region_name,
-            session=mock_session)
-
-    @mock.patch.object(clients.OpenStackClients, 'session')
-    @mock.patch.object(ceclient_v2.Client, '_get_redirect_client')
-    def test_clients_ceilometer_diff_vers(self, mock_get_redirect_client,
-                                          mock_session):
-        '''ceilometerclient currently only has one version (v2)'''
-        mock_get_redirect_client.return_value = [mock.Mock(), mock.Mock()]
-        CONF.set_override('api_version', '2',
-                          group='ceilometer_client')
-        osc = clients.OpenStackClients()
-        osc._ceilometer = None
-        osc.ceilometer()
-        self.assertEqual(ceclient_v2.Client,
-                         type(osc.ceilometer()))
-
-    @mock.patch.object(clients.OpenStackClients, 'session')
-    @mock.patch.object(ceclient_v2.Client, '_get_redirect_client')
-    def test_clients_ceilometer_diff_endpoint(self, mock_get_redirect_client,
-                                              mock_session):
-        mock_get_redirect_client.return_value = [mock.Mock(), mock.Mock()]
-        CONF.set_override('endpoint_type', 'publicURL',
-                          group='ceilometer_client')
-        osc = clients.OpenStackClients()
-        osc._ceilometer = None
-        osc.ceilometer()
-        self.assertEqual('publicURL', osc.ceilometer().http_client.interface)
-
-    @mock.patch.object(clients.OpenStackClients, 'session')
-    @mock.patch.object(ceclient_v2.Client, '_get_redirect_client')
-    def test_clients_ceilometer_cached(self, mock_get_redirect_client,
-                                       mock_session):
-        mock_get_redirect_client.return_value = [mock.Mock(), mock.Mock()]
-        osc = clients.OpenStackClients()
-        osc._ceilometer = None
-        ceilometer = osc.ceilometer()
-        ceilometer_cached = osc.ceilometer()
-        self.assertEqual(ceilometer, ceilometer_cached)
-
    @mock.patch.object(netclient, 'Client')
    @mock.patch.object(clients.OpenStackClients, 'session')
    def test_clients_neutron(self, mock_session, mock_call):
--- a/watcher/tests/common/test_service.py
+++ b/watcher/tests/common/test_service.py
@@ -80,13 +80,13 @@ class TestService(base.TestCase):
        super(TestService, self).setUp()

    @mock.patch.object(om.rpc.server, "RPCServer")
-    def test_start(self, m_handler):
+    def _test_start(self, m_handler):
        dummy_service = service.Service(DummyManager)
        dummy_service.start()
        self.assertEqual(1, m_handler.call_count)

    @mock.patch.object(om.rpc.server, "RPCServer")
-    def test_stop(self, m_handler):
+    def _test_stop(self, m_handler):
        dummy_service = service.Service(DummyManager)
        dummy_service.stop()
        self.assertEqual(1, m_handler.call_count)
--- a/watcher/tests/common/test_utils.py
+++ b/watcher/tests/common/test_utils.py
@@ -0,0 +1,52 @@
+# Copyright 2023 Cloudbase Solutions
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import asyncio
+import time
+from unittest import mock
+
+from watcher.common import utils
+from watcher.tests import base
+
+
+class TestCommonUtils(base.TestCase):
+    async def test_coro(self, sleep=0, raise_exc=None):
+        time.sleep(sleep)
+        if raise_exc:
+            raise raise_exc
+        return mock.sentinel.ret_val
+
+    def test_async_compat(self):
+        ret_val = utils.async_compat_call(self.test_coro)
+        self.assertEqual(mock.sentinel.ret_val, ret_val)
+
+    def test_async_compat_exc(self):
+        self.assertRaises(
+            IOError,
+            utils.async_compat_call,
+            self.test_coro,
+            raise_exc=IOError('fake error'))
+
+    def test_async_compat_timeout(self):
+        # Timeout not reached.
+        ret_val = utils.async_compat_call(self.test_coro, timeout=10)
+        self.assertEqual(mock.sentinel.ret_val, ret_val)
+
+        # Timeout reached.
+        self.assertRaises(
+            asyncio.TimeoutError,
+            utils.async_compat_call,
+            self.test_coro,
+            sleep=0.5, timeout=0.1)
--- a/watcher/tests/decision_engine/datasources/test_ceilometer_helper.py
+++ b/watcher/tests/decision_engine/datasources/test_ceilometer_helper.py
@@ -1,168 +0,0 @@
-# -*- encoding: utf-8 -*-
-# Copyright (c) 2015 b<>com
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from unittest import mock
-
-from watcher.common import clients
-from watcher.common import exception
-from watcher.decision_engine.datasources import ceilometer as ceilometer_helper
-from watcher.tests import base
-
-
-@mock.patch.object(clients.OpenStackClients, 'ceilometer')
-class TestCeilometerHelper(base.BaseTestCase):
-
-    def setUp(self):
-        super(TestCeilometerHelper, self).setUp()
-        self.osc_mock = mock.Mock()
-        self.helper = ceilometer_helper.CeilometerHelper(osc=self.osc_mock)
-        stat_agg_patcher = mock.patch.object(
-            self.helper, 'statistic_aggregation',
-            spec=ceilometer_helper.CeilometerHelper.statistic_aggregation)
-        self.mock_aggregation = stat_agg_patcher.start()
-        self.addCleanup(stat_agg_patcher.stop)
-
-    def test_build_query(self, mock_ceilometer):
-        mock_ceilometer.return_value = mock.MagicMock()
-        cm = ceilometer_helper.CeilometerHelper()
-        expected = [{'field': 'user_id', 'op': 'eq', 'value': u'user_id'},
-                    {'field': 'project_id', 'op': 'eq', 'value': u'tenant_id'},
-                    {'field': 'resource_id', 'op': 'eq',
-                     'value': u'resource_id'}]
-
-        query = cm.build_query(user_id="user_id",
-                               tenant_id="tenant_id",
-                               resource_id="resource_id",
-                               user_ids=["user_ids"],
-                               tenant_ids=["tenant_ids"],
-                               resource_ids=["resource_ids"])
-        self.assertEqual(expected, query)
-
-    def test_statistic_aggregation(self, mock_ceilometer):
-        ceilometer = mock.MagicMock()
-        statistic = mock.MagicMock()
-        expected_result = 100
-        statistic[-1]._info = {'aggregate': {'avg': expected_result}}
-        ceilometer.statistics.list.return_value = statistic
-        mock_ceilometer.return_value = ceilometer
-        cm = ceilometer_helper.CeilometerHelper()
-        val = cm.statistic_aggregation(
-            resource=mock.Mock(id="INSTANCE_ID"),
-            resource_type='instance',
-            meter_name="instance_cpu_usage",
-            period="7300",
-            granularity=None
-        )
-        self.assertEqual(expected_result, val)
-
-    def test_statistic_aggregation_metric_unavailable(self, mock_ceilometer):
-        helper = ceilometer_helper.CeilometerHelper()
-
-        # invalidate instance_cpu_usage in metric map
-        original_metric_value = helper.METRIC_MAP.get('instance_cpu_usage')
-        helper.METRIC_MAP.update(
-            instance_cpu_usage=None
-        )
-
-        self.assertRaises(
-            exception.MetricNotAvailable,
-            helper.statistic_aggregation, resource=mock.Mock(id="INSTANCE_ID"),
-            resource_type='instance', meter_name="instance_cpu_usage",
-            period="7300",
-            granularity=None
-        )
-
-        # restore the metric map as it is a static attribute that does not get
-        # restored between unit tests!
-        helper.METRIC_MAP.update(
-            instance_cpu_usage=original_metric_value
-        )
-
-    def test_get_host_cpu_usage(self, mock_ceilometer):
-        self.helper.get_host_cpu_usage('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_cpu_usage', 600, 'mean', None)
-
-    def test_get_host_ram_usage(self, mock_ceilometer):
-        self.helper.get_host_ram_usage('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_ram_usage', 600, 'mean', None)
-
-    def test_get_host_outlet_temp(self, mock_ceilometer):
-        self.helper.get_host_outlet_temp('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_outlet_temp', 600, 'mean', None)
-
-    def test_get_host_inlet_temp(self, mock_ceilometer):
-        self.helper.get_host_inlet_temp('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_inlet_temp', 600, 'mean', None)
-
-    def test_get_host_airflow(self, mock_ceilometer):
-        self.helper.get_host_airflow('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_airflow', 600, 'mean', None)
-
-    def test_get_host_power(self, mock_ceilometer):
-        self.helper.get_host_power('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'compute_node', 'host_power', 600, 'mean', None)
-
-    def test_get_instance_cpu_usage(self, mock_ceilometer):
-        self.helper.get_instance_cpu_usage('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'instance', 'instance_cpu_usage', 600, 'mean',
-            None)
-
-    def test_get_instance_ram_usage(self, mock_ceilometer):
-        self.helper.get_instance_ram_usage('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'instance', 'instance_ram_usage', 600, 'mean',
-            None)
-
-    def test_get_instance_ram_allocated(self, mock_ceilometer):
-        self.helper.get_instance_ram_allocated('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'instance', 'instance_ram_allocated', 600, 'mean',
-            None)
-
-    def test_get_instance_l3_cache_usage(self, mock_ceilometer):
-        self.helper.get_instance_l3_cache_usage('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'instance', 'instance_l3_cache_usage', 600, 'mean',
-            None)
-
-    def test_get_instance_root_disk_size(self, mock_ceilometer):
-        self.helper.get_instance_root_disk_size('compute1', 600, 'mean')
-        self.mock_aggregation.assert_called_once_with(
-            'compute1', 'instance', 'instance_root_disk_size', 600, 'mean',
-            None)
-
-    def test_check_availability(self, mock_ceilometer):
-        ceilometer = mock.MagicMock()
-        ceilometer.resources.list.return_value = True
-        mock_ceilometer.return_value = ceilometer
-        helper = ceilometer_helper.CeilometerHelper()
-        result = helper.check_availability()
-        self.assertEqual('available', result)
-
-    def test_check_availability_with_failure(self, mock_ceilometer):
-        ceilometer = mock.MagicMock()
-        ceilometer.resources.list.side_effect = Exception()
-        mock_ceilometer.return_value = ceilometer
-        helper = ceilometer_helper.CeilometerHelper()
-
-        self.assertEqual('not available', helper.check_availability())
--- a/watcher/tests/decision_engine/datasources/test_gnocchi_helper.py
+++ b/watcher/tests/decision_engine/datasources/test_gnocchi_helper.py
@@ -40,17 +40,25 @@ class TestGnocchiHelper(base.BaseTestCase):
        self.addCleanup(stat_agg_patcher.stop)

    def test_gnocchi_statistic_aggregation(self, mock_gnocchi):
+        vcpus = 2
+        mock_instance = mock.Mock(
+            id='16a86790-327a-45f9-bc82-45839f062fdc',
+            vcpus=vcpus)
+
        gnocchi = mock.MagicMock()
+        # cpu time rate of change (ns)
+        mock_rate_measure = 360 * 10e+8 * vcpus * 5.5 / 100
        expected_result = 5.5

-        expected_measures = [["2017-02-02T09:00:00.000000", 360, 5.5]]
+        expected_measures = [
+            ["2017-02-02T09:00:00.000000", 360, mock_rate_measure]]

        gnocchi.metric.get_measures.return_value = expected_measures
        mock_gnocchi.return_value = gnocchi

        helper = gnocchi_helper.GnocchiHelper()
        result = helper.statistic_aggregation(
-            resource=mock.Mock(id='16a86790-327a-45f9-bc82-45839f062fdc'),
+            resource=mock_instance,
            resource_type='instance',
            meter_name='instance_cpu_usage',
            period=300,
@@ -59,6 +67,14 @@ class TestGnocchiHelper(base.BaseTestCase):
        )
        self.assertEqual(expected_result, result)

+        gnocchi.metric.get_measures.assert_called_once_with(
+            metric="cpu",
+            start=mock.ANY,
+            stop=mock.ANY,
+            resource_id=mock_instance.uuid,
+            granularity=360,
+            aggregation="rate:mean")
+
    def test_gnocchi_statistic_series(self, mock_gnocchi):
        gnocchi = mock.MagicMock()
        expected_result = {
--- a/watcher/tests/decision_engine/fake_metal_helper.py
+++ b/watcher/tests/decision_engine/fake_metal_helper.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2023 Cloudbase Solutions
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from unittest import mock
+import uuid
+
+from watcher.common.metal_helper import constants as m_constants
+
+
+def get_mock_metal_node(node_id=None,
+                        power_state=m_constants.PowerState.ON,
+                        running_vms=0,
+                        hostname=None,
+                        compute_state='up'):
+    node_id = node_id or str(uuid.uuid4())
+    # NOTE(lpetrut): the hostname is important for some of the tests,
+    # which expect it to match the fake cluster model.
+    hostname = hostname or "compute-" + str(uuid.uuid4()).split('-')[0]
+
+    hypervisor_node_dict = {
+        'hypervisor_hostname': hostname,
+        'running_vms': running_vms,
+        'service': {
+            'host': hostname,
+        },
+        'state': compute_state,
+    }
+    hypervisor_node = mock.Mock(**hypervisor_node_dict)
+    hypervisor_node.to_dict.return_value = hypervisor_node_dict
+
+    node = mock.Mock()
+    node.get_power_state.return_value = power_state
+    node.get_id.return_value = uuid
+    node.get_hypervisor_node.return_value = hypervisor_node
+    return node
--- a/watcher/tests/decision_engine/model/faker_cluster_and_metrics.py
+++ b/watcher/tests/decision_engine/model/faker_cluster_and_metrics.py
@@ -80,7 +80,7 @@ class FakerModelCollector(base.BaseClusterDataModelCollector):
        return self.load_model('scenario_4_with_metrics.xml')


-class FakeCeilometerMetrics(object):
+class FakeGnocchiMetrics(object):
    def __init__(self, model):
        self.model = model

@@ -90,6 +90,9 @@ class FakeCeilometerMetrics(object):
        if meter_name == 'host_cpu_usage':
            return self.get_compute_node_cpu_util(
                resource, period, aggregate, granularity)
+        elif meter_name == 'host_ram_usage':
+            return self.get_compute_node_ram_util(
+                resource, period, aggregate, granularity)
        elif meter_name == 'instance_cpu_usage':
            return self.get_instance_cpu_util(
                resource, period, aggregate, granularity)
@@ -110,109 +113,27 @@ class FakeCeilometerMetrics(object):
        Returns relative node CPU utilization <0, 100>.
        :param r_id: resource id
        """
-        node_uuid = '%s_%s' % (resource.uuid, resource.hostname)
-        node = self.model.get_node_by_uuid(node_uuid)
+        node = self.model.get_node_by_uuid(resource.uuid)
        instances = self.model.get_node_instances(node)
        util_sum = 0.0
-        for instance_uuid in instances:
-            instance = self.model.get_instance_by_uuid(instance_uuid)
+        for instance in instances:
            total_cpu_util = instance.vcpus * self.get_instance_cpu_util(
-                instance.uuid)
+                instance, period, aggregate, granularity)
            util_sum += total_cpu_util / 100.0
        util_sum /= node.vcpus
        return util_sum * 100.0

-    @staticmethod
-    def get_instance_cpu_util(resource, period, aggregate,
-                              granularity):
-        instance_cpu_util = dict()
-        instance_cpu_util['INSTANCE_0'] = 10
-        instance_cpu_util['INSTANCE_1'] = 30
-        instance_cpu_util['INSTANCE_2'] = 60
-        instance_cpu_util['INSTANCE_3'] = 20
-        instance_cpu_util['INSTANCE_4'] = 40
-        instance_cpu_util['INSTANCE_5'] = 50
-        instance_cpu_util['INSTANCE_6'] = 100
-        instance_cpu_util['INSTANCE_7'] = 100
-        instance_cpu_util['INSTANCE_8'] = 100
-        instance_cpu_util['INSTANCE_9'] = 100
-        return instance_cpu_util[str(resource.uuid)]
-
-    @staticmethod
-    def get_instance_ram_util(resource, period, aggregate,
-                              granularity):
-        instance_ram_util = dict()
-        instance_ram_util['INSTANCE_0'] = 1
-        instance_ram_util['INSTANCE_1'] = 2
-        instance_ram_util['INSTANCE_2'] = 4
-        instance_ram_util['INSTANCE_3'] = 8
-        instance_ram_util['INSTANCE_4'] = 3
-        instance_ram_util['INSTANCE_5'] = 2
-        instance_ram_util['INSTANCE_6'] = 1
-        instance_ram_util['INSTANCE_7'] = 2
-        instance_ram_util['INSTANCE_8'] = 4
-        instance_ram_util['INSTANCE_9'] = 8
-        return instance_ram_util[str(resource.uuid)]
-
-    @staticmethod
-    def get_instance_disk_root_size(resource, period, aggregate,
-                                    granularity):
-        instance_disk_util = dict()
-        instance_disk_util['INSTANCE_0'] = 10
-        instance_disk_util['INSTANCE_1'] = 15
-        instance_disk_util['INSTANCE_2'] = 30
-        instance_disk_util['INSTANCE_3'] = 35
-        instance_disk_util['INSTANCE_4'] = 20
-        instance_disk_util['INSTANCE_5'] = 25
-        instance_disk_util['INSTANCE_6'] = 25
-        instance_disk_util['INSTANCE_7'] = 25
-        instance_disk_util['INSTANCE_8'] = 25
-        instance_disk_util['INSTANCE_9'] = 25
-        return instance_disk_util[str(resource.uuid)]
-
-
-class FakeGnocchiMetrics(object):
-    def __init__(self, model):
-        self.model = model
-
-    def mock_get_statistics(self, resource=None, resource_type=None,
-                            meter_name=None, period=300, aggregate='mean',
-                            granularity=300):
-        if meter_name == 'host_cpu_usage':
-            return self.get_compute_node_cpu_util(
-                resource, period, aggregate, granularity)
-        elif meter_name == 'instance_cpu_usage':
-            return self.get_instance_cpu_util(
-                resource, period, aggregate, granularity)
-        elif meter_name == 'instance_ram_usage':
-            return self.get_instance_ram_util(
-                resource, period, aggregate, granularity)
-        elif meter_name == 'instance_root_disk_size':
-            return self.get_instance_disk_root_size(
-                resource, period, aggregate, granularity)
-
-    def get_compute_node_cpu_util(self, resource, period, aggregate,
+    def get_compute_node_ram_util(self, resource, period, aggregate,
                                  granularity):
-        """Calculates node utilization dynamicaly.
-
-        node CPU utilization should consider
-        and corelate with actual instance-node mappings
-        provided within a cluster model.
-        Returns relative node CPU utilization <0, 100>.
-
-        :param r_id: resource id
-        """
-        node_uuid = "%s_%s" % (resource.uuid, resource.hostname)
-        node = self.model.get_node_by_uuid(node_uuid)
+        # Returns mock host ram usage in KB based on the allocated
+        # instances.
+        node = self.model.get_node_by_uuid(resource.uuid)
        instances = self.model.get_node_instances(node)
        util_sum = 0.0
-        for instance_uuid in instances:
-            instance = self.model.get_instance_by_uuid(instance_uuid)
-            total_cpu_util = instance.vcpus * self.get_instance_cpu_util(
-                instance.uuid)
-            util_sum += total_cpu_util / 100.0
-        util_sum /= node.vcpus
-        return util_sum * 100.0
+        for instance in instances:
+            util_sum += self.get_instance_ram_util(
+                instance, period, aggregate, granularity)
+        return util_sum / 1024

    @staticmethod
    def get_instance_cpu_util(resource, period, aggregate,
@@ -261,3 +182,9 @@ class FakeGnocchiMetrics(object):
        instance_disk_util['INSTANCE_8'] = 25
        instance_disk_util['INSTANCE_9'] = 25
        return instance_disk_util[str(resource.uuid)]
+
+
+# TODO(lpetrut): consider dropping Ceilometer support, it was deprecated
+# in Ocata.
+class FakeCeilometerMetrics(FakeGnocchiMetrics):
+    pass
--- a/watcher/tests/decision_engine/strategy/strategies/test_saving_energy.py
+++ b/watcher/tests/decision_engine/strategy/strategies/test_saving_energy.py
@@ -18,8 +18,10 @@
 from unittest import mock

 from watcher.common import clients
+from watcher.common.metal_helper import constants as m_constants
 from watcher.common import utils
 from watcher.decision_engine.strategy import strategies
+from watcher.tests.decision_engine import fake_metal_helper
 from watcher.tests.decision_engine.strategy.strategies.test_base \
    import TestBaseStrategy

@@ -29,26 +31,15 @@ class TestSavingEnergy(TestBaseStrategy):
    def setUp(self):
        super(TestSavingEnergy, self).setUp()

-        mock_node1_dict = {
-            'uuid': '922d4762-0bc5-4b30-9cb9-48ab644dd861'}
-        mock_node2_dict = {
-            'uuid': '922d4762-0bc5-4b30-9cb9-48ab644dd862'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.fake_nodes = [mock_node1, mock_node2]
+        self.fake_nodes = [fake_metal_helper.get_mock_metal_node(),
+                           fake_metal_helper.get_mock_metal_node()]
+        self._metal_helper = mock.Mock()
+        self._metal_helper.list_compute_nodes.return_value = self.fake_nodes

-        p_ironic = mock.patch.object(
-            clients.OpenStackClients, 'ironic')
-        self.m_ironic = p_ironic.start()
-        self.addCleanup(p_ironic.stop)
-
-        p_nova = mock.patch.object(
-            clients.OpenStackClients, 'nova')
+        p_nova = mock.patch.object(clients.OpenStackClients, 'nova')
        self.m_nova = p_nova.start()
        self.addCleanup(p_nova.stop)

-        self.m_ironic.node.list.return_value = self.fake_nodes
-
        self.m_c_model.return_value = self.fake_c_cluster.generate_scenario_1()

        self.strategy = strategies.SavingEnergy(
@@ -59,27 +50,20 @@ class TestSavingEnergy(TestBaseStrategy):
             'min_free_hosts_num': 1})
        self.strategy.free_used_percent = 10.0
        self.strategy.min_free_hosts_num = 1
-        self.strategy._ironic_client = self.m_ironic
+        self.strategy._metal_helper = self._metal_helper
        self.strategy._nova_client = self.m_nova

    def test_get_hosts_pool_with_vms_node_pool(self):
-        mock_node1_dict = {
-            'extra': {'compute_node_id': 1},
-            'power_state': 'power on'}
-        mock_node2_dict = {
-            'extra': {'compute_node_id': 2},
-            'power_state': 'power off'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.m_ironic.node.get.side_effect = [mock_node1, mock_node2]
-
-        mock_hyper1 = mock.Mock()
-        mock_hyper2 = mock.Mock()
-        mock_hyper1.to_dict.return_value = {
-            'running_vms': 2, 'service': {'host': 'hostname_0'}, 'state': 'up'}
-        mock_hyper2.to_dict.return_value = {
-            'running_vms': 2, 'service': {'host': 'hostname_1'}, 'state': 'up'}
-        self.m_nova.hypervisors.get.side_effect = [mock_hyper1, mock_hyper2]
+        self._metal_helper.list_compute_nodes.return_value = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON,
+                hostname='hostname_0',
+                running_vms=2),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF,
+                hostname='hostname_1',
+                running_vms=2),
+        ]

        self.strategy.get_hosts_pool()

@@ -88,23 +72,16 @@ class TestSavingEnergy(TestBaseStrategy):
        self.assertEqual(len(self.strategy.free_poweroff_node_pool), 0)

    def test_get_hosts_pool_free_poweron_node_pool(self):
-        mock_node1_dict = {
-            'extra': {'compute_node_id': 1},
-            'power_state': 'power on'}
-        mock_node2_dict = {
-            'extra': {'compute_node_id': 2},
-            'power_state': 'power on'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.m_ironic.node.get.side_effect = [mock_node1, mock_node2]
-
-        mock_hyper1 = mock.Mock()
-        mock_hyper2 = mock.Mock()
-        mock_hyper1.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_0'}, 'state': 'up'}
-        mock_hyper2.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_1'}, 'state': 'up'}
-        self.m_nova.hypervisors.get.side_effect = [mock_hyper1, mock_hyper2]
+        self._metal_helper.list_compute_nodes.return_value = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON,
+                hostname='hostname_0',
+                running_vms=0),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON,
+                hostname='hostname_1',
+                running_vms=0),
+        ]

        self.strategy.get_hosts_pool()

@@ -113,23 +90,16 @@ class TestSavingEnergy(TestBaseStrategy):
        self.assertEqual(len(self.strategy.free_poweroff_node_pool), 0)

    def test_get_hosts_pool_free_poweroff_node_pool(self):
-        mock_node1_dict = {
-            'extra': {'compute_node_id': 1},
-            'power_state': 'power off'}
-        mock_node2_dict = {
-            'extra': {'compute_node_id': 2},
-            'power_state': 'power off'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.m_ironic.node.get.side_effect = [mock_node1, mock_node2]
-
-        mock_hyper1 = mock.Mock()
-        mock_hyper2 = mock.Mock()
-        mock_hyper1.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_0'}, 'state': 'up'}
-        mock_hyper2.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_1'}, 'state': 'up'}
-        self.m_nova.hypervisors.get.side_effect = [mock_hyper1, mock_hyper2]
+        self._metal_helper.list_compute_nodes.return_value = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF,
+                hostname='hostname_0',
+                running_vms=0),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF,
+                hostname='hostname_1',
+                running_vms=0),
+        ]

        self.strategy.get_hosts_pool()

@@ -138,26 +108,16 @@ class TestSavingEnergy(TestBaseStrategy):
        self.assertEqual(len(self.strategy.free_poweroff_node_pool), 2)

    def test_get_hosts_pool_with_node_out_model(self):
-        mock_node1_dict = {
-            'extra': {'compute_node_id': 1},
-            'power_state': 'power off'}
-        mock_node2_dict = {
-            'extra': {'compute_node_id': 2},
-            'power_state': 'power off'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.m_ironic.node.get.side_effect = [mock_node1, mock_node2]
-
-        mock_hyper1 = mock.Mock()
-        mock_hyper2 = mock.Mock()
-        mock_hyper1.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_0'},
-            'state': 'up'}
-        mock_hyper2.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_10'},
-            'state': 'up'}
-        self.m_nova.hypervisors.get.side_effect = [mock_hyper1, mock_hyper2]
-
+        self._metal_helper.list_compute_nodes.return_value = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF,
+                hostname='hostname_0',
+                running_vms=0),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.OFF,
+                hostname='hostname_10',
+                running_vms=0),
+        ]
        self.strategy.get_hosts_pool()

        self.assertEqual(len(self.strategy.with_vms_node_pool), 0)
@@ -166,9 +126,9 @@ class TestSavingEnergy(TestBaseStrategy):

    def test_save_energy_poweron(self):
        self.strategy.free_poweroff_node_pool = [
-            mock.Mock(uuid='922d4762-0bc5-4b30-9cb9-48ab644dd861'),
-            mock.Mock(uuid='922d4762-0bc5-4b30-9cb9-48ab644dd862')
-            ]
+            fake_metal_helper.get_mock_metal_node(),
+            fake_metal_helper.get_mock_metal_node(),
+        ]
        self.strategy.save_energy()
        self.assertEqual(len(self.strategy.solution.actions), 1)
        action = self.strategy.solution.actions[0]
@@ -185,23 +145,16 @@ class TestSavingEnergy(TestBaseStrategy):
        self.assertEqual(action.get('input_parameters').get('state'), 'off')

    def test_execute(self):
-        mock_node1_dict = {
-            'extra': {'compute_node_id': 1},
-            'power_state': 'power on'}
-        mock_node2_dict = {
-            'extra': {'compute_node_id': 2},
-            'power_state': 'power on'}
-        mock_node1 = mock.Mock(**mock_node1_dict)
-        mock_node2 = mock.Mock(**mock_node2_dict)
-        self.m_ironic.node.get.side_effect = [mock_node1, mock_node2]
-
-        mock_hyper1 = mock.Mock()
-        mock_hyper2 = mock.Mock()
-        mock_hyper1.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_0'}, 'state': 'up'}
-        mock_hyper2.to_dict.return_value = {
-            'running_vms': 0, 'service': {'host': 'hostname_1'}, 'state': 'up'}
-        self.m_nova.hypervisors.get.side_effect = [mock_hyper1, mock_hyper2]
+        self._metal_helper.list_compute_nodes.return_value = [
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON,
+                hostname='hostname_0',
+                running_vms=0),
+            fake_metal_helper.get_mock_metal_node(
+                power_state=m_constants.PowerState.ON,
+                hostname='hostname_1',
+                running_vms=0),
+        ]

        model = self.fake_c_cluster.generate_scenario_1()
        self.m_c_model.return_value = model
--- a/watcher/tests/decision_engine/strategy/strategies/test_vm_workload_consolidation.py
+++ b/watcher/tests/decision_engine/strategy/strategies/test_vm_workload_consolidation.py
@@ -64,6 +64,10 @@ class TestVMWorkloadConsolidation(TestBaseStrategy):
                self.fake_metrics.get_instance_ram_util),
            get_instance_root_disk_size=(
                self.fake_metrics.get_instance_disk_root_size),
+            get_host_cpu_usage=(
+                self.fake_metrics.get_compute_node_cpu_util),
+            get_host_ram_usage=(
+                self.fake_metrics.get_compute_node_ram_util)
            )
        self.strategy = strategies.VMWorkloadConsolidation(
            config=mock.Mock(datasources=self.datasource))
@@ -88,6 +92,71 @@ class TestVMWorkloadConsolidation(TestBaseStrategy):
            node_util,
            self.strategy.get_node_utilization(node_0))

+    def test_get_node_utilization_using_host_metrics(self):
+        model = self.fake_c_cluster.generate_scenario_1()
+        self.m_c_model.return_value = model
+        self.fake_metrics.model = model
+        node_0 = model.get_node_by_uuid("Node_0")
+
+        # "get_node_utilization" is expected to return the maximum
+        # between the host metrics and the sum of the instance metrics.
+        data_src = self.m_datasource.return_value
+        cpu_usage = 30
+        data_src.get_host_cpu_usage = mock.Mock(return_value=cpu_usage)
+        data_src.get_host_ram_usage = mock.Mock(return_value=512 * 1024)
+
+        exp_cpu_usage = cpu_usage * node_0.vcpus / 100
+        exp_node_util = dict(cpu=exp_cpu_usage, ram=512, disk=10)
+        self.assertEqual(
+            exp_node_util,
+            self.strategy.get_node_utilization(node_0))
+
+    def test_get_node_utilization_after_migrations(self):
+        model = self.fake_c_cluster.generate_scenario_1()
+        self.m_c_model.return_value = model
+        self.fake_metrics.model = model
+        node_0 = model.get_node_by_uuid("Node_0")
+        node_1 = model.get_node_by_uuid("Node_1")
+
+        data_src = self.m_datasource.return_value
+        cpu_usage = 30
+        host_ram_usage_mb = 512
+        data_src.get_host_cpu_usage = mock.Mock(return_value=cpu_usage)
+        data_src.get_host_ram_usage = mock.Mock(
+            return_value=host_ram_usage_mb * 1024)
+
+        instance_uuid = 'INSTANCE_0'
+        instance = model.get_instance_by_uuid(instance_uuid)
+        self.strategy.add_migration(instance, node_0, node_1)
+
+        instance_util = self.strategy.get_instance_utilization(instance)
+
+        # Ensure that we take into account planned migrations when
+        # determining node utilization
+        exp_node_0_cpu_usage = (
+            cpu_usage * node_0.vcpus) / 100 - instance_util['cpu']
+        exp_node_1_cpu_usage = (
+            cpu_usage * node_1.vcpus) / 100 + instance_util['cpu']
+
+        exp_node_0_ram_usage = host_ram_usage_mb - instance.memory
+        exp_node_1_ram_usage = host_ram_usage_mb + instance.memory
+
+        exp_node_0_util = dict(
+            cpu=exp_node_0_cpu_usage,
+            ram=exp_node_0_ram_usage,
+            disk=0)
+        exp_node_1_util = dict(
+            cpu=exp_node_1_cpu_usage,
+            ram=exp_node_1_ram_usage,
+            disk=25)
+
+        self.assertEqual(
+            exp_node_0_util,
+            self.strategy.get_node_utilization(node_0))
+        self.assertEqual(
+            exp_node_1_util,
+            self.strategy.get_node_utilization(node_1))
+
    def test_get_node_capacity(self):
        model = self.fake_c_cluster.generate_scenario_1()
        self.m_c_model.return_value = model
@@ -113,7 +182,8 @@ class TestVMWorkloadConsolidation(TestBaseStrategy):
        expected_cru = {'cpu': 0.05, 'disk': 0.05, 'ram': 0.0234375}
        self.assertEqual(expected_cru, cru)

-    def test_add_migration_with_active_state(self):
+    def _test_add_migration(self, instance_state, expect_migration=True,
+                            expected_migration_type="live"):
        model = self.fake_c_cluster.generate_scenario_1()
        self.m_c_model.return_value = model
        self.fake_metrics.model = model
@@ -121,38 +191,36 @@ class TestVMWorkloadConsolidation(TestBaseStrategy):
        n2 = model.get_node_by_uuid('Node_1')
        instance_uuid = 'INSTANCE_0'
        instance = model.get_instance_by_uuid(instance_uuid)
+        instance.state = instance_state
        self.strategy.add_migration(instance, n1, n2)
-        self.assertEqual(1, len(self.strategy.solution.actions))
-        expected = {'action_type': 'migrate',
-                    'input_parameters': {'destination_node': n2.hostname,
-                                         'source_node': n1.hostname,
-                                         'migration_type': 'live',
-                                         'resource_id': instance.uuid,
-                                         'resource_name': instance.name}}
-        self.assertEqual(expected, self.strategy.solution.actions[0])
+
+        if expect_migration:
+            self.assertEqual(1, len(self.strategy.solution.actions))
+
+            expected = {'action_type': 'migrate',
+                        'input_parameters': {
+                            'destination_node': n2.hostname,
+                            'source_node': n1.hostname,
+                            'migration_type': expected_migration_type,
+                            'resource_id': instance.uuid,
+                            'resource_name': instance.name}}
+            self.assertEqual(expected, self.strategy.solution.actions[0])
+        else:
+            self.assertEqual(0, len(self.strategy.solution.actions))
+
+    def test_add_migration_with_active_state(self):
+        self._test_add_migration(element.InstanceState.ACTIVE.value)

    def test_add_migration_with_paused_state(self):
-        model = self.fake_c_cluster.generate_scenario_1()
-        self.m_c_model.return_value = model
-        self.fake_metrics.model = model
-        n1 = model.get_node_by_uuid('Node_0')
-        n2 = model.get_node_by_uuid('Node_1')
-        instance_uuid = 'INSTANCE_0'
-        instance = model.get_instance_by_uuid(instance_uuid)
-        setattr(instance, 'state', element.InstanceState.ERROR.value)
-        self.strategy.add_migration(instance, n1, n2)
-        self.assertEqual(0, len(self.strategy.solution.actions))
+        self._test_add_migration(element.InstanceState.PAUSED.value)

-        setattr(instance, 'state', element.InstanceState.PAUSED.value)
-        self.strategy.add_migration(instance, n1, n2)
-        self.assertEqual(1, len(self.strategy.solution.actions))
-        expected = {'action_type': 'migrate',
-                    'input_parameters': {'destination_node': n2.hostname,
-                                         'source_node': n1.hostname,
-                                         'migration_type': 'live',
-                                         'resource_id': instance.uuid,
-                                         'resource_name': instance.name}}
-        self.assertEqual(expected, self.strategy.solution.actions[0])
+    def test_add_migration_with_error_state(self):
+        self._test_add_migration(element.InstanceState.ERROR.value,
+                                 expect_migration=False)
+
+    def test_add_migration_with_stopped_state(self):
+        self._test_add_migration(element.InstanceState.STOPPED.value,
+                                 expected_migration_type="cold")

    def test_is_overloaded(self):
        model = self.fake_c_cluster.generate_scenario_1()
--- a/watcher/tests/objects/test_objects.py
+++ b/watcher/tests/objects/test_objects.py
@@ -531,6 +531,7 @@ class TestRegistry(test_base.TestCase):

    @mock.patch('watcher.objects.base.objects')
    def test_hook_chooses_newer_properly(self, mock_objects):
+        mock_objects.MyObj.VERSION = MyObj.VERSION
        reg = base.WatcherObjectRegistry()
        reg.registration_hook(MyObj, 0)

@@ -547,6 +548,7 @@ class TestRegistry(test_base.TestCase):

    @mock.patch('watcher.objects.base.objects')
    def test_hook_keeps_newer_properly(self, mock_objects):
+        mock_objects.MyObj.VERSION = MyObj.VERSION
        reg = base.WatcherObjectRegistry()
        reg.registration_hook(MyObj, 0)
Author	SHA1	Message	Date
OpenStack Proposal Bot	a9dc3794a6	Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I2b2afb0c0e590b737871bf4c43293df2ed88e534	2024-06-01 02:47:52 +00:00
Takashi Kajinami	d6f169197e	SQLAlchemy 2.0: Omnibus fixes patch This was originally five patches, but they are all needed to pass any of the test jobs now, so they have been squashed into one: Co-Authored-By: Dan Smith (dms@danplanet.com) First: The autoload argument was removed[1] in SQLAlchemy and only the autoload_with argument should be passed. The autoload argument is set according to the autoload_with argument automatically even in SQLAlchemy 1.x[2] so is not at all needed. [1] `c932123bac` [2] `ad8f921e96` Second: Remove _warn_on_bytestring for newer SA, AFAICT, this flag has been removed from SQLAlchemy and that is why watcher-db-manage fails to initialize the DB for me on jammy. This migration was passing the default value (=False) anyway, so I assume this is the right "fix". Third: Fix joinedload passing string attribute names Fourth: Fix engine.select pattern to use begin() per the migration guide. Fifth: Override the apscheduler get_next_run_time() which appears to be trivially not compatible with SQLAlchemy 2.0 because of a return type from scalar(). Change-Id: I000e5e78f97f82ed4ea64d42f1c38354c3252e08	2024-05-29 06:49:32 -07:00
James Page	bc5922c684	Fix oslo.db >= 15.0.0 compatibility Minimal refactor of SQLAlchemy api module to be compatible with oslo.db >= 15.0.0 where autocommit behaviour was dropped. Closes-Bug: #2056181 Change-Id: I33be53f647faae2aad30a43c10980df950d5d7c2	2024-03-27 09:41:23 +00:00
Ghanshyam Mann	9d58a6d457	Update python classifier in setup.cfg As per the current release tested runtime, we test python version from 3.8 to 3.11 so updating the same in python classifier in setup.cfg Change-Id: Ie010eea38eb0861699b60f16dfd3e2e95ae33709	2024-01-09 19:22:04 -08:00
Lucian Petrut	c95ce4ec17	Add MAAS support At the moment, Watcher can use a single bare metal provisioning service: Openstack Ironic. We're now adding support for Canonical's MAAS service [1], which is commonly used along with Juju [2] to deploy Openstack. In order to do so, we're building a metal client abstraction, with concrete implementations for Ironic and MAAS. We'll pick the MAAS client if the MAAS url is provided, otherwise defaulting to Ironic. For now, we aren't updating the baremetal model collector since it doesn't seem to be used by any of the existing Watcher strategy implementations. [1] https://maas.io/docs [2] https://juju.is/docs Implements: blueprint maas-support Change-Id: I6861995598f6c542fa9c006131f10203f358e0a6	2023-12-11 10:21:33 +00:00
Zuul	9492c2190e	Merge "vm workload consolidation: use actual host metrics"	2023-12-01 01:51:39 +00:00
Lucian Petrut	808f1bcee3	Update action json schema Power-off actions created by the energy saving strategy include a resource name property, which currently isn't part of the action json schema. For this reason, json schema validation fails. Additional properties are not allowed ('resource_name' was unexpected) We'll update the json schema, including the resource name property. Change-Id: I924d36732a917c0be98b08c2f4128e9136356215	2023-11-15 01:11:56 +00:00
Lucian Petrut	3b224b5629	Fix object tests A couple of object tests are failing, probably after a dependency bump. watcher.objects.base.objects is mocked, so the registered object version isn't properly retrieved, leading to a type error: File "/mnt/data/workspace/watcher/watcher/tests/objects/test_objects.py", line 535, in test_hook_chooses_newer_properly reg.registration_hook(MyObj, 0) File "/mnt/data/workspace/watcher/watcher/objects/base.py", line 46, in registration_hook cur_version = versionutils.convert_version_to_tuple( File "/home/ubuntu/openstack_venv/lib/python3.10/site-packages/oslo_utils/versionutils.py", line 91, in convert_version_to_tuple version_str = re.sub(r'(\d+)(a\|alpha\|b\|beta\|rc)\d+$', '\\1', version_str) File "/usr/lib/python3.10/re.py", line 209, in sub return _compile(pattern, flags).sub(repl, string, count) TypeError: expected string or bytes-like object We'll solve the issue by setting the VERSION attribute against the mock object. Change-Id: Ifeb38b98f1d702908531de5fc5c846bd1c53de4b	2023-11-14 10:38:40 +00:00
Lucian Petrut	424e9a76af	vm workload consolidation: use actual host metrics The "vm workload consolidation" strategy is summing up instance usage in order to estimate host usage. The problem is that some infrastructure services (e.g. OVS or Ceph clients) may also use a significant amount of resources, which would be ignored. This can impact Watcher's ability to detect overloaded nodes and correctly rebalance the workload. This commit will use the host metrics, if available. The proposed implementation uses the maximum value between the host metric and the sum of the instance metrics. Note that we're holding a dict of host metric deltas in order to account for planned migrations. Change-Id: I82f474ee613f6c9a7c0a9d24a05cba41d2f68edb	2023-10-27 21:54:42 +03:00
Zuul	40e93407c7	Merge "Handle deprecated "cpu_util" metric"	2023-10-27 09:47:38 +00:00
Zuul	721aec1cb6	Merge "vm workload consolidation: allow cold migrations"	2023-10-27 09:47:36 +00:00
Zuul	8a3ee8f931	Merge "Improve vm_consolidation logging"	2023-10-27 09:20:13 +00:00
Lucian Petrut	00fea975e2	Handle deprecated "cpu_util" metric The "cpu_util" metric has been deprecated a few years ago. We'll obtain the same result by converting the cumulative cpu time to a percentage, leveraging the rate of change aggregation. Change-Id: I18fe0de6f74c785e674faceea0c48f44055818fe	2023-10-24 10:47:23 +00:00
Lucian Petrut	fd6562382e	Avoid performing retries in case of missing resources There may be no available metrics for instances that are stopped or were recently spawned. This makes retries unnecessary and time consuming. For this reason, we'll ignore gnocchi MetricNotFound errors. Change-Id: I79cd03bf04db634b931d6dfd32d5150f58e82044	2023-10-23 14:14:21 +00:00
Lucian Petrut	ec90891636	Improve vm_consolidation logging We're adding a few info log messages in order to trace the "vm consolidation" strategy more easily. Change-Id: I8ce1a9dd173733f1b801839d3ad0c1269c4306bb	2023-10-23 14:10:02 +00:00
Lucian Petrut	7336a48057	vm workload consolidation: allow cold migrations Although Watcher supports cold migrations, the vm workload consolidation workflow only allows live migrations to be performed. We'll remove this unnecessary limitation so that stopped instances could be cold migrated. Change-Id: I4b41550f2255560febf8586722a0e02045c3a486	2023-10-23 13:03:18 +00:00
Lucian Petrut	922478fbda	Unblock the CI gate The Nova collector json schema validation started [1][2] failing after the jsonschema upper constraint was bumped from 4.17.3 to 4.19.1 [3]. The reason is that jsonschema v4.18.0a1 switched to a reference resolving library [4], which treats the aggregate "id" as a jsonschema id and expects it to be a string [5]. For this reason, we're now getting AttributeError exceptions. As a workaround, we'll rename the "id" ref element as "host_aggr_id". Also, the watcher-tempest-multinode job is configured to use Focal, which is no longer supported by Devstack [6]. That being considered, we'll switch to Ubuntu Jammy (22.04). While at it, we're disabling Cinder Backup, which isn't used while testing Watched. It currently causes Devstack failures since it uses the Swift backend by default, which is disabled. [1] https://paste.opendev.org/raw/bjQ1uIdbDMnmA1UEhxLL/ [2] https://paste.opendev.org/raw/bNgxqulBwBLYB7tNhrU4/ [3] `ab0dcbdda2` [4] https://github.com/python-jsonschema/jsonschema/releases/tag/v4.18.0a1 [5] `c23a5dc1c9/referencing/jsonschema.py (L54-L55C18)` [6] https://paste.openstack.org/raw/bSoSyXgbtmq6d9768HQn/ Change-Id: I300620c2ec4857b1e0d402a9b57a637f576eeb24	2023-10-23 09:21:55 +03:00
OpenStack Release Bot	9f0eca2343	Update master for stable/2023.2 Add file to the reno documentation build to show release notes for stable/2023.2. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/2023.2. Sem-Ver: feature Change-Id: I8a0c75ce5a4e5ae5cccd8eb1cb0325747a619122	2023-09-14 01:24:43 +00:00
Zuul	1e11c490a7	Merge "Add timeout option for Grafana request"	2023-08-29 11:21:46 +00:00
Zuul	8a7a8db661	Merge "Imported Translations from Zanata"	2023-08-28 06:21:40 +00:00
BubaVV	0610070e59	Add timeout option for Grafana request Implemented config option to setup Grafana API request timeout Change-Id: I8cbf8ce22f199fe22c0b162ba1f419169881f193	2023-08-23 17:46:19 +03:00
OpenStack Proposal Bot	a0997a0423	Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I37201577bd8d9c53db8ce6700f47d911359da6d2	2023-08-14 04:24:29 +00:00
chenker	4ea3eada3e	Fix watcher comment Change-Id: I4512cf1032e08934886d5e3ca858b3e05c3da76c	2023-08-13 00:00:12 +00:00
Zuul	cd1c0f3054	Merge "Imported Translations from Zanata"	2023-03-08 07:04:33 +00:00
OpenStack Proposal Bot	684350977d	Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I4ee251e6d37a1b955c22dc6fdc04c1a08c9ae9b8	2023-03-02 03:28:31 +00:00
OpenStack Release Bot	d28630b759	Update master for stable/2023.1 Add file to the reno documentation build to show release notes for stable/2023.1. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/2023.1. Sem-Ver: feature Change-Id: Ia585893e7fef42e9991a2b81f604d1ff28c0a5ad	2023-02-28 13:31:08 +00:00
Zuul	f7fbaf46a2	Merge "Use new get_rpc_client API from oslo.messaging"	2023-02-09 01:25:15 +00:00
Zuul	e7cda537e7	Merge "Modify saving_energy log info"	2023-02-07 12:18:58 +00:00
chenker	c7be34fbaa	update saving_energy docs Change-Id: I3b0c86911a8d32912c2de2e2392af9539b8d9be0	2023-02-07 10:27:54 +00:00
chenker	52da088011	Modify saving_energy log info Change-Id: I84879a453aa3ff78917d1136c62978b9d0e606de	2023-02-07 10:20:04 +00:00
Tobias Urdin	6ac3a6febf	Fix passenv in tox.ini Change-Id: If1ddb1d48eeb96191bcbfadd1a5e14f4350a02e4	2023-02-07 08:02:20 +00:00
Tobias Urdin	e36b77ad6d	Use new get_rpc_client API from oslo.messaging Use the new API that is consistent with the existing API instead of instantiating the client class directly. This was introduced in release 14.1.0 here [1] and added into oslo.messaging here [2] [1] https://review.opendev.org/c/openstack/requirements/+/869340 [2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419 Change-Id: I43c399a0c68473e40b8b71e9617c8334a439e675	2023-01-19 20:50:26 +00:00
Thierry Carrez	6003322711	Move queue declaration to project level This moves the watcher queue declaration from the pipeline level (where it is no longer valid) to the project level. https: //lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html Change-Id: I06923abb00f7eecd59587f44cd1f6a069e88a9fc	2022-09-26 14:19:58 +02:00
Zuul	f4ffca01b8	Merge "Switch to 2023.1 Python3 unit tests and generic template name"	2022-09-16 06:36:21 +00:00
Alfredo Moralejo	5d70c207cd	Fix compatibility with oslo.db 12.1.0 oslo.db 12.1.0 has changed the default value for the 'autocommit' parameter of 'LegacyEngineFacade' from 'True' to 'False'. This is a necessary step to ensure compatibility with SQLAlchemy 2.0. However, we are currently relying on the autocommit behavior and need changes to explicitly manage sessions. Until that happens, we need to override the default. Co-Authored-By: Stephen Finucane <stephenfin@redhat.com> Change-Id: I7db39d958d087322bfa0aad70dfbd04de9228dd7	2022-09-15 16:52:41 +02:00
OpenStack Release Bot	0b2e641d00	Switch to 2023.1 Python3 unit tests and generic template name This is an automatically generated patch to ensure unit testing is in place for all the of the tested runtimes for antelope. Also, updating the template name to generic one. See also the PTI in governance [1]. [1]: https://governance.openstack.org/tc/reference/project-testing-interface.html Change-Id: Ide6c6c398f8e6cdd590c6620a752ad802a1f5cf8	2022-09-13 12:30:33 +00:00
OpenStack Release Bot	ff84b052a5	Update master for stable/zed Add file to the reno documentation build to show release notes for stable/zed. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/zed. Sem-Ver: feature Change-Id: I1726e33a14038712dbb9fd5e5c0cddf8ad872e69	2022-09-13 12:30:32 +00:00
Zuul	a43b040ebc	Merge "Imported Translations from Zanata"	2022-08-30 10:44:52 +00:00
Zuul	749fa2507a	Merge "Tests: fix requirements for unit tests"	2022-08-30 08:15:05 +00:00
OpenStack Proposal Bot	76d61362ee	Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I95133dece6fdaf931dfed64015806430ba8d04f0	2022-08-29 04:12:15 +00:00
wangjiaqi07	c55143bc21	remove unicode from code Change-Id: I747445d482a2fb40c2f39139c5fd2a0cb26c27bc	2022-08-19 14:17:10 +08:00
suzhengwei	7609df3370	Tests: fix requirements for unit tests Add WebTest to test-requirements which used to be imported as a transitive requirement via pecan, but the latest release of pecan dropped this dependency. So make this requirement explicit. Related-Bug: #1982110 Change-Id: I4852be23b489257aaa56d3fa22d27f72bcabf919	2022-07-28 16:14:13 +08:00
chenker	b57eac12cb	Watcher DB upgrde compatibility consideration for add_apscheduler_jobs Change-Id: I8896ff5731bb8c1bf88a5d7b926bd2a884100ea8	2022-04-28 02:21:06 +00:00
OpenStack Release Bot	ac6911d3c4	Add Python3 zed unit tests This is an automatically generated patch to ensure unit testing is in place for all the of the tested runtimes for zed. See also the PTI in governance [1]. [1]: https://governance.openstack.org/tc/reference/project-testing-interface.html Change-Id: I5cf874842550de18ff777b909fd28e2c32e6d530	2022-03-10 12:14:06 +00:00
OpenStack Release Bot	23c2010681	Update master for stable/yoga Add file to the reno documentation build to show release notes for stable/yoga. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/yoga. Sem-Ver: feature Change-Id: Ic7c275b38fef9afc29577f81fe92546bb94b2930	2022-03-10 12:14:04 +00:00
zhurong	01d74d0a87	Remove ceilometerclient dependecy Change-Id: Ifa0f2493aa8414a29dc2722b6636a33bc5808be6	2022-01-07 05:48:22 +00:00
OpenStack Release Bot	e4fab0ce7f	Add Python3 yoga unit tests This is an automatically generated patch to ensure unit testing is in place for all the of the tested runtimes for yoga. See also the PTI in governance [1]. [1]: https://governance.openstack.org/tc/reference/project-testing-interface.html Change-Id: I328b3ccb76153fa0dbb4d174dd976412be049200	2021-09-15 17:14:09 +00:00
OpenStack Release Bot	76ecaaeb3a	Update master for stable/xena Add file to the reno documentation build to show release notes for stable/xena. Use pbr instruction to increment the minor version number automatically so that master versions are higher than the versions on stable/xena. Sem-Ver: feature Change-Id: If1c02305a153575c6a550844b0c6f45b74ea5ef3	2021-09-15 17:14:07 +00:00