Cluster data model collector plugin documentation

In this changeset, I wrote down the documentation on how to implement
a cluster data model collector plugin for Watcher.

This documentation corresponds to part 1 of the associated
specification.

Change-Id: Iac72b933df95252163033cd559d13348075a9b16
Partially-Implements: blueprint cluster-model-objects-wrapper

doc/source/conf.py

@@ -19,12 +19,13 @@ from watcher import version as watcher_version
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     'oslo_config.sphinxconfiggen',
+    'oslosphinx',
     'sphinx.ext.autodoc',
     'sphinx.ext.viewcode',
     'sphinxcontrib.httpdomain',
     'sphinxcontrib.pecanwsme.rest',
+    'stevedore.sphinxext',
     'wsmeext.sphinxext',
-    'oslosphinx',
     'watcher.doc',
 ]
 

doc/source/dev/plugin/action-plugin.rst

@@ -80,12 +80,9 @@ Here is an example showing how you can write a plugin called ``DummyAction``:
             pass
 
 
-This implementation is the most basic one. So if you want to have more advanced
+This implementation is the most basic one. So in order to get a better
-examples, have a look at the implementation of the actions already provided
+understanding on how to implement a more advanced action, have a look at the
-by Watcher like.
+:py:class:`~watcher.applier.actions.migration.Migrate` class.
-To get a better understanding on how to implement a more advanced action,
-have a look at the :py:class:`~watcher.applier.actions.migration.Migrate`
-class.
 
 Input validation
 ----------------

doc/source/dev/plugin/base-setup.rst

@@ -88,10 +88,13 @@ Now that the project skeleton has been created, you can start the
 implementation of your plugin. As of now, you can implement the following
 plugins for Watcher:
 
+- A :ref:`goal plugin <implement_goal_plugin>`
 - A :ref:`strategy plugin <implement_strategy_plugin>`
-- A :ref:`planner plugin <implement_planner_plugin>`
 - An :ref:`action plugin <implement_action_plugin>`
-- A :ref:`workflow engine plugin <implement_workflow_engine_plugin>`
+- A :ref:`planner plugin <implement_planner_plugin>`
+- A workflow engine plugin
+- A :ref:`cluster data model collector plugin
+  <implement_cluster_data_model_collector_plugin>`
 
 If you want to learn more on how to implement them, you can refer to their
 dedicated documentation.

doc/source/dev/plugin/cdmc-plugin.rst

@@ -0,0 +1,150 @@
+..
+      Except where otherwise noted, this document is licensed under Creative
+      Commons Attribution 3.0 License.  You can view the license at:
+
+          https://creativecommons.org/licenses/by/3.0/
+
+.. _implement_cluster_data_model_collector_plugin:
+
+========================================
+Build a new cluster data model collector
+========================================
+
+Watcher Decision Engine has an external cluster data model plugin interface
+which gives anyone the ability to integrate an external cluster data model
+collector in order to extend the initial set of cluster data model collectors
+Watcher provides.
+
+This section gives some guidelines on how to implement and integrate custom
+cluster data model collectors within Watcher.
+
+
+Creating a new plugin
+=====================
+
+First of all, you have to extend the :class:`~.BaseClusterDataModelCollector`
+base class which defines the :py:meth:`~.BaseClusterDataModelCollector.execute`
+abstract method you will have to implement. This method is responsible for
+building an entire cluster data model.
+
+Here is an example showing how you can write a plugin called
+``DummyClusterDataModelCollector``:
+
+.. code-block:: python
+
+    # Filepath = <PROJECT_DIR>/thirdparty/dummy.py
+    # Import path = thirdparty.dummy
+
+    from watcher.decision_engine.model import model_root
+    from watcher.metrics_engine.cluster_model_collector import base
+
+
+    class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
+
+        def execute(self):
+            model = model_root.ModelRoot()
+            # Do something here...
+            return model
+
+This implementation is the most basic one. So in order to get a better
+understanding on how to implement a more advanced cluster data model collector,
+have a look at the :py:class:`~.NovaClusterDataModelCollector` class.
+
+Define configuration parameters
+===============================
+
+At this point, you have a fully functional cluster data model collector.
+However, in more complex implementation, you may want to define some
+configuration options so one can tune the cluster data model collector to its
+needs. To do so, you can implement the :py:meth:`~.Loadable.get_config_opts`
+class method as followed:
+
+.. code-block:: python
+
+    from oslo_config import cfg
+    from watcher.decision_engine.model import model_root
+    from watcher.metrics_engine.cluster_model_collector import base
+
+
+    class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
+
+        def execute(self):
+            model = model_root.ModelRoot()
+            # Do something here...
+            return model
+
+        def get_config_opts(self):
+            return [
+                cfg.StrOpt('test_opt', help="Demo Option.", default=0),
+                # Some more options ...
+            ]
+
+The configuration options defined within this class method will be included
+within the global ``watcher.conf`` configuration file under a section named by
+convention: ``{namespace}.{plugin_name}``. In our case, the ``watcher.conf``
+configuration would have to be modified as followed:
+
+.. code-block:: ini
+
+    [watcher_cluster_data_model_collectors.dummy]
+    # Option used for testing.
+    test_opt = test_value
+
+Then, the configuration options you define within this method will then be
+injected in each instantiated object via the  ``config`` parameter of the
+:py:meth:`~.BaseClusterDataModelCollector.__init__` method.
+
+
+Abstract Plugin Class
+=====================
+
+Here below is the abstract ``BaseClusterDataModelCollector`` class that every
+single cluster data model collector should implement:
+
+.. autoclass:: watcher.metrics_engine.cluster_model_collector.base.BaseClusterDataModelCollector
+    :members:
+    :special-members: __init__
+    :noindex:
+
+
+Register a new entry point
+==========================
+
+In order for the Watcher Decision Engine to load your new cluster data model
+collector, the latter must be registered as a named entry point under the
+``watcher_cluster_data_model_collectors`` entry point of your ``setup.py``
+file. If you are using pbr_, this entry point should be placed in your
+``setup.cfg`` file.
+
+The name you give to your entry point has to be unique.
+
+Here below is how to register ``DummyClusterDataModelCollector`` using pbr_:
+
+.. code-block:: ini
+
+    [entry_points]
+    watcher_cluster_data_model_collectors =
+        dummy = thirdparty.dummy:DummyClusterDataModelCollector
+
+.. _pbr: http://docs.openstack.org/developer/pbr/
+
+
+Using cluster data model collector plugins
+==========================================
+
+The Watcher Decision Engine service will automatically discover any installed
+plugins when it is restarted. If a Python package containing a custom plugin is
+installed within the same environment as Watcher, Watcher will automatically
+make that plugin available for use.
+
+At this point, you can use your new cluster data model plugin in your
+:ref:`strategy plugin <implement_strategy_plugin>` by using the
+:py:attr:`~.BaseStrategy.collector_manager` property as followed:
+
+.. code-block:: python
+
+    # [...]
+    dummy_collector = self.collector_manager.get_cluster_model_collector(
+        "dummy")  # "dummy" is the name of the entry point we declared earlier
+    dummy_model = collector.get_latest_cluster_data_model()
+    # Do some stuff with this model

doc/source/dev/plugins.rst

@@ -18,32 +18,43 @@ use the :ref:`Guru Meditation Reports <watcher_gmr>` to display them.
 Goals
 =====
 
-.. drivers-doc:: watcher_goals
+.. list-plugins:: watcher_goals
+   :detailed:
 
 .. _watcher_strategies:
 
 Strategies
 ==========
 
-.. drivers-doc:: watcher_strategies
+.. list-plugins:: watcher_strategies
+   :detailed:
 
 .. _watcher_actions:
 
 Actions
 =======
 
-.. drivers-doc:: watcher_actions
+.. list-plugins:: watcher_actions
+   :detailed:
 
 .. _watcher_workflow_engines:
 
 Workflow Engines
 ================
 
-.. drivers-doc:: watcher_workflow_engines
+.. list-plugins:: watcher_workflow_engines
+   :detailed:
 
 .. _watcher_planners:
 
 Planners
 ========
 
-.. drivers-doc:: watcher_planners
+.. list-plugins:: watcher_planners
+   :detailed:
+
+Cluster Data Model Collectors
+=============================
+
+.. list-plugins:: watcher_cluster_data_model_collectors
+   :detailed:

doc/source/index.rst

@@ -75,6 +75,7 @@ Plugins
   dev/plugin/base-setup
   dev/plugin/goal-plugin
   dev/plugin/strategy-plugin
+  dev/plugin/cdmc-plugin
   dev/plugin/action-plugin
   dev/plugin/planner-plugin
   dev/plugins

watcher/decision_engine/goal/goals.py

@@ -72,7 +72,7 @@ class Unclassified(base.Goal):
 
 
 class ServerConsolidation(base.Goal):
-    """ServerConsolidation
+    """Server Consolidation
 
     This goal is for efficient usage of compute server resources in order to
     reduce the total number of servers.
@@ -97,7 +97,7 @@ class ServerConsolidation(base.Goal):
 
 
 class ThermalOptimization(base.Goal):
-    """ThermalOptimization
+    """Thermal Optimization
 
     This goal is used to balance the temperature across different servers.
     """
@@ -121,7 +121,7 @@ class ThermalOptimization(base.Goal):
 
 
 class WorkloadBalancing(base.Goal):
-    """WorkloadBalancing
+    """Workload Balancing
 
     This goal is used to evenly distribute workloads across different servers.
     """
@@ -145,6 +145,10 @@ class WorkloadBalancing(base.Goal):
 
 
 class AirflowOptimization(base.Goal):
+    """Workload Balancing
+
+    This goal is used to optimize the air flow within a cloud infrastructure.
+    """
 
     @classmethod
     def get_name(cls):

watcher/decision_engine/model/mapping.py

@@ -30,7 +30,7 @@ class Mapping(object):
         self.lock = threading.Lock()
 
     def map(self, hypervisor, vm):
-        """Select the hypervisor where the instance are launched
+        """Select the hypervisor where the instance is launched
 
         :param hypervisor: the hypervisor
         :param vm: the virtual machine or instance

watcher/doc.py

@@ -22,7 +22,6 @@ import inspect
 from docutils import nodes
 from docutils.parsers import rst
 from docutils import statemachine
-from stevedore import extension
 
 from watcher.version import version_info
 
@@ -98,74 +97,6 @@ class WatcherTerm(BaseWatcherDirective):
         return node.children
 
 
-class DriversDoc(BaseWatcherDirective):
-    """Directive to import an RST formatted docstring into the Watcher doc
-
-    This directive imports the RST formatted docstring of every driver declared
-    within an entry point namespace provided as argument
-
-    **How to use it**
-
-    # inside your .py file
-    class DocumentedClassReferencedInEntrypoint(object):
-        '''My *.rst* docstring'''
-
-        def foo(self):
-            '''Foo docstring'''
-
-    # Inside your .rst file
-    .. drivers-doc:: entrypoint_namespace
-       :append_methods_doc: foo
-
-    This directive will then import the docstring and then interprete it.
-
-    Note that no section/sub-section can be imported via this directive as it
-    is a Sphinx restriction.
-    """
-
-    # You need to put an import path as an argument for this directive to work
-    required_arguments = 1
-    optional_arguments = 0
-    final_argument_whitespace = True
-    has_content = False
-
-    option_spec = dict(
-        # CSV formatted list of method names whose return values will be zipped
-        # together in the given order
-        append_methods_doc=lambda opts: [
-            opt.strip() for opt in opts.split(",") if opt.strip()],
-        # By default, we always start by adding the driver object docstring
-        exclude_driver_docstring=rst.directives.flag,
-    )
-
-    def run(self):
-        ext_manager = extension.ExtensionManager(namespace=self.arguments[0])
-        extensions = ext_manager.extensions
-        # Aggregates drivers based on their module name (i.e import path)
-        classes = [(ext.name, ext.plugin) for ext in extensions]
-
-        for name, cls in classes:
-            self.add_line(".. rubric:: %s" % name)
-            self.add_line("")
-
-            if "exclude_driver_docstring" not in self.options:
-                self.add_object_docstring(cls)
-                self.add_line("")
-
-            for method_name in self.options.get("append_methods_doc", []):
-                if hasattr(cls, method_name):
-                    method = getattr(cls, method_name)
-                    method_result = inspect.cleandoc(method)
-                    self.add_textblock(method_result())
-                    self.add_line("")
-
-        node = nodes.paragraph()
-        node.document = self.state.document
-        self.state.nested_parse(self.result, 0, node)
-        return node.children
-
-
 def setup(app):
-    app.add_directive('drivers-doc', DriversDoc)
     app.add_directive('watcher-term', WatcherTerm)
     return {'version': version_info.version_string()}

watcher/metrics_engine/cluster_model_collector/base.py

@@ -2,6 +2,7 @@
 # Copyright (c) 2015 b<>com
 #
 # Authors: Jean-Emile DARTOIS <jean-emile.dartois@b-com.com>
+#          Vincent FRANCOISE <vincent.francoise@b-com.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,93 +20,85 @@
 
 """
 A :ref:`Cluster Data Model <cluster_data_model_definition>` is a logical
-representation of the current state and topology of the
+representation of the current state and topology of the :ref:`Cluster
-:ref:`Cluster <cluster_definition>`
+<cluster_definition>` :ref:`Managed resources <managed_resource_definition>`.
-:ref:`Managed resources <managed_resource_definition>`.
 
-It is represented as a set of
+It is represented as a set of :ref:`Managed resources
-:ref:`Managed resources <managed_resource_definition>`
+<managed_resource_definition>` (which may be a simple tree or a flat list of
-(which may be a simple tree or a flat list of key-value pairs)
+key-value pairs) which enables Watcher :ref:`Strategies <strategy_definition>`
-which enables Watcher :ref:`Strategies <strategy_definition>` to know the
+to know the current relationships between the different :ref:`resources
-current relationships between the different
+<managed_resource_definition>`) of the :ref:`Cluster <cluster_definition>`
-:ref:`resources <managed_resource_definition>`) of the
+during an :ref:`Audit <audit_definition>` and enables the :ref:`Strategy
-:ref:`Cluster <cluster_definition>` during an :ref:`Audit <audit_definition>`
+<strategy_definition>` to request information such as:
-and enables the :ref:`Strategy <strategy_definition>` to request information
-such as:
 
--  What compute nodes are in a given
+- What compute nodes are in a given :ref:`Availability Zone
-   :ref:`Availability Zone <availability_zone_definition>`
+  <availability_zone_definition>` or a given :ref:`Host Aggregate
-   or a given :ref:`Host Aggregate <host_aggregates_definition>` ?
+  <host_aggregates_definition>`?
--  What :ref:`Instances <instance_definition>` are hosted on a given compute
+- What :ref:`Instances <instance_definition>` are hosted on a given compute
-   node ?
+  node?
--  What is the current load of a compute node ?
+- What is the current load of a compute node?
--  What is the current free memory of a compute node ?
+- What is the current free memory of a compute node?
--  What is the network link between two compute nodes ?
+- What is the network link between two compute nodes?
--  What is the available bandwidth on a given network link ?
+- What is the available bandwidth on a given network link?
--  What is the current space available on a given virtual disk of a given
+- What is the current space available on a given virtual disk of a given
-   :ref:`Instance <instance_definition>` ?
+  :ref:`Instance <instance_definition>` ?
--  What is the current state of a given :ref:`Instance <instance_definition>`?
+- What is the current state of a given :ref:`Instance <instance_definition>`?
--  ...
+- ...
 
 In a word, this data model enables the :ref:`Strategy <strategy_definition>`
 to know:
 
--  the current topology of the :ref:`Cluster <cluster_definition>`
+- the current topology of the :ref:`Cluster <cluster_definition>`
--  the current capacity for each
+- the current capacity for each :ref:`Managed resource
-   :ref:`Managed resource <managed_resource_definition>`
+  <managed_resource_definition>`
--  the current amount of used/free space for each
+- the current amount of used/free space for each :ref:`Managed resource
-   :ref:`Managed resource <managed_resource_definition>`
+  <managed_resource_definition>`
--  the current state of each
+- the current state of each :ref:`Managed resources
-   :ref:`Managed resources <managed_resource_definition>`
+  <managed_resource_definition>`
 
-In the Watcher project, we aim at providing a generic and very basic
+In the Watcher project, we aim at providing a some generic and basic
-:ref:`Cluster Data Model <cluster_data_model_definition>` for each
+:ref:`Cluster Data Model <cluster_data_model_definition>` for each :ref:`Goal
-:ref:`Goal <goal_definition>`, usable in the associated
+<goal_definition>`, usable in the associated :ref:`Strategies
-:ref:`Strategies <strategy_definition>` through some helper classes in order
+<strategy_definition>` through a plugin-based mechanism that are directly
-to:
+accessible from the strategies classes in order to:
 
--  simplify the development of a new
+- simplify the development of a new :ref:`Strategy <strategy_definition>` for a
-   :ref:`Strategy <strategy_definition>` for a given
+  given :ref:`Goal <goal_definition>` when there already are some existing
-   :ref:`Goal <goal_definition>` when there already are some existing
+  :ref:`Strategies <strategy_definition>` associated to the same :ref:`Goal
-   :ref:`Strategies <strategy_definition>` associated to the same
+  <goal_definition>`
-   :ref:`Goal <goal_definition>`
+- avoid duplicating the same code in several :ref:`Strategies
--  avoid duplicating the same code in several
+  <strategy_definition>` associated to the same :ref:`Goal <goal_definition>`
-   :ref:`Strategies <strategy_definition>` associated to the same
+- have a better consistency between the different :ref:`Strategies
-   :ref:`Goal <goal_definition>`
+  <strategy_definition>` for a given :ref:`Goal <goal_definition>`
--  have a better consistency between the different
+- avoid any strong coupling with any external :ref:`Cluster Data Model
-   :ref:`Strategies <strategy_definition>` for a given
+  <cluster_data_model_definition>` (the proposed data model acts as a pivot
-   :ref:`Goal <goal_definition>`
+  data model)
--  avoid any strong coupling with any external
-   :ref:`Cluster Data Model <cluster_data_model_definition>`
-   (the proposed data model acts as a pivot data model)
 
-There may be various
+There may be various :ref:`generic and basic Cluster Data Models
-:ref:`generic and basic Cluster Data Models <cluster_data_model_definition>`
+<cluster_data_model_definition>` proposed in Watcher helpers, each of them
-proposed in Watcher helpers, each of them being adapted to achieving a given
+being adapted to achieving a given :ref:`Goal <goal_definition>`:
-:ref:`Goal <goal_definition>`:
 
--  For example, for a
+- For example, for a :ref:`Goal <goal_definition>` which aims at optimizing
-   :ref:`Goal <goal_definition>` which aims at optimizing the network
+  the network :ref:`resources <managed_resource_definition>` the :ref:`Strategy
-   :ref:`resources <managed_resource_definition>` the
+  <strategy_definition>` may need to know which :ref:`resources
-   :ref:`Strategy <strategy_definition>` may need to know which
+  <managed_resource_definition>` are communicating together.
-   :ref:`resources <managed_resource_definition>` are communicating together.
+- Whereas for a :ref:`Goal <goal_definition>` which aims at optimizing thermal
--  Whereas for a :ref:`Goal <goal_definition>` which aims at optimizing thermal
+  and power conditions, the :ref:`Strategy <strategy_definition>` may need to
-   and power conditions, the :ref:`Strategy <strategy_definition>` may need to
+  know the location of each compute node in the racks and the location of each
-   know the location of each compute node in the racks and the location of each
+  rack in the room.
-   rack in the room.
 
-Note however that a developer can use his/her own
+Note however that a developer can use his/her own :ref:`Cluster Data Model
-:ref:`Cluster Data Model <cluster_data_model_definition>` if the proposed data
+<cluster_data_model_definition>` if the proposed data model does not fit
-model does not fit his/her needs as long as the
+his/her needs as long as the :ref:`Strategy <strategy_definition>` is able to
-:ref:`Strategy <strategy_definition>` is able to produce a
+produce a :ref:`Solution <solution_definition>` for the requested :ref:`Goal
-:ref:`Solution <solution_definition>` for the requested
+<goal_definition>`. For example, a developer could rely on the Nova Data Model
-:ref:`Goal <goal_definition>`.
+to optimize some compute resources.
-For example, a developer could rely on the Nova Data Model to optimize some
-compute resources.
 
 The :ref:`Cluster Data Model <cluster_data_model_definition>` may be persisted
 in any appropriate storage system (SQL database, NoSQL database, JSON file,
-XML File, In Memory Database, ...).
+XML File, In Memory Database, ...). As of now, an in-memory model is built and
+maintained in the background in order to accelerate the execution of
+strategies.
 """
 
 import abc

watcher/opts.py

@@ -26,6 +26,7 @@ from watcher.common import utils
 from watcher.decision_engine.loading import default as decision_engine_loader
 from watcher.decision_engine import manager as decision_engine_manger
 from watcher.decision_engine.planner import manager as planner_manager
+from watcher.metrics_engine.loading import default as cdm_loader
 
 
 PLUGIN_LOADERS = (
@@ -33,6 +34,7 @@ PLUGIN_LOADERS = (
     decision_engine_loader.DefaultPlannerLoader,
     decision_engine_loader.DefaultStrategyLoader,
     applier_loader.DefaultWorkFlowEngineLoader,
+    cdm_loader.ClusterDataModelCollectorLoader,
 )