Cluster data model collector plugin documentation

In this changeset, I wrote down the documentation on how to implement
a cluster data model collector plugin for Watcher.

This documentation corresponds to part 1 of the associated
specification.

Change-Id: Iac72b933df95252163033cd559d13348075a9b16
Partially-Implements: blueprint cluster-model-objects-wrapper

doc/source/conf.py

@@ -19,12 +19,13 @@ from watcher import version as watcher_version
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     'oslo_config.sphinxconfiggen',
+    'oslosphinx',
     'sphinx.ext.autodoc',
     'sphinx.ext.viewcode',
     'sphinxcontrib.httpdomain',
     'sphinxcontrib.pecanwsme.rest',
+    'stevedore.sphinxext',
     'wsmeext.sphinxext',
-    'oslosphinx',
     'watcher.doc',
 ]
 

doc/source/dev/plugin/action-plugin.rst

@@ -80,12 +80,9 @@ Here is an example showing how you can write a plugin called ``DummyAction``:
             pass
 
 
-This implementation is the most basic one. So if you want to have more advanced
-examples, have a look at the implementation of the actions already provided
-by Watcher like.
-To get a better understanding on how to implement a more advanced action,
-have a look at the :py:class:`~watcher.applier.actions.migration.Migrate`
-class.
+This implementation is the most basic one. So in order to get a better
+understanding on how to implement a more advanced action, have a look at the
+:py:class:`~watcher.applier.actions.migration.Migrate` class.
 
 Input validation
 ----------------

doc/source/dev/plugin/base-setup.rst

@@ -88,10 +88,13 @@ Now that the project skeleton has been created, you can start the
 implementation of your plugin. As of now, you can implement the following
 plugins for Watcher:
 
+- A :ref:`goal plugin <implement_goal_plugin>`
 - A :ref:`strategy plugin <implement_strategy_plugin>`
-- A :ref:`planner plugin <implement_planner_plugin>`
 - An :ref:`action plugin <implement_action_plugin>`
-- A :ref:`workflow engine plugin <implement_workflow_engine_plugin>`
+- A :ref:`planner plugin <implement_planner_plugin>`
+- A workflow engine plugin
+- A :ref:`cluster data model collector plugin
+  <implement_cluster_data_model_collector_plugin>`
 
 If you want to learn more on how to implement them, you can refer to their
 dedicated documentation.

doc/source/dev/plugin/cdmc-plugin.rst

@@ -0,0 +1,150 @@
+..
+      Except where otherwise noted, this document is licensed under Creative
+      Commons Attribution 3.0 License.  You can view the license at:
+
+          https://creativecommons.org/licenses/by/3.0/
+
+.. _implement_cluster_data_model_collector_plugin:
+
+========================================
+Build a new cluster data model collector
+========================================
+
+Watcher Decision Engine has an external cluster data model plugin interface
+which gives anyone the ability to integrate an external cluster data model
+collector in order to extend the initial set of cluster data model collectors
+Watcher provides.
+
+This section gives some guidelines on how to implement and integrate custom
+cluster data model collectors within Watcher.
+
+
+Creating a new plugin
+=====================
+
+First of all, you have to extend the :class:`~.BaseClusterDataModelCollector`
+base class which defines the :py:meth:`~.BaseClusterDataModelCollector.execute`
+abstract method you will have to implement. This method is responsible for
+building an entire cluster data model.
+
+Here is an example showing how you can write a plugin called
+``DummyClusterDataModelCollector``:
+
+.. code-block:: python
+
+    # Filepath = <PROJECT_DIR>/thirdparty/dummy.py
+    # Import path = thirdparty.dummy
+
+    from watcher.decision_engine.model import model_root
+    from watcher.metrics_engine.cluster_model_collector import base
+
+
+    class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
+
+        def execute(self):
+            model = model_root.ModelRoot()
+            # Do something here...
+            return model
+
+This implementation is the most basic one. So in order to get a better
+understanding on how to implement a more advanced cluster data model collector,
+have a look at the :py:class:`~.NovaClusterDataModelCollector` class.
+
+Define configuration parameters
+===============================
+
+At this point, you have a fully functional cluster data model collector.
+However, in more complex implementation, you may want to define some
+configuration options so one can tune the cluster data model collector to its
+needs. To do so, you can implement the :py:meth:`~.Loadable.get_config_opts`
+class method as followed:
+
+.. code-block:: python
+
+    from oslo_config import cfg
+    from watcher.decision_engine.model import model_root
+    from watcher.metrics_engine.cluster_model_collector import base
+
+
+    class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
+
+        def execute(self):
+            model = model_root.ModelRoot()
+            # Do something here...
+            return model
+
+        def get_config_opts(self):
+            return [
+                cfg.StrOpt('test_opt', help="Demo Option.", default=0),
+                # Some more options ...
+            ]
+
+The configuration options defined within this class method will be included
+within the global ``watcher.conf`` configuration file under a section named by
+convention: ``{namespace}.{plugin_name}``. In our case, the ``watcher.conf``
+configuration would have to be modified as followed:
+
+.. code-block:: ini
+
+    [watcher_cluster_data_model_collectors.dummy]
+    # Option used for testing.
+    test_opt = test_value
+
+Then, the configuration options you define within this method will then be
+injected in each instantiated object via the  ``config`` parameter of the
+:py:meth:`~.BaseClusterDataModelCollector.__init__` method.
+
+
+Abstract Plugin Class
+=====================
+
+Here below is the abstract ``BaseClusterDataModelCollector`` class that every
+single cluster data model collector should implement:
+
+.. autoclass:: watcher.metrics_engine.cluster_model_collector.base.BaseClusterDataModelCollector
+    :members:
+    :special-members: __init__
+    :noindex:
+
+
+Register a new entry point
+==========================
+
+In order for the Watcher Decision Engine to load your new cluster data model
+collector, the latter must be registered as a named entry point under the
+``watcher_cluster_data_model_collectors`` entry point of your ``setup.py``
+file. If you are using pbr_, this entry point should be placed in your
+``setup.cfg`` file.
+
+The name you give to your entry point has to be unique.
+
+Here below is how to register ``DummyClusterDataModelCollector`` using pbr_:
+
+.. code-block:: ini
+
+    [entry_points]
+    watcher_cluster_data_model_collectors =
+        dummy = thirdparty.dummy:DummyClusterDataModelCollector
+
+.. _pbr: http://docs.openstack.org/developer/pbr/
+
+
+Using cluster data model collector plugins
+==========================================
+
+The Watcher Decision Engine service will automatically discover any installed
+plugins when it is restarted. If a Python package containing a custom plugin is
+installed within the same environment as Watcher, Watcher will automatically
+make that plugin available for use.
+
+At this point, you can use your new cluster data model plugin in your
+:ref:`strategy plugin <implement_strategy_plugin>` by using the
+:py:attr:`~.BaseStrategy.collector_manager` property as followed:
+
+.. code-block:: python
+
+    # [...]
+    dummy_collector = self.collector_manager.get_cluster_model_collector(
+        "dummy")  # "dummy" is the name of the entry point we declared earlier
+    dummy_model = collector.get_latest_cluster_data_model()
+    # Do some stuff with this model

doc/source/dev/plugins.rst

@@ -18,32 +18,43 @@ use the :ref:`Guru Meditation Reports <watcher_gmr>` to display them.
 Goals
 =====
 
-.. drivers-doc:: watcher_goals
+.. list-plugins:: watcher_goals
+   :detailed:
 
 .. _watcher_strategies:
 
 Strategies
 ==========
 
-.. drivers-doc:: watcher_strategies
+.. list-plugins:: watcher_strategies
+   :detailed:
 
 .. _watcher_actions:
 
 Actions
 =======
 
-.. drivers-doc:: watcher_actions
+.. list-plugins:: watcher_actions
+   :detailed:
 
 .. _watcher_workflow_engines:
 
 Workflow Engines
 ================
 
-.. drivers-doc:: watcher_workflow_engines
+.. list-plugins:: watcher_workflow_engines
+   :detailed:
 
 .. _watcher_planners:
 
 Planners
 ========
 
-.. drivers-doc:: watcher_planners
+.. list-plugins:: watcher_planners
+   :detailed:
+
+Cluster Data Model Collectors
+=============================
+
+.. list-plugins:: watcher_cluster_data_model_collectors
+   :detailed:

doc/source/index.rst

@@ -75,6 +75,7 @@ Plugins
   dev/plugin/base-setup
   dev/plugin/goal-plugin
   dev/plugin/strategy-plugin
+  dev/plugin/cdmc-plugin
   dev/plugin/action-plugin
   dev/plugin/planner-plugin
   dev/plugins