Arnike/watcher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Adapt watcher documentation for new standards

Browse Source 
This patch set makes the following changes:

 * Add index file to each subdirectory of doc/source
 * Update doc/source/index.rst with new links
 * Move content of install-guide to the doc/source/install
 * Minor changes

Depends-On: Ifc5512c0e2373cf3387e0e0498268eab092e52bb
Change-Id: Iecb4f60efb015a56b9b37331859848b287112842
This commit is contained in:
Alexander Chadin
2017-07-03 15:31:10 +03:00
parent 5f179609d0
commit c7ec186576
47 changed files with 106 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
doc/source/contributor/contributing.rst Normal file
View File

@@ -0,0 +1,71 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _contributing:
=======================
Contributing to Watcher
=======================
If you're interested in contributing to the Watcher project,
the following will help get you started.
Contributor License Agreement
-----------------------------
.. index::
single: license; agreement
In order to contribute to the Watcher project, you need to have
signed OpenStack's contributor's agreement.
.. seealso::
* http://docs.openstack.org/infra/manual/developers.html
* http://wiki.openstack.org/CLA
LaunchPad Project
-----------------
Most of the tools used for OpenStack depend on a launchpad.net ID for
authentication. After signing up for a launchpad account, join the
"openstack" team to have access to the mailing list and receive
notifications of important events.
.. seealso::
* http://launchpad.net
* http://launchpad.net/watcher
* http://launchpad.net/~openstack
Project Hosting Details
-----------------------
Bug tracker
http://launchpad.net/watcher
Mailing list (prefix subjects with ``[watcher]`` for faster responses)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Wiki
http://wiki.openstack.org/Watcher
Code Hosting
https://git.openstack.org/cgit/openstack/watcher
Code Review
https://review.openstack.org/#/q/status:open+project:openstack/watcher,n,z
IRC Channel
``#openstack-watcher`` (changelog_)
Weekly Meetings
on Wednesdays at 14:00 UTC on even weeks, 9:00 UTC on odd weeks, in the
``#openstack-meeting-4`` IRC channel (`meetings logs`_)
.. _changelog: http://eavesdrop.openstack.org/irclogs/%23openstack-watcher/
.. _meetings logs: http://eavesdrop.openstack.org/meetings/watcher/

241
doc/source/contributor/devstack.rst Normal file
View File

@@ -0,0 +1,241 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
=============================================
Set up a development environment via DevStack
=============================================
Watcher is currently able to optimize compute resources - specifically Nova
compute hosts - via operations such as live migrations. In order for you to
fully be able to exercise what Watcher can do, it is necessary to have a
multinode environment to use.
You can set up the Watcher services quickly and easily using a Watcher
DevStack plugin. See `PluginModelDocs`_ for information on DevStack's plugin
model. To enable the Watcher plugin with DevStack, add the following to the
`[[local|localrc]]` section of your controller's `local.conf` to enable the
Watcher plugin::
enable_plugin watcher git://git.openstack.org/openstack/watcher
For more detailed instructions, see `Detailed DevStack Instructions`_. Check
out the `DevStack documentation`_ for more information regarding DevStack.
.. _PluginModelDocs: http://docs.openstack.org/developer/devstack/plugins.html
.. _DevStack documentation: http://docs.openstack.org/developer/devstack/
Detailed DevStack Instructions
==============================
#. Obtain N (where N >= 1) servers (virtual machines preferred for DevStack).
One of these servers will be the controller node while the others will be
compute nodes. N is preferably >= 3 so that you have at least 2 compute
nodes, but in order to stand up the Watcher services only 1 server is
needed (i.e., no computes are needed if you want to just experiment with
the Watcher services). These servers can be VMs running on your local
machine via VirtualBox if you prefer. DevStack currently recommends that
you use Ubuntu 14.04 LTS. The servers should also have connections to the
same network such that they are all able to communicate with one another.
#. For each server, clone the DevStack repository and create the stack user::
sudo apt-get update
sudo apt-get install git
git clone https://git.openstack.org/openstack-dev/devstack
sudo ./devstack/tools/create-stack-user.sh
Now you have a stack user that is used to run the DevStack processes. You
may want to give your stack user a password to allow SSH via a password::
sudo passwd stack
#. Switch to the stack user and clone the DevStack repo again::
sudo su stack
cd ~
git clone https://git.openstack.org/openstack-dev/devstack
#. For each compute node, copy the provided `local.conf.compute`_ example file
to the compute node's system at ~/devstack/local.conf. Make sure the
HOST_IP and SERVICE_HOST values are changed appropriately - i.e., HOST_IP
is set to the IP address of the compute node and SERVICE_HOST is set to the
IP address of the controller node.
If you need specific metrics collected (or want to use something other
than Ceilometer), be sure to configure it. For example, in the
`local.conf.compute`_ example file, the appropriate ceilometer plugins and
services are enabled and disabled. If you were using something other than
Ceilometer, then you would likely want to configure it likewise. The
example file also sets the compute monitors nova configuration option to
use the CPU virt driver. If you needed other metrics, it may be necessary
to configure similar configuration options for the projects providing those
metrics.
#. For the controller node, copy the provided `local.conf.controller`_ example
file to the controller node's system at ~/devstack/local.conf. Make sure
the HOST_IP value is changed appropriately - i.e., HOST_IP is set to the IP
address of the controller node.
Note: if you want to use another Watcher git repository (such as a local
one), then change the enable plugin line::
enable_plugin watcher <your_local_git_repo> [optional_branch]
If you do this, then the Watcher DevStack plugin will try to pull the
python-watcherclient repo from <your_local_git_repo>/../, so either make
sure that is also available or specify WATCHERCLIENT_REPO in the local.conf
file.
Note: if you want to use a specific branch, specify WATCHER_BRANCH in the
local.conf file. By default it will use the master branch.
Note: watcher-api will default run under apache/httpd, set the variable
WATCHER_USE_MOD_WSGI=FALSE if you do not wish to run under apache/httpd.
For development environment it is suggested to set WATHCER_USE_MOD_WSGI
to FALSE. For Production environment it is suggested to keep it at the
default TRUE value.
#. Start stacking from the controller node::
./devstack/stack.sh
#. Start stacking on each of the compute nodes using the same command.
#. Configure the environment for live migration via NFS. See the
`Multi-Node DevStack Environment`_ section for more details.
.. _local.conf.controller: https://github.com/openstack/watcher/tree/master/devstack/local.conf.controller
.. _local.conf.compute: https://github.com/openstack/watcher/tree/master/devstack/local.conf.compute
Multi-Node DevStack Environment
===============================
Since deploying Watcher with only a single compute node is not very useful, a
few tips are given here for enabling a multi-node environment with live
migration.
Configuring NFS Server
----------------------
If you would like to use live migration for shared storage, then the controller
can serve as the NFS server if needed::
sudo apt-get install nfs-kernel-server
sudo mkdir -p /nfs/instances
sudo chown stack:stack /nfs/instances
Add an entry to `/etc/exports` with the appropriate gateway and netmask
information::
/nfs/instances <gateway>/<netmask>(rw,fsid=0,insecure,no_subtree_check,async,no_root_squash)
Export the NFS directories::
sudo exportfs -ra
Make sure the NFS server is running::
sudo service nfs-kernel-server status
If the server is not running, then start it::
sudo service nfs-kernel-server start
Configuring NFS on Compute Node
-------------------------------
Each compute node needs to use the NFS server to hold the instance data::
sudo apt-get install rpcbind nfs-common
mkdir -p /opt/stack/data/instances
sudo mount <nfs-server-ip>:/nfs/instances /opt/stack/data/instances
If you would like to have the NFS directory automatically mounted on reboot,
then add the following to `/etc/fstab`::
<nfs-server-ip>:/nfs/instances /opt/stack/data/instances nfs auto 0 0
Edit `/etc/libvirt/libvirtd.conf` to make sure the following values are set::
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
Edit `/etc/default/libvirt-bin`::
libvirtd_opts="-d -l"
Restart the libvirt service::
sudo service libvirt-bin restart
Setting up SSH keys between compute nodes to enable live migration
------------------------------------------------------------------
In order for live migration to work, SSH keys need to be exchanged between
each compute node:
1. The SOURCE root user's public RSA key (likely in /root/.ssh/id_rsa.pub)
needs to be in the DESTINATION stack user's authorized_keys file
(~stack/.ssh/authorized_keys). This can be accomplished by manually
copying the contents from the file on the SOURCE to the DESTINATION. If
you have a password configured for the stack user, then you can use the
following command to accomplish the same thing::
ssh-copy-id -i /root/.ssh/id_rsa.pub stack@DESTINATION
2. The DESTINATION host's public ECDSA key (/etc/ssh/ssh_host_ecdsa_key.pub)
needs to be in the SOURCE root user's known_hosts file
(/root/.ssh/known_hosts). This can be accomplished by running the
following on the SOURCE machine (hostname must be used)::
ssh-keyscan -H DEST_HOSTNAME | sudo tee -a /root/.ssh/known_hosts
In essence, this means that every compute node's root user's public RSA key
must exist in every other compute node's stack user's authorized_keys file and
every compute node's public ECDSA key needs to be in every other compute
node's root user's known_hosts file.
Disable serial console
----------------------
Serial console needs to be disabled for live migration to work.
On both the controller and compute node, in /etc/nova/nova.conf
[serial_console]
enabled = False
Alternatively, in devstack's local.conf:
[[post-config|$NOVA_CONF]]
[serial_console]
#enabled=false
VNC server configuration
------------------------
The VNC server listening parameter needs to be set to any address so
that the server can accept connections from all of the compute nodes.
On both the controller and compute node, in /etc/nova/nova.conf
vncserver_listen = 0.0.0.0
Alternatively, in devstack's local.conf:
VNCSERVER_LISTEN=0.0.0.0
Environment final checkup
-------------------------
If you are willing to make sure everything is in order in your DevStack
environment, you can run the Watcher Tempest tests which will validate its API
but also that you can perform the typical Watcher workflows. To do so, have a
look at the :ref:`Tempest tests <tempest_tests>` section which will explain to
you how to run them.

275
doc/source/contributor/environment.rst Normal file
View File

@@ -0,0 +1,275 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _watcher_developement_environment:
=========================================
Set up a development environment manually
=========================================
This document describes getting the source from watcher `Git repository`_
for development purposes.
To install Watcher from packaging, refer instead to Watcher `User
Documentation`_.
.. _`Git Repository`: https://git.openstack.org/cgit/openstack/watcher
.. _`User Documentation`: https://docs.openstack.org/developer/watcher/
Prerequisites
=============
This document assumes you are using Ubuntu or Fedora, and that you have the
following tools available on your system:
- Python_ 2.7 and 3.4
- git_
- setuptools_
- pip_
- msgfmt (part of the gettext package)
- virtualenv and virtualenvwrapper_
**Reminder**: If you're successfully using a different platform, or a
different version of the above, please document your configuration here!
.. _Python: https://www.python.org/
.. _git: https://git-scm.com/
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.io/en/latest/install.html
Getting the latest code
=======================
Make a clone of the code from our `Git repository`:
.. code-block:: bash
$ git clone https://git.openstack.org/openstack/watcher.git
When that is complete, you can:
.. code-block:: bash
$ cd watcher
Installing dependencies
=======================
Watcher maintains two lists of dependencies::
requirements.txt
test-requirements.txt
The first is the list of dependencies needed for running Watcher, the second
list includes dependencies used for active development and testing of Watcher
itself.
These dependencies can be installed from PyPi_ using the Python tool pip_.
.. _PyPi: http://pypi.python.org/
.. _pip: http://pypi.python.org/pypi/pip
However, your system *may* need additional dependencies that `pip` (and by
extension, PyPi) cannot satisfy. These dependencies should be installed
prior to using `pip`, and the installation method may vary depending on
your platform.
* Ubuntu 14.04::
$ sudo apt-get install python-dev libssl-dev libmysqlclient-dev libffi-dev
* Fedora 19+::
$ sudo yum install openssl-devel libffi-devel mysql-devel
* CentOS 7::
$ sudo yum install gcc python-devel libxml2-devel libxslt-devel mariadb-devel
PyPi Packages and VirtualEnv
----------------------------
We recommend establishing a virtualenv to run Watcher within. virtualenv
limits the Python environment to just what you're installing as dependencies,
useful to keep a clean environment for working on Watcher.
.. code-block:: bash
$ mkvirtualenv watcher
$ git clone https://git.openstack.org/openstack/watcher
# Use 'python setup.py' to link Watcher into Python's site-packages
$ cd watcher && python setup.py install
# Install the dependencies for running Watcher
$ pip install -r ./requirements.txt
# Install the dependencies for developing, testing, and running Watcher
$ pip install -r ./test-requirements.txt
This will create a local virtual environment in the directory ``$WORKON_HOME``.
The virtual environment can be disabled using the command:
.. code-block:: bash
$ deactivate
You can re-activate this virtualenv for your current shell using:
.. code-block:: bash
$ workon watcher
For more information on virtual environments, see virtualenv_.
.. _virtualenv: http://www.virtualenv.org/
Verifying Watcher is set up
===========================
Once set up, either directly or within a virtualenv, you should be able to
invoke Python and import the libraries. If you're using a virtualenv, don't
forget to activate it:
.. code-block:: bash
$ workon watcher
You should then be able to `import watcher` using Python without issue:
.. code-block:: bash
$ python -c "import watcher"
If you can import watcher without a traceback, you should be ready to develop.
Run Watcher tests
=================
Watcher provides both :ref:`unit tests <unit_tests>` and
:ref:`functional/tempest tests <tempest_tests>`. Please refer to :doc:`testing`
to understand how to run them.
Build the Watcher documentation
===============================
You can easily build the HTML documentation from ``doc/source`` files, by using
``tox``:
.. code-block:: bash
$ workon watcher
(watcher) $ cd watcher
(watcher) $ tox -edocs
The HTML files are available into ``doc/build`` directory.
Configure the Watcher services
==============================
Watcher services require a configuration file. Use tox to generate
a sample configuration file that can be used to get started:
.. code-block:: bash
$ tox -e genconfig
$ cp etc/watcher.conf.sample etc/watcher.conf
Most of the default configuration should be enough to get you going, but you
still need to configure the following sections:
- The ``[database]`` section to configure the
:ref:`Watcher database <watcher-db_configuration>`
- The ``[keystone_authtoken]`` section to configure the
:ref:`Identity service <identity-service_configuration>` i.e. Keystone
- The ``[watcher_messaging]`` section to configure the OpenStack AMQP-based
message bus
So if you need some more details on how to configure one or more of these
sections, please do have a look at :doc:`../deploy/configuration` before
continuing.
Create Watcher SQL database
===========================
When initially getting set up, after you've configured which databases to use,
you're probably going to need to run the following to your database schema in
place:
.. code-block:: bash
$ workon watcher
(watcher) $ watcher-db-manage create_schema
Running Watcher services
========================
To run the Watcher API service, use:
.. code-block:: bash
$ workon watcher
(watcher) $ watcher-api
To run the Watcher Decision Engine service, use:
.. code-block:: bash
$ workon watcher
(watcher) $ watcher-decision-engine
To run the Watcher Applier service, use:
.. code-block:: bash
$ workon watcher
(watcher) $ watcher-applier
Default configuration of these services are available into ``/etc/watcher``
directory. See :doc:`../deploy/configuration` for details on how Watcher is
configured. By default, Watcher is configured with SQL backends.
Interact with Watcher
=====================
You can also interact with Watcher through its REST API. There is a Python
Watcher client library `python-watcherclient`_ which interacts exclusively
through the REST API, and which Watcher itself uses to provide its command-line
interface.
.. _`python-watcherclient`: https://github.com/openstack/python-watcherclient
There is also an Horizon plugin for Watcher `watcher-dashboard`_ which
allows to interact with Watcher through a web-based interface.
.. _`watcher-dashboard`: https://github.com/openstack/watcher-dashboard
Exercising the Watcher Services locally
=======================================
If you would like to exercise the Watcher services in isolation within a local
virtual environment, you can do this without starting any other OpenStack
services. For example, this is useful for rapidly prototyping and debugging
interactions over the RPC channel, testing database migrations, and so forth.
You will find in the `watcher-tools`_ project, Ansible playbooks and Docker
template files to easily play with Watcher services within a minimal OpenStack
isolated environment (Identity, Message Bus, SQL database, Horizon, ...).
.. _`watcher-tools`: https://github.com/b-com/watcher-tools

8
doc/source/contributor/index.rst Normal file
View File

@@ -0,0 +1,8 @@
.. toctree::
:maxdepth: 1
environment
devstack
notifications
testing
rally_link

13
doc/source/contributor/notifications.rst Normal file
View File

@@ -0,0 +1,13 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _watcher_notifications:
========================
Notifications in Watcher
========================
.. versioned_notifications::

219
doc/source/contributor/plugin/action-plugin.rst Normal file
View File

@@ -0,0 +1,219 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_action_plugin:
==================
Build a new action
==================
Watcher Applier has an external :ref:`action <action_definition>` plugin
interface which gives anyone the ability to integrate an external
:ref:`action <action_definition>` in order to extend the initial set of actions
Watcher provides.
This section gives some guidelines on how to implement and integrate custom
actions with Watcher.
Creating a new plugin
=====================
First of all you have to extend the base :py:class:`BaseAction` class which
defines a set of abstract methods and/or properties that you will have to
implement:
- The :py:attr:`~.BaseAction.schema` is an abstract property that you have to
implement. This is the first function to be called by the
:ref:`applier <watcher_applier_definition>` before any further processing
and its role is to validate the input parameters that were provided to it.
- The :py:meth:`~.BaseAction.pre_condition` is called before the execution of
an action. This method is a hook that can be used to perform some
initializations or to make some more advanced validation on its input
parameters. If you wish to block the execution based on this factor, you
simply have to ``raise`` an exception.
- The :py:meth:`~.BaseAction.post_condition` is called after the execution of
an action. As this function is called regardless of whether an action
succeeded or not, this can prove itself useful to perform cleanup
operations.
- The :py:meth:`~.BaseAction.execute` is the main component of an action.
This is where you should implement the logic of your action.
- The :py:meth:`~.BaseAction.revert` allows you to roll back the targeted
resource to its original state following a faulty execution. Indeed, this
method is called by the workflow engine whenever an action raises an
exception.
Here is an example showing how you can write a plugin called ``DummyAction``:
.. code-block:: python
# Filepath = <PROJECT_DIR>/thirdparty/dummy.py
# Import path = thirdparty.dummy
import voluptuous
from watcher.applier.actions import base
class DummyAction(base.BaseAction):
@property
def schema(self):
return voluptuous.Schema({})
def execute(self):
# Does nothing
pass # Only returning False is considered as a failure
def revert(self):
# Does nothing
pass
def pre_condition(self):
# No pre-checks are done here
pass
def post_condition(self):
# Nothing done here
pass
This implementation is the most basic one. So in order to get a better
understanding on how to implement a more advanced action, have a look at the
:py:class:`~watcher.applier.actions.migration.Migrate` class.
Input validation
----------------
As you can see in the previous example, we are using `Voluptuous`_ to validate
the input parameters of an action. So if you want to learn more about how to
work with `Voluptuous`_, you can have a look at their `documentation`_:
.. _Voluptuous: https://github.com/alecthomas/voluptuous
.. _documentation: https://github.com/alecthomas/voluptuous/blob/master/README.md
Define configuration parameters
===============================
At this point, you have a fully functional action. However, in more complex
implementation, you may want to define some configuration options so one can
tune the action to its needs. To do so, you can implement the
:py:meth:`~.Loadable.get_config_opts` class method as followed:
.. code-block:: python
from oslo_config import cfg
class DummyAction(base.BaseAction):
# [...]
def execute(self):
assert self.config.test_opt == 0
@classmethod
def get_config_opts(cls):
return super(
DummyAction, cls).get_config_opts() + [
cfg.StrOpt('test_opt', help="Demo Option.", default=0),
# Some more options ...
]
The configuration options defined within this class method will be included
within the global ``watcher.conf`` configuration file under a section named by
convention: ``{namespace}.{plugin_name}``. In our case, the ``watcher.conf``
configuration would have to be modified as followed:
.. code-block:: ini
[watcher_actions.dummy]
# Option used for testing.
test_opt = test_value
Then, the configuration options you define within this method will then be
injected in each instantiated object via the ``config`` parameter of the
:py:meth:`~.BaseAction.__init__` method.
Abstract Plugin Class
=====================
Here below is the abstract ``BaseAction`` class that every single action
should implement:
.. autoclass:: watcher.applier.actions.base.BaseAction
:members:
:special-members: __init__
:noindex:
.. py:attribute:: schema
Defines a Schema that the input parameters shall comply to
:returns: A schema declaring the input parameters this action should be
provided along with their respective constraints
(e.g. type, value range, ...)
:rtype: :py:class:`voluptuous.Schema` instance
Register a new entry point
==========================
In order for the Watcher Applier to load your new action, the
action must be registered as a named entry point under the
``watcher_actions`` entry point of your ``setup.py`` file. If you are using
pbr_, this entry point should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique.
Here below is how you would proceed to register ``DummyAction`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_actions =
dummy = thirdparty.dummy:DummyAction
.. _pbr: http://docs.openstack.org/developer/pbr/
Using action plugins
====================
The Watcher Applier service will automatically discover any installed plugins
when it is restarted. If a Python package containing a custom plugin is
installed within the same environment as Watcher, Watcher will automatically
make that plugin available for use.
At this point, you can use your new action plugin in your :ref:`strategy plugin
<implement_strategy_plugin>` if you reference it via the use of the
:py:meth:`~.Solution.add_action` method:
.. code-block:: python
# [...]
self.solution.add_action(
action_type="dummy", # Name of the entry point we registered earlier
applies_to="",
input_parameters={})
By doing so, your action will be saved within the Watcher Database, ready to be
processed by the planner for creating an action plan which can then be executed
by the Watcher Applier via its workflow engine.
At the last, remember to add the action into the weights in ``watcher.conf``,
otherwise you will get an error when the action be referenced in a strategy.
Scheduling of an action plugin
==============================
Watcher provides a basic built-in :ref:`planner <watcher_planner_definition>`
which is only able to process the Watcher built-in actions. Therefore, you will
either have to use an existing third-party planner or :ref:`implement another
planner <implement_planner_plugin>` that will be able to take into account your
new action plugin.

100
doc/source/contributor/plugin/base-setup.rst Normal file
View File

@@ -0,0 +1,100 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _plugin-base_setup:
=======================================
Create a third-party plugin for Watcher
=======================================
Watcher provides a plugin architecture which allows anyone to extend the
existing functionalities by implementing third-party plugins. This process can
be cumbersome so this documentation is there to help you get going as quickly
as possible.
Pre-requisites
==============
We assume that you have set up a working Watcher development environment. So if
this not already the case, you can check out our documentation which explains
how to set up a :ref:`development environment
<watcher_developement_environment>`.
.. _development environment:
Third party project scaffolding
===============================
First off, we need to create the project structure. To do so, we can use
`cookiecutter`_ and the `OpenStack cookiecutter`_ project scaffolder to
generate the skeleton of our project::
$ virtualenv thirdparty
$ source thirdparty/bin/activate
$ pip install cookiecutter
$ cookiecutter https://github.com/openstack-dev/cookiecutter
The last command will ask you for many information, and If you set
``module_name`` and ``repo_name`` as ``thirdparty``, you should end up with a
structure that looks like this::
$ cd thirdparty
$ tree .
.
├── babel.cfg
├── CONTRIBUTING.rst
├── doc
│   └── source
│   ├── conf.py
│   ├── contributing.rst
│   ├── index.rst
│   ├── installation.rst
│   ├── readme.rst
│   └── usage.rst
├── HACKING.rst
├── LICENSE
├── MANIFEST.in
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── test-requirements.txt
├── thirdparty
│   ├── __init__.py
│   └── tests
│   ├── base.py
│   ├── __init__.py
│   └── test_thirdparty.py
└── tox.ini
**Note:** You should add `python-watcher`_ as a dependency in the
requirements.txt file::
# Watcher-specific requirements
python-watcher
.. _cookiecutter: https://github.com/audreyr/cookiecutter
.. _OpenStack cookiecutter: https://github.com/openstack-dev/cookiecutter
.. _python-watcher: https://pypi.python.org/pypi/python-watcher
Implementing a plugin for Watcher
=================================
Now that the project skeleton has been created, you can start the
implementation of your plugin. As of now, you can implement the following
plugins for Watcher:
- A :ref:`goal plugin <implement_goal_plugin>`
- A :ref:`strategy plugin <implement_strategy_plugin>`
- An :ref:`action plugin <implement_action_plugin>`
- A :ref:`planner plugin <implement_planner_plugin>`
- A workflow engine plugin
- A :ref:`cluster data model collector plugin
<implement_cluster_data_model_collector_plugin>`
If you want to learn more on how to implement them, you can refer to their
dedicated documentation.

272
doc/source/contributor/plugin/cdmc-plugin.rst Normal file
View File

@@ -0,0 +1,272 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_cluster_data_model_collector_plugin:
========================================
Build a new cluster data model collector
========================================
Watcher Decision Engine has an external cluster data model (CDM) plugin
interface which gives anyone the ability to integrate an external cluster data
model collector (CDMC) in order to extend the initial set of cluster data model
collectors Watcher provides.
This section gives some guidelines on how to implement and integrate custom
cluster data model collectors within Watcher.
Creating a new plugin
=====================
In order to create a new cluster data model collector, you have to:
- Extend the :py:class:`~.base.BaseClusterDataModelCollector` class.
- Implement its :py:meth:`~.BaseClusterDataModelCollector.execute` abstract
method to return your entire cluster data model that this method should
build.
- Implement its :py:meth:`~.Goal.notification_endpoints` abstract property to
return the list of all the :py:class:`~.base.NotificationEndpoint` instances
that will be responsible for handling incoming notifications in order to
incrementally update your cluster data model.
First of all, you have to extend the :class:`~.BaseClusterDataModelCollector`
base class which defines the :py:meth:`~.BaseClusterDataModelCollector.execute`
abstract method you will have to implement. This method is responsible for
building an entire cluster data model.
Here is an example showing how you can write a plugin called
``DummyClusterDataModelCollector``:
.. code-block:: python
# Filepath = <PROJECT_DIR>/thirdparty/dummy.py
# Import path = thirdparty.dummy
from watcher.decision_engine.model import model_root
from watcher.decision_engine.model.collector import base
class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
def execute(self):
model = model_root.ModelRoot()
# Do something here...
return model
@property
def notification_endpoints(self):
return []
This implementation is the most basic one. So in order to get a better
understanding on how to implement a more advanced cluster data model collector,
have a look at the :py:class:`~.NovaClusterDataModelCollector` class.
Define a custom model
=====================
As you may have noticed in the above example, we are reusing an existing model
provided by Watcher. However, this model can be easily customized by
implementing a new class that would implement the :py:class:`~.Model` abstract
base class. Here below is simple example on how to proceed in implementing a
custom Model:
.. code-block:: python
# Filepath = <PROJECT_DIR>/thirdparty/dummy.py
# Import path = thirdparty.dummy
from watcher.decision_engine.model import base as modelbase
from watcher.decision_engine.model.collector import base
class MyModel(modelbase.Model):
def to_string(self):
return 'MyModel'
class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
def execute(self):
model = MyModel()
# Do something here...
return model
@property
def notification_endpoints(self):
return []
Here below is the abstract ``Model`` class that every single cluster data model
should implement:
.. autoclass:: watcher.decision_engine.model.base.Model
:members:
:special-members: __init__
:noindex:
Define configuration parameters
===============================
At this point, you have a fully functional cluster data model collector.
By default, cluster data model collectors define a ``period`` option (see
:py:meth:`~.BaseClusterDataModelCollector.get_config_opts`) that corresponds
to the interval of time between each synchronization of the in-memory model.
However, in more complex implementation, you may want to define some
configuration options so one can tune the cluster data model collector to your
needs. To do so, you can implement the :py:meth:`~.Loadable.get_config_opts`
class method as followed:
.. code-block:: python
from oslo_config import cfg
from watcher.decision_engine.model import model_root
from watcher.decision_engine.model.collector import base
class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
def execute(self):
model = model_root.ModelRoot()
# Do something here...
return model
@property
def notification_endpoints(self):
return []
@classmethod
def get_config_opts(cls):
return super(
DummyClusterDataModelCollector, cls).get_config_opts() + [
cfg.StrOpt('test_opt', help="Demo Option.", default=0),
# Some more options ...
]
The configuration options defined within this class method will be included
within the global ``watcher.conf`` configuration file under a section named by
convention: ``{namespace}.{plugin_name}`` (see section :ref:`Register a new
entry point <register_new_cdmc_entrypoint>`). The namespace for CDMC plugins is
``watcher_cluster_data_model_collectors``, so in our case, the ``watcher.conf``
configuration would have to be modified as followed:
.. code-block:: ini
[watcher_cluster_data_model_collectors.dummy]
# Option used for testing.
test_opt = test_value
Then, the configuration options you define within this method will then be
injected in each instantiated object via the ``config`` parameter of the
:py:meth:`~.BaseClusterDataModelCollector.__init__` method.
Abstract Plugin Class
=====================
Here below is the abstract ``BaseClusterDataModelCollector`` class that every
single cluster data model collector should implement:
.. autoclass:: watcher.decision_engine.model.collector.base.BaseClusterDataModelCollector
:members:
:special-members: __init__
:noindex:
.. _register_new_cdmc_entrypoint:
Register a new entry point
==========================
In order for the Watcher Decision Engine to load your new cluster data model
collector, the latter must be registered as a named entry point under the
``watcher_cluster_data_model_collectors`` entry point namespace of your
``setup.py`` file. If you are using pbr_, this entry point should be placed in
your ``setup.cfg`` file.
The name you give to your entry point has to be unique.
Here below is how to register ``DummyClusterDataModelCollector`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_cluster_data_model_collectors =
dummy = thirdparty.dummy:DummyClusterDataModelCollector
.. _pbr: http://docs.openstack.org/developer/pbr/
Add new notification endpoints
==============================
At this point, you have a fully functional cluster data model collector.
However, this CDMC is only refreshed periodically via a background scheduler.
As you may sometimes execute a strategy with a stale CDM due to a high activity
on your infrastructure, you can define some notification endpoints that will be
responsible for incrementally updating the CDM based on notifications emitted
by other services such as Nova. To do so, you can implement and register a new
``DummyEndpoint`` notification endpoint regarding a ``dummy`` event as shown
below:
.. code-block:: python
from watcher.decision_engine.model import model_root
from watcher.decision_engine.model.collector import base
class DummyNotification(base.NotificationEndpoint):
@property
def filter_rule(self):
return filtering.NotificationFilter(
publisher_id=r'.*',
event_type=r'^dummy$',
)
def info(self, ctxt, publisher_id, event_type, payload, metadata):
# Do some CDM modifications here...
pass
class DummyClusterDataModelCollector(base.BaseClusterDataModelCollector):
def execute(self):
model = model_root.ModelRoot()
# Do something here...
return model
@property
def notification_endpoints(self):
return [DummyNotification(self)]
Note that if the event you are trying to listen to is published by a new
service, you may have to also add a new topic Watcher will have to subscribe to
in the ``notification_topics`` option of the ``[watcher_decision_engine]``
section.
Using cluster data model collector plugins
==========================================
The Watcher Decision Engine service will automatically discover any installed
plugins when it is restarted. If a Python package containing a custom plugin is
installed within the same environment as Watcher, Watcher will automatically
make that plugin available for use.
At this point, you can use your new cluster data model plugin in your
:ref:`strategy plugin <implement_strategy_plugin>` by using the
:py:attr:`~.BaseStrategy.collector_manager` property as followed:
.. code-block:: python
# [...]
dummy_collector = self.collector_manager.get_cluster_model_collector(
"dummy") # "dummy" is the name of the entry point we declared earlier
dummy_model = dummy_collector.get_latest_cluster_data_model()
# Do some stuff with this model

215
doc/source/contributor/plugin/goal-plugin.rst Normal file
View File

@@ -0,0 +1,215 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_goal_plugin:
================
Build a new goal
================
Watcher Decision Engine has an external :ref:`goal <goal_definition>`
plugin interface which gives anyone the ability to integrate an external
goal which can be achieved by a :ref:`strategy <strategy_definition>`.
This section gives some guidelines on how to implement and integrate custom
goals with Watcher. If you wish to create a third-party package for your
plugin, you can refer to our :ref:`documentation for third-party package
creation <plugin-base_setup>`.
Pre-requisites
==============
Before using any goal, please make sure that none of the existing goals fit
your needs. Indeed, the underlying value of defining a goal is to be able to
compare the efficacy of the action plans resulting from the various strategies
satisfying the same goal. By doing so, Watcher can assist the administrator
in his choices.
Create a new plugin
===================
In order to create a new goal, you have to:
- Extend the :py:class:`~.base.Goal` class.
- Implement its :py:meth:`~.Goal.get_name` class method to return the
**unique** ID of the new goal you want to create. This unique ID should
be the same as the name of :ref:`the entry point you will declare later on
<goal_plugin_add_entrypoint>`.
- Implement its :py:meth:`~.Goal.get_display_name` class method to
return the translated display name of the goal you want to create.
Note: Do not use a variable to return the translated string so it can be
automatically collected by the translation tool.
- Implement its :py:meth:`~.Goal.get_translatable_display_name`
class method to return the translation key (actually the english display
name) of your new goal. The value return should be the same as the
string translated in :py:meth:`~.Goal.get_display_name`.
- Implement its :py:meth:`~.Goal.get_efficacy_specification` method to return
the :ref:`efficacy specification <efficacy_specification_definition>` for
your goal.
Here is an example showing how you can define a new ``NewGoal`` goal plugin:
.. code-block:: python
# filepath: thirdparty/new.py
# import path: thirdparty.new
from watcher._i18n import _
from watcher.decision_engine.goal import base
from watcher.decision_engine.goal.efficacy import specs
class NewGoal(base.Goal):
@classmethod
def get_name(cls):
return "new_goal" # Will be the name of the entry point
@classmethod
def get_display_name(cls):
return _("New Goal")
@classmethod
def get_translatable_display_name(cls):
return "New Goal"
@classmethod
def get_efficacy_specification(cls):
return specs.Unclassified()
As you may have noticed, the :py:meth:`~.Goal.get_efficacy_specification`
method returns an :py:meth:`~.Unclassified` instance which
is provided by Watcher. This efficacy specification is useful during the
development process of your goal as it corresponds to an empty specification.
If you want to learn more about what efficacy specifications are used for or to
define your own efficacy specification, please refer to the :ref:`related
section below <implement_efficacy_specification>`.
Abstract Plugin Class
=====================
Here below is the abstract :py:class:`~.base.Goal` class:
.. autoclass:: watcher.decision_engine.goal.base.Goal
:members:
:noindex:
.. _goal_plugin_add_entrypoint:
Add a new entry point
=====================
In order for the Watcher Decision Engine to load your new goal, the
goal must be registered as a named entry point under the ``watcher_goals``
entry point namespace of your ``setup.py`` file. If you are using pbr_, this
entry point should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique and should be the same
as the value returned by the :py:meth:`~.base.Goal.get_name` class method of
your goal.
Here below is how you would proceed to register ``NewGoal`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_goals =
new_goal = thirdparty.new:NewGoal
To get a better understanding on how to implement a more advanced goal,
have a look at the :py:class:`~.ServerConsolidation` class.
.. _pbr: http://docs.openstack.org/developer/pbr/
.. _implement_efficacy_specification:
Implement a customized efficacy specification
=============================================
What is it for?
---------------
Efficacy specifications define a set of specifications for a given goal.
These specifications actually define a list of indicators which are to be used
to compute a global efficacy that outlines how well a strategy performed when
trying to achieve the goal it is associated to.
The idea behind such specification is to give the administrator the possibility
to run an audit using different strategies satisfying the same goal and be able
to judge how they performed at a glance.
Implementation
--------------
In order to create a new efficacy specification, you have to:
- Extend the :py:class:`~.EfficacySpecification` class.
- Implement :py:meth:`~.EfficacySpecification.get_indicators_specifications`
by returning a list of :py:class:`~.IndicatorSpecification` instances.
* Each :py:class:`~.IndicatorSpecification` instance should actually extend
the latter.
* Each indicator specification should have a **unique name** which should be
a valid Python variable name.
* They should implement the :py:attr:`~.EfficacySpecification.schema`
abstract property by returning a :py:class:`~.voluptuous.Schema` instance.
This schema is the contract the strategy will have to comply with when
setting the value associated to the indicator specification within its
solution (see the :ref:`architecture of Watcher
<sequence_diagrams_create_and_launch_audit>` for more information on
the audit execution workflow).
- Implement the :py:meth:`~.EfficacySpecification.get_global_efficacy` method:
it should compute the global efficacy for the goal it achieves based on the
efficacy indicators you just defined.
Here below is an example of an efficacy specification containing one indicator
specification:
.. code-block:: python
from watcher._i18n import _
from watcher.decision_engine.goal.efficacy import base as efficacy_base
from watcher.decision_engine.goal.efficacy import indicators
from watcher.decision_engine.solution import efficacy
class IndicatorExample(IndicatorSpecification):
def __init__(self):
super(IndicatorExample, self).__init__(
name="indicator_example",
description=_("Example of indicator specification."),
unit=None,
)
@property
def schema(self):
return voluptuous.Schema(voluptuous.Range(min=0), required=True)
class UnclassifiedStrategySpecification(efficacy_base.EfficacySpecification):
def get_indicators_specifications(self):
return [IndicatorExample()]
def get_global_efficacy(self, indicators_map):
return efficacy.Indicator(
name="global_efficacy_indicator",
description="Example of global efficacy indicator",
unit="%",
value=indicators_map.indicator_example % 100)
To get a better understanding on how to implement an efficacy specification,
have a look at :py:class:`~.ServerConsolidationSpecification`.
Also, if you want to see a concrete example of an indicator specification,
have a look at :py:class:`~.ReleasedComputeNodesCount`.

11
doc/source/contributor/plugin/index.rst Normal file
View File

@@ -0,0 +1,11 @@
.. toctree::
:maxdepth: 1
base-setup
action-plugin
cdmc-plugin
goal-plugin
planner-plugin
scoring-engine-plugin
strategy-plugin
plugins

174
doc/source/contributor/plugin/planner-plugin.rst Normal file
View File

@@ -0,0 +1,174 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_planner_plugin:
===================
Build a new planner
===================
Watcher :ref:`Decision Engine <watcher_decision_engine_definition>` has an
external :ref:`planner <watcher_planner_definition>` plugin interface which
gives anyone the ability to integrate an external :ref:`planner
<watcher_planner_definition>` in order to extend the initial set of planners
Watcher provides.
This section gives some guidelines on how to implement and integrate custom
planners with Watcher.
.. _Decision Engine: watcher_decision_engine_definition
Creating a new plugin
=====================
First of all you have to extend the base :py:class:`~.BasePlanner` class which
defines an abstract method that you will have to implement. The
:py:meth:`~.BasePlanner.schedule` is the method being called by the Decision
Engine to schedule a given solution (:py:class:`~.BaseSolution`) into an
:ref:`action plan <action_plan_definition>` by ordering/sequencing an unordered
set of actions contained in the proposed solution (for more details, see
:ref:`definition of a solution <solution_definition>`).
Here is an example showing how you can write a planner plugin called
``DummyPlanner``:
.. code-block:: python
# Filepath = third-party/third_party/dummy.py
# Import path = third_party.dummy
from oslo_utils import uuidutils
from watcher.decision_engine.planner import base
class DummyPlanner(base.BasePlanner):
def _create_action_plan(self, context, audit_id):
action_plan_dict = {
'uuid': uuidutils.generate_uuid(),
'audit_id': audit_id,
'first_action_id': None,
'state': objects.action_plan.State.RECOMMENDED
}
new_action_plan = objects.ActionPlan(context, **action_plan_dict)
new_action_plan.create(context)
new_action_plan.save()
return new_action_plan
def schedule(self, context, audit_id, solution):
# Empty action plan
action_plan = self._create_action_plan(context, audit_id)
# todo: You need to create the workflow of actions here
# and attach it to the action plan
return action_plan
This implementation is the most basic one. So if you want to have more advanced
examples, have a look at the implementation of planners already provided by
Watcher like :py:class:`~.DefaultPlanner`. A list with all available planner
plugins can be found :ref:`here <watcher_planners>`.
Define configuration parameters
===============================
At this point, you have a fully functional planner. However, in more complex
implementation, you may want to define some configuration options so one can
tune the planner to its needs. To do so, you can implement the
:py:meth:`~.Loadable.get_config_opts` class method as followed:
.. code-block:: python
from oslo_config import cfg
class DummyPlanner(base.BasePlanner):
# [...]
def schedule(self, context, audit_uuid, solution):
assert self.config.test_opt == 0
# [...]
@classmethod
def get_config_opts(cls):
return super(
DummyPlanner, cls).get_config_opts() + [
cfg.StrOpt('test_opt', help="Demo Option.", default=0),
# Some more options ...
]
The configuration options defined within this class method will be included
within the global ``watcher.conf`` configuration file under a section named by
convention: ``{namespace}.{plugin_name}``. In our case, the ``watcher.conf``
configuration would have to be modified as followed:
.. code-block:: ini
[watcher_planners.dummy]
# Option used for testing.
test_opt = test_value
Then, the configuration options you define within this method will then be
injected in each instantiated object via the ``config`` parameter of the
:py:meth:`~.BasePlanner.__init__` method.
Abstract Plugin Class
=====================
Here below is the abstract ``BasePlanner`` class that every single planner
should implement:
.. autoclass:: watcher.decision_engine.planner.base.BasePlanner
:members:
:special-members: __init__
:noindex:
Register a new entry point
==========================
In order for the Watcher Decision Engine to load your new planner, the
latter must be registered as a new entry point under the
``watcher_planners`` entry point namespace of your ``setup.py`` file. If you
are using pbr_, this entry point should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique.
Here below is how you would proceed to register ``DummyPlanner`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_planners =
dummy = third_party.dummy:DummyPlanner
.. _pbr: http://docs.openstack.org/developer/pbr/
Using planner plugins
=====================
The :ref:`Watcher Decision Engine <watcher_decision_engine_definition>` service
will automatically discover any installed plugins when it is started. This
means that if Watcher is already running when you install your plugin, you will
have to restart the related Watcher services. If a Python package containing a
custom plugin is installed within the same environment as Watcher, Watcher will
automatically make that plugin available for use.
At this point, Watcher will use your new planner if you referenced it in the
``planner`` option under the ``[watcher_planner]`` section of your
``watcher.conf`` configuration file when you started it. For example, if you
want to use the ``dummy`` planner you just installed, you would have to
select it as followed:
.. code-block:: ini
[watcher_planner]
planner = dummy
As you may have noticed, only a single planner implementation can be activated
at a time, so make sure it is generic enough to support all your strategies
and actions.

76
doc/source/contributor/plugin/plugins.rst Normal file
View File

@@ -0,0 +1,76 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
=================
Available Plugins
=================
In this section we present all the plugins that are shipped along with Watcher.
If you want to know which plugins your Watcher services have access to, you can
use the :ref:`Guru Meditation Reports <watcher_gmr>` to display them.
.. _watcher_goals:
Goals
=====
.. list-plugins:: watcher_goals
:detailed:
.. _watcher_scoring_engines:
Scoring Engines
===============
.. list-plugins:: watcher_scoring_engines
:detailed:
.. _watcher_scoring_engine_containers:
Scoring Engine Containers
=========================
.. list-plugins:: watcher_scoring_engine_containers
:detailed:
.. _watcher_strategies:
Strategies
==========
.. list-plugins:: watcher_strategies
:detailed:
.. _watcher_actions:
Actions
=======
.. list-plugins:: watcher_actions
:detailed:
.. _watcher_workflow_engines:
Workflow Engines
================
.. list-plugins:: watcher_workflow_engines
:detailed:
.. _watcher_planners:
Planners
========
.. list-plugins:: watcher_planners
:detailed:
Cluster Data Model Collectors
=============================
.. list-plugins:: watcher_cluster_data_model_collectors
:detailed:

210
doc/source/contributor/plugin/scoring-engine-plugin.rst Normal file
View File

@@ -0,0 +1,210 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_scoring_engine_plugin:
==========================
Build a new scoring engine
==========================
Watcher Decision Engine has an external :ref:`scoring engine
<scoring_engine_definition>` plugin interface which gives anyone the ability
to integrate an external scoring engine in order to make use of it in a
:ref:`strategy <strategy_definition>`.
This section gives some guidelines on how to implement and integrate custom
scoring engines with Watcher. If you wish to create a third-party package for
your plugin, you can refer to our :ref:`documentation for third-party package
creation <plugin-base_setup>`.
Pre-requisites
==============
Because scoring engines execute a purely mathematical tasks, they typically do
not have any additional dependencies. Additional requirements might be defined
by specific scoring engine implementations. For example, some scoring engines
might require to prepare learning data, which has to be loaded during the
scoring engine startup. Some other might require some external services to be
available (e.g. if the scoring infrastructure is running in the cloud).
Create a new scoring engine plugin
==================================
In order to create a new scoring engine you have to:
- Extend the :py:class:`~.ScoringEngine` class
- Implement its :py:meth:`~.ScoringEngine.get_name` method to return the
**unique** ID of the new scoring engine you want to create. This unique ID
should be the same as the name of :ref:`the entry point we will declare later
on <scoring_engine_plugin_add_entrypoint>`.
- Implement its :py:meth:`~.ScoringEngine.get_description` method to return the
user-friendly description of the implemented scoring engine. It might contain
information about algorithm used, learning data etc.
- Implement its :py:meth:`~.ScoringEngine.get_metainfo` method to return the
machine-friendly metadata about this scoring engine. For example, it could be
a JSON formatted text with information about the data model used, its input
and output data format, column names, etc.
- Implement its :py:meth:`~.ScoringEngine.calculate_score` method to return the
result calculated by this scoring engine.
Here is an example showing how you can write a plugin called ``NewScorer``:
.. code-block:: python
# filepath: thirdparty/new.py
# import path: thirdparty.new
from watcher.decision_engine.scoring import base
class NewScorer(base.ScoringEngine):
def get_name(self):
return 'new_scorer'
def get_description(self):
return ''
def get_metainfo(self):
return """{
"feature_columns": [
"column1",
"column2",
"column3"],
"result_columns": [
"value",
"probability"]
}"""
def calculate_score(self, features):
return '[12, 0.83]'
As you can see in the above example, the
:py:meth:`~.ScoringEngine.calculate_score` method returns a string. Both this
class and the client (caller) should perform all the necessary serialization
or deserialization.
(Optional) Create a new scoring engine container plugin
=======================================================
Optionally, it's possible to implement a container plugin, which can return a
list of scoring engines. This list can be re-evaluated multiple times during
the lifecycle of :ref:`Watcher Decision Engine
<watcher_decision_engine_definition>` and synchronized with :ref:`Watcher
Database <watcher_database_definition>` using the ``watcher-sync`` command line
tool.
Below is an example of a container using some scoring engine implementation
that is simply made of a client responsible for communicating with a real
scoring engine deployed as a web service on external servers:
.. code-block:: python
class NewScoringContainer(base.ScoringEngineContainer):
@classmethod
def get_scoring_engine_list(self):
return [
RemoteScoringEngine(
name='scoring_engine1',
description='Some remote Scoring Engine 1',
remote_url='http://engine1.example.com/score'),
RemoteScoringEngine(
name='scoring_engine2',
description='Some remote Scoring Engine 2',
remote_url='http://engine2.example.com/score'),
]
Abstract Plugin Class
=====================
Here below is the abstract :py:class:`~.ScoringEngine` class:
.. autoclass:: watcher.decision_engine.scoring.base.ScoringEngine
:members:
:special-members: __init__
:noindex:
Abstract Plugin Container Class
===============================
Here below is the abstract :py:class:`~.ScoringContainer` class:
.. autoclass:: watcher.decision_engine.scoring.base.ScoringEngineContainer
:members:
:special-members: __init__
:noindex:
.. _scoring_engine_plugin_add_entrypoint:
Add a new entry point
=====================
In order for the Watcher Decision Engine to load your new scoring engine, it
must be registered as a named entry point under the ``watcher_scoring_engines``
entry point of your ``setup.py`` file. If you are using pbr_, this entry point
should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique and should be the same
as the value returned by the :py:meth:`~.ScoringEngine.get_name` method of your
strategy.
Here below is how you would proceed to register ``NewScorer`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_scoring_engines =
new_scorer = thirdparty.new:NewScorer
To get a better understanding on how to implement a more advanced scoring
engine, have a look at the :py:class:`~.DummyScorer` class. This implementation
is not really using machine learning, but other than that it contains all the
pieces which the "real" implementation would have.
In addition, for some use cases there is a need to register a list (possibly
dynamic, depending on the implementation and configuration) of scoring engines
in a single plugin, so there is no need to restart :ref:`Watcher Decision
Engine <watcher_decision_engine_definition>` every time such list changes. For
these cases, an additional ``watcher_scoring_engine_containers`` entry point
can be used.
For the example how to use scoring engine containers, please have a look at
the :py:class:`~.DummyScoringContainer` and the way it is configured in
``setup.cfg``. For new containers it could be done like this:
.. code-block:: ini
[entry_points]
watcher_scoring_engine_containers =
new_scoring_container = thirdparty.new:NewContainer
.. _pbr: http://docs.openstack.org/developer/pbr/
Using scoring engine plugins
============================
The Watcher Decision Engine service will automatically discover any installed
plugins when it is restarted. If a Python package containing a custom plugin is
installed within the same environment as Watcher, Watcher will automatically
make that plugin available for use.
At this point, Watcher will scan and register inside the :ref:`Watcher Database
<watcher_database_definition>` all the scoring engines you implemented upon
restarting the :ref:`Watcher Decision Engine
<watcher_decision_engine_definition>`.
In addition, ``watcher-sync`` tool can be used to trigger :ref:`Watcher
Database <watcher_database_definition>` synchronization. This might be used for
"dynamic" scoring containers, which can return different scoring engines based
on some external configuration (if they support that).

314
doc/source/contributor/plugin/strategy-plugin.rst Normal file
View File

@@ -0,0 +1,314 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
.. _implement_strategy_plugin:
=================================
Build a new optimization strategy
=================================
Watcher Decision Engine has an external :ref:`strategy <strategy_definition>`
plugin interface which gives anyone the ability to integrate an external
strategy in order to make use of placement algorithms.
This section gives some guidelines on how to implement and integrate custom
strategies with Watcher. If you wish to create a third-party package for your
plugin, you can refer to our :ref:`documentation for third-party package
creation <plugin-base_setup>`.
Pre-requisites
==============
Before using any strategy, you should make sure you have your Telemetry service
configured so that it would provide you all the metrics you need to be able to
use your strategy.
Create a new strategy plugin
============================
In order to create a new strategy, you have to:
- Extend the :py:class:`~.UnclassifiedStrategy` class
- Implement its :py:meth:`~.BaseStrategy.get_name` class method to return the
**unique** ID of the new strategy you want to create. This unique ID should
be the same as the name of :ref:`the entry point we will declare later on
<strategy_plugin_add_entrypoint>`.
- Implement its :py:meth:`~.BaseStrategy.get_display_name` class method to
return the translated display name of the strategy you want to create.
Note: Do not use a variable to return the translated string so it can be
automatically collected by the translation tool.
- Implement its :py:meth:`~.BaseStrategy.get_translatable_display_name`
class method to return the translation key (actually the English display
name) of your new strategy. The value return should be the same as the
string translated in :py:meth:`~.BaseStrategy.get_display_name`.
- Implement its :py:meth:`~.BaseStrategy.execute` method to return the
solution you computed within your strategy.
Here is an example showing how you can write a plugin called ``NewStrategy``:
.. code-block:: python
# filepath: thirdparty/new.py
# import path: thirdparty.new
import abc
import six
from watcher._i18n import _
from watcher.decision_engine.strategy.strategies import base
class NewStrategy(base.UnclassifiedStrategy):
def __init__(self, osc=None):
super(NewStrategy, self).__init__(osc)
def execute(self, original_model):
self.solution.add_action(action_type="nop",
input_parameters=parameters)
# Do some more stuff here ...
return self.solution
@classmethod
def get_name(cls):
return "new_strategy"
@classmethod
def get_display_name(cls):
return _("New strategy")
@classmethod
def get_translatable_display_name(cls):
return "New strategy"
As you can see in the above example, the :py:meth:`~.BaseStrategy.execute`
method returns a :py:class:`~.BaseSolution` instance as required. This solution
is what wraps the abstract set of actions the strategy recommends to you. This
solution is then processed by a :ref:`planner <watcher_planner_definition>` to
produce an action plan which contains the sequenced flow of actions to be
executed by the :ref:`Watcher Applier <watcher_applier_definition>`. This
solution also contains the various :ref:`efficacy indicators
<efficacy_indicator_definition>` alongside its computed :ref:`global efficacy
<efficacy_definition>`.
Please note that your strategy class will expect to find the same constructor
signature as BaseStrategy to instantiate you strategy. Therefore, you should
ensure that your ``__init__`` signature is identical to the
:py:class:`~.BaseStrategy` one.
Strategy efficacy
=================
As stated before, the ``NewStrategy`` class extends a class called
:py:class:`~.UnclassifiedStrategy`. This class actually implements a set of
abstract methods which are defined within the :py:class:`~.BaseStrategy` parent
class.
One thing this :py:class:`~.UnclassifiedStrategy` class defines is that our
``NewStrategy`` achieves the ``unclassified`` goal. This goal is a peculiar one
as it does not contain any indicator nor does it calculate a global efficacy.
This proves itself to be quite useful during the development of a new strategy
for which the goal has yet to be defined or in case a :ref:`new goal
<implement_goal_plugin>` has yet to be implemented.
Define Strategy Parameters
==========================
For each new added strategy, you can add parameters spec so that an operator
can input strategy parameters when creating an audit to control the
:py:meth:`~.BaseStrategy.execute` behavior of strategy. This is useful to
define some threshold for your strategy, and tune them at runtime.
To define parameters, just implements :py:meth:`~.BaseStrategy.get_schema` to
return parameters spec with `jsonschema
<http://json-schema.org/>`_ format.
It is strongly encouraged that provide default value for each parameter, or
else reference fails if operator specify no parameters.
Here is an example showing how you can define 2 parameters for
``DummyStrategy``:
.. code-block:: python
class DummyStrategy(base.DummyBaseStrategy):
@classmethod
def get_schema(cls):
return {
"properties": {
"para1": {
"description": "number parameter example",
"type": "number",
"default": 3.2,
"minimum": 1.0,
"maximum": 10.2,
},
"para2": {
"description": "string parameter example",
"type": "string",
"default": "hello",
},
},
}
You can reference parameters in :py:meth:`~.BaseStrategy.execute`:
.. code-block:: python
class DummyStrategy(base.DummyBaseStrategy):
def execute(self):
para1 = self.input_parameters.para1
para2 = self.input_parameters.para2
if para1 > 5:
...
Operator can specify parameters with following commands:
.. code:: bash
$ watcher audit create -a <your_audit_template> -p para1=6.0 -p para2=hi
Pls. check user-guide for details.
Abstract Plugin Class
=====================
Here below is the abstract :py:class:`~.BaseStrategy` class:
.. autoclass:: watcher.decision_engine.strategy.strategies.base.BaseStrategy
:members:
:special-members: __init__
:noindex:
.. _strategy_plugin_add_entrypoint:
Add a new entry point
=====================
In order for the Watcher Decision Engine to load your new strategy, the
strategy must be registered as a named entry point under the
``watcher_strategies`` entry point of your ``setup.py`` file. If you are using
pbr_, this entry point should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique and should be the same
as the value returned by the :py:meth:`~.BaseStrategy.get_name` class method of
your strategy.
Here below is how you would proceed to register ``NewStrategy`` using pbr_:
.. code-block:: ini
[entry_points]
watcher_strategies =
new_strategy = thirdparty.new:NewStrategy
To get a better understanding on how to implement a more advanced strategy,
have a look at the :py:class:`~.BasicConsolidation` class.
.. _pbr: http://docs.openstack.org/developer/pbr/
Using strategy plugins
======================
The Watcher Decision Engine service will automatically discover any installed
plugins when it is restarted. If a Python package containing a custom plugin is
installed within the same environment as Watcher, Watcher will automatically
make that plugin available for use.
At this point, Watcher will scan and register inside the :ref:`Watcher Database
<watcher_database_definition>` all the strategies (alongside the goals they
should satisfy) you implemented upon restarting the :ref:`Watcher Decision
Engine <watcher_decision_engine_definition>`.
You should take care when installing strategy plugins. By their very nature,
there are no guarantees that utilizing them as is will be supported, as
they may require a set of metrics which is not yet available within the
Telemetry service. In such a case, please do make sure that you first
check/configure the latter so your new strategy can be fully functional.
Querying metrics
----------------
A large set of metrics, generated by OpenStack modules, can be used in your
strategy implementation. To collect these metrics, Watcher provides a
`Helper`_ for two data sources which are `Ceilometer`_ and `Monasca`_. If you
wish to query metrics from a different data source, you can implement your own
and directly use it from within your new strategy. Indeed, strategies in
Watcher have the cluster data models decoupled from the data sources which
means that you may keep the former while changing the latter.
The recommended way for you to support a new data source is to implement a new
helper that would encapsulate within separate methods the queries you need to
perform. To then use it, you would just have to instantiate it within your
strategy.
If you want to use Ceilometer but with your own metrics database backend,
please refer to the `Ceilometer developer guide`_. The list of the available
Ceilometer backends is located here_. The `Ceilosca`_ project is a good example
of how to create your own pluggable backend. Moreover, if your strategy
requires new metrics not covered by Ceilometer, you can add them through a
`Ceilometer plugin`_.
.. _`Helper`: https://github.com/openstack/watcher/blob/master/watcher/decision_engine/cluster/history/ceilometer.py
.. _`Ceilometer developer guide`: http://docs.openstack.org/developer/ceilometer/architecture.html#storing-the-data
.. _`Ceilometer`: http://docs.openstack.org/developer/ceilometer/
.. _`Monasca`: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
.. _`here`: http://docs.openstack.org/developer/ceilometer/install/dbreco.html#choosing-a-database-backend
.. _`Ceilometer plugin`: http://docs.openstack.org/developer/ceilometer/plugins.html
.. _`Ceilosca`: https://github.com/openstack/monasca-ceilometer/blob/master/ceilosca/ceilometer/storage/impl_monasca.py
Read usage metrics using the Watcher Datasource Helper
------------------------------------------------------
The following code snippet shows how to invoke a Datasource Helper class:
.. code-block:: py
from watcher.datasource import ceilometer as ceil
from watcher.datasource import monasca as mon
@property
def ceilometer(self):
if self._ceilometer is None:
self._ceilometer = ceil.CeilometerHelper(osc=self.osc)
return self._ceilometer
@property
def monasca(self):
if self._monasca is None:
self._monasca = mon.MonascaHelper(osc=self.osc)
return self._monasca
Using that you can now query the values for that specific metric:
.. code-block:: py
if self.config.datasource == "ceilometer":
resource_id = "%s_%s" % (node.uuid, node.hostname)
return self.ceilometer.statistic_aggregation(
resource_id=resource_id,
meter_name='compute.node.cpu.percent',
period="7200",
aggregate='avg',
)
elif self.config.datasource == "monasca":
statistics = self.monasca.statistic_aggregation(
meter_name='compute.node.cpu.percent',
dimensions=dict(hostname=node.uuid),
period=7200,
aggregate='avg'
)

1
doc/source/contributor/rally_link.rst Normal file
View File

@@ -0,0 +1 @@
.. include:: ../../../rally-jobs/README.rst

50
doc/source/contributor/testing.rst Normal file
View File

@@ -0,0 +1,50 @@
..
Except where otherwise noted, this document is licensed under Creative
Commons Attribution 3.0 License. You can view the license at:
https://creativecommons.org/licenses/by/3.0/
=======
Testing
=======
.. _unit_tests:
Unit tests
==========
All unit tests should be run using `tox`_. To run the same unit tests that are
executing onto `Gerrit`_ which includes ``py35``, ``py27`` and ``pep8``, you
can issue the following command::
$ workon watcher
(watcher) $ pip install tox
(watcher) $ cd watcher
(watcher) $ tox
If you want to only run one of the aforementioned, you can then issue one of
the following::
$ workon watcher
(watcher) $ tox -e py35
(watcher) $ tox -e py27
(watcher) $ tox -e pep8
.. _tox: https://tox.readthedocs.org/
.. _Gerrit: http://review.openstack.org/
You may pass options to the test programs using positional arguments. To run a
specific unit test, you can pass extra options to `os-testr`_ after putting
the ``--`` separator. So using the ``-r`` option followed by a regex string,
you can run the desired test::
$ workon watcher
(watcher) $ tox -e py27 -- -r watcher.tests.api
.. _os-testr: http://docs.openstack.org/developer/os-testr/
When you're done, deactivate the virtualenv::
$ deactivate
.. include:: ../../../watcher_tempest_plugin/README.rst