Arnike/watcher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a checker for the documentation

Browse Source 
This patchset add some unit tests on the documentation:
Checking wrapping
Checking trailing spaces
Checking no_cr

Change-Id: I3fa56d3e7dd3218dcd398e6750bdd2fb3a8e75b4
This commit is contained in:
Jean-Emile DARTOIS
2015-12-04 17:59:21 +01:00
parent 7710b1670e
commit 022b15dc1e
5 changed files with 178 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
doc/source/architecture.rst
View File

@@ -68,8 +68,9 @@ You can also read the detailed description of `Watcher API`_.
Watcher Applier
---------------
This component is in charge of executing the :ref:`Action Plan <action_plan_definition>`
built by the :ref:`Watcher Decision Engine <watcher_decision_engine_definition>`.
This component is in charge of executing the
:ref:`Action Plan <action_plan_definition>` built by the
:ref:`Watcher Decision Engine <watcher_decision_engine_definition>`.
It connects to the :ref:`message bus <amqp_bus_definition>` and launches the
:ref:`Action Plan <action_plan_definition>` whenever a triggering message is
@@ -93,9 +94,11 @@ handling a given :ref:`Action <action_definition>` from the list, a
notification message is sent on the :ref:`message bus <amqp_bus_definition>`
indicating that the state of the action has changed to **ONGOING**.
If the :ref:`Action <action_definition>` is successful, the :ref:`Watcher Applier <watcher_applier_definition>`
sends a notification message on :ref:`the bus <amqp_bus_definition>` informing
the other components of this.
If the :ref:`Action <action_definition>` is successful,
the :ref:`Watcher Applier <watcher_applier_definition>` sends a notification
message on :ref:`the bus <amqp_bus_definition>` informing the other components
of this.
If the :ref:`Action <action_definition>` fails, the
:ref:`Watcher Applier <watcher_applier_definition>` tries to rollback to the
@@ -136,8 +139,8 @@ Watcher Decision Engine
-----------------------
This component is responsible for computing a set of potential optimization
:ref:`Actions <action_definition>` in order to fulfill the :ref:`Goal <goal_definition>`
of an :ref:`Audit <audit_definition>`.
:ref:`Actions <action_definition>` in order to fulfill
the :ref:`Goal <goal_definition>` of an :ref:`Audit <audit_definition>`.
It first reads the parameters of the :ref:`Audit <audit_definition>` from the
associated :ref:`Audit Template <audit_template_definition>` and knows the
@@ -159,10 +162,13 @@ These :ref:`Actions <action_definition>` are scheduled in time by the
In order to compute the potential :ref:`Solution <solution_definition>` for the
Audit, the :ref:`Strategy <strategy_definition>` relies on two sets of data:
- the current state of the :ref:`Managed resources <managed_resource_definition>`
(e.g., the data stored in the Nova database)
- the data stored in the :ref:`Cluster History Database <cluster_history_db_definition>`
which provides information about the past of the :ref:`Cluster <cluster_definition>`
- the current state of the
:ref:`Managed resources <managed_resource_definition>`
(e.g., the data stored in the Nova database)
- the data stored in the
:ref:`Cluster History Database <cluster_history_db_definition>`
which provides information about the past of the
:ref:`Cluster <cluster_definition>`
So far, only one :ref:`Strategy <strategy_definition>` can be associated to a
given :ref:`Goal <goal_definition>` via the main Watcher configuration file.

8
doc/source/cmds/watcher-db-manage.rst
View File

@@ -54,7 +54,8 @@ Usage
=====
Options for the various :ref:`commands <db-manage_cmds>` for
:command:`watcher-db-manage` are listed when the :option:`-h` or :option:`--help`
:command:`watcher-db-manage` are listed when the :option:`-h` or
:option:`--help`
option is used after the command.
For example::
@@ -81,8 +82,9 @@ If no configuration file is specified with the :option:`--config-file` option,
Command Options
===============
:command:`watcher-db-manage` is given a command that tells the utility what actions
to perform. These commands can take arguments. Several commands are available:
:command:`watcher-db-manage` is given a command that tells the utility
what actions to perform.
These commands can take arguments. Several commands are available:
.. _create_schema:

2
doc/source/dev/contributing.rst
View File

@@ -44,7 +44,7 @@ Bug tracker
Mailing list (prefix subjects with ``[watcher]`` for faster responses)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Wiki
http://wiki.openstack.org/Watcher

125
doc/source/dev/glossary.rst
View File

@@ -71,23 +71,27 @@ An :ref:`Action Plan <action_plan_definition>` is a flow of
a given :ref:`Goal <goal_definition>`.
An :ref:`Action Plan <action_plan_definition>` is generated by Watcher when an
:ref:`Audit <audit_definition>` is successful which implies that the :ref:`Strategy <strategy_definition>`
:ref:`Audit <audit_definition>` is successful which implies that the
:ref:`Strategy <strategy_definition>`
which was used has found a :ref:`Solution <solution_definition>` to achieve the
:ref:`Goal <goal_definition>` of this :ref:`Audit <audit_definition>`.
In the default implementation of Watcher, an :ref:`Action Plan <action_plan_definition>`
In the default implementation of Watcher, an
:ref:`Action Plan <action_plan_definition>`
is only composed of successive :ref:`Actions <action_definition>`
(i.e., a Workflow of :ref:`Actions <action_definition>` belonging to a unique
branch).
However, Watcher provides abstract interfaces for many of its components,
allowing other implementations to generate and handle more complex :ref:`Action Plan(s) <action_plan_definition>`
allowing other implementations to generate and handle more complex
:ref:`Action Plan(s) <action_plan_definition>`
composed of two types of Action Item(s):
- simple :ref:`Actions <action_definition>`: atomic tasks, which means it
can not be split into smaller tasks or commands from an OpenStack point of
view.
- composite Actions: which are composed of several simple :ref:`Actions <action_definition>`
- composite Actions: which are composed of several simple
:ref:`Actions <action_definition>`
ordered in sequential and/or parallel flows.
An :ref:`Action Plan <action_plan_definition>` may be described using
@@ -134,7 +138,8 @@ is a role for users which allows them to run any Watcher commands, such as:
- Launch an :ref:`Audit <audit_definition>`
- Get the :ref:`Action Plan <action_plan_definition>`
- Launch a recommended :ref:`Action Plan <action_plan_definition>` manually
- Archive previous :ref:`Audits <audit_definition>` and :ref:`Action Plans <action_plan_definition>`
- Archive previous :ref:`Audits <audit_definition>` and
:ref:`Action Plans <action_plan_definition>`
The :ref:`Administrator <administrator_definition>` is also allowed to modify
@@ -163,7 +168,8 @@ be one of the following:
event handling mechanism) and is in the queue for being processed by the
:ref:`Watcher Decision Engine <watcher_decision_engine_definition>`
- **ONGOING** : the :ref:`Audit <audit_definition>` is currently being
processed by the :ref:`Watcher Decision Engine <watcher_decision_engine_definition>`
processed by the
:ref:`Watcher Decision Engine <watcher_decision_engine_definition>`
- **SUCCEEDED** : the :ref:`Audit <audit_definition>` has been executed
successfully (note that it may not necessarily produce a
:ref:`Solution <solution_definition>`).
@@ -191,7 +197,8 @@ An :ref:`Audit Template <audit_template_definition>` contains at least the
It may also contain some error handling settings indicating whether:
- :ref:`Watcher Applier <watcher_applier_definition>` stops the entire operation
- :ref:`Watcher Applier <watcher_applier_definition>` stops the
entire operation
- :ref:`Watcher Applier <watcher_applier_definition>` performs a rollback
and how many retries should be attempted before failure occurs (also the latter
@@ -241,10 +248,12 @@ Cluster Data Model
==================
A :ref:`Cluster Data Model <cluster_data_model_definition>` is a logical
representation of the current state and topology of the :ref:`Cluster <cluster_definition>`
representation of the current state and topology of the
:ref:`Cluster <cluster_definition>`
:ref:`Managed resources <managed_resource_definition>`.
It is represented as a set of :ref:`Managed resources <managed_resource_definition>`
It is represented as a set of
:ref:`Managed resources <managed_resource_definition>`
(which may be a simple tree or a flat list of key-value pairs)
which enables Watcher :ref:`Strategies <strategy_definition>` to know the
current relationships between the different
@@ -253,7 +262,8 @@ current relationships between the different
and enables the :ref:`Strategy <strategy_definition>` to request information
such as:
- What compute nodes are in a given :ref:`Availability Zone <availability_zone_definition>`
- What compute nodes are in a given
:ref:`Availability Zone <availability_zone_definition>`
or a given :ref:`Host Aggregate <host_aggregates_definition>` ?
- What :ref:`Instances <instance_definition>` are hosted on a given compute
node ?
@@ -263,16 +273,19 @@ such as:
- What is the available bandwidth on a given network link ?
- What is the current space available on a given virtual disk of a given
:ref:`Instance <instance_definition>` ?
- What is the current state of a given :ref:`Instance <instance_definition>` ?
- What is the current state of a given :ref:`Instance <instance_definition>`?
- ...
In a word, this data model enables the :ref:`Strategy <strategy_definition>`
to know:
- the current topology of the :ref:`Cluster <cluster_definition>`
- the current capacity for each :ref:`Managed resource <managed_resource_definition>`
- the current amount of used/free space for each :ref:`Managed resource <managed_resource_definition>`
- the current state of each :ref:`Managed resources <managed_resource_definition>`
- the current capacity for each
:ref:`Managed resource <managed_resource_definition>`
- the current amount of used/free space for each
:ref:`Managed resource <managed_resource_definition>`
- the current state of each
:ref:`Managed resources <managed_resource_definition>`
In the Watcher project, we aim at providing a generic and very basic
:ref:`Cluster Data Model <cluster_data_model_definition>` for each
@@ -295,7 +308,8 @@ to:
:ref:`Cluster Data Model <cluster_data_model_definition>`
(the proposed data model acts as a pivot data model)
There may be various :ref:`generic and basic Cluster Data Models <cluster_data_model_definition>`
There may be various
:ref:`generic and basic Cluster Data Models <cluster_data_model_definition>`
proposed in Watcher helpers, each of them being adapted to achieving a given
:ref:`Goal <goal_definition>`:
@@ -311,8 +325,10 @@ proposed in Watcher helpers, each of them being adapted to achieving a given
Note however that a developer can use his/her own
:ref:`Cluster Data Model <cluster_data_model_definition>` if the proposed data
model does not fit his/her needs as long as the :ref:`Strategy <strategy_definition>`
is able to produce a :ref:`Solution <solution_definition>` for the requested :ref:`Goal <goal_definition>`.
model does not fit his/her needs as long as the
:ref:`Strategy <strategy_definition>` is able to produce a
:ref:`Solution <solution_definition>` for the requested
:ref:`Goal <goal_definition>`.
For example, a developer could rely on the Nova Data Model to optimize some
compute resources.
@@ -335,16 +351,21 @@ history may be used by any :ref:`Strategy <strategy_definition>` in order to
find the most optimal :ref:`Solution <solution_definition>` during an
:ref:`Audit <audit_definition>`.
In the Watcher project, a generic :ref:`Cluster History <cluster_history_definition>`
In the Watcher project, a generic
:ref:`Cluster History <cluster_history_definition>`
API is proposed with some helper classes in order to :
- share a common measurement (events or metrics) naming based on what is
defined in Ceilometer. See `the full list of available measurements <http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html>`_
defined in Ceilometer.
See `the full list of available measurements <http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html>`_
- share common meter types (Cumulative, Delta, Gauge) based on what is
defined in Ceilometer. See `the full list of meter types <http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html>`_
defined in Ceilometer.
See `the full list of meter types <http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html>`_
- simplify the development of a new :ref:`Strategy <strategy_definition>`
- avoid duplicating the same code in several :ref:`Strategies <strategy_definition>`
- have a better consistency between the different :ref:`Strategies <strategy_definition>`
- avoid duplicating the same code in several
:ref:`Strategies <strategy_definition>`
- have a better consistency between the different
:ref:`Strategies <strategy_definition>`
- avoid any strong coupling with any external metrics/events storage system
(the proposed API and measurement naming system acts as a pivot format)
@@ -369,8 +390,8 @@ services:
- Cinder scheduler: for volumes management
- Glance controller: for image management
- Neutron controller: for network management
- Nova controller: for global compute resources management with services such as
nova-scheduler, nova-conductor and nova-network
- Nova controller: for global compute resources management with services
such as nova-scheduler, nova-conductor and nova-network.
In many configurations, Watcher will reside on a controller node even if it
can potentially be hosted on a dedicated machine.
@@ -388,7 +409,8 @@ Customer
========
A :ref:`Customer <customer_definition>` is the person or company which
subscribes to the cloud provider offering. A customer may have several :ref:`Project(s) <project_definition>`
subscribes to the cloud provider offering. A customer may have several
:ref:`Project(s) <project_definition>`
hosted on the same :ref:`Cluster <cluster_definition>` or dispatched on
different clusters.
@@ -484,11 +506,12 @@ measure of how much of the :ref:`Goal <goal_definition>` has been achieved in
respect with constraints and :ref:`SLAs <sla_definition>` defined by the
:ref:`Customer <customer_definition>`.
The way efficiency is evaluated will depend on the :ref:`Goal <goal_definition>`
to achieve.
The way efficiency is evaluated will depend on the
:ref:`Goal <goal_definition>` to achieve.
Of course, the efficiency will be relevant only as long as the :ref:`Action Plan <action_plan_definition>`
is relevant (i.e., the current state of the :ref:`Cluster <cluster_definition>`
Of course, the efficiency will be relevant only as long as the
:ref:`Action Plan <action_plan_definition>` is relevant
(i.e., the current state of the :ref:`Cluster <cluster_definition>`
has not changed in a way that a new :ref:`Audit <audit_definition>` would need
to be launched).
@@ -551,21 +574,21 @@ which provides a good definition.
SLA violation
=============
A :ref:`SLA violation <sla_violation_definition>` happens when a :ref:`SLA <sla_definition>`
defined with a given :ref:`Customer <customer_definition>` could not be
respected by the cloud provider within the timeframe defined by the official
contract document.
A :ref:`SLA violation <sla_violation_definition>` happens when a
:ref:`SLA <sla_definition>` defined with a given
:ref:`Customer <customer_definition>` could not be respected by the
cloud provider within the timeframe defined by the official contract document.
.. _slo_definition:
SLO
===
A Service Level Objective (SLO) is a key element of a :ref:`SLA <sla_definition>`
between a service provider and a :ref:`Customer <customer_definition>`. SLOs
are agreed as a means of measuring the performance of the Service Provider and
are outlined as a way of avoiding disputes between the two parties based on
misunderstanding.
A Service Level Objective (SLO) is a key element of a
:ref:`SLA <sla_definition>` between a service provider and a
:ref:`Customer <customer_definition>`. SLOs are agreed as a means of measuring
the performance of the Service Provider and are outlined as a way of avoiding
disputes between the two parties based on misunderstanding.
You can also read `the Wikipedia page for SLO <https://en.wikipedia.org/wiki/Service_level_objective>`_
which provides a good definition.
@@ -575,9 +598,10 @@ which provides a good definition.
Solution
========
A :ref:`Solution <solution_definition>` is a set of :ref:`Actions <action_definition>`
generated by a :ref:`Strategy <strategy_definition>` (i.e., an algorithm) in
order to achieve the :ref:`Goal <goal_definition>` of an :ref:`Audit <audit_definition>`.
A :ref:`Solution <solution_definition>` is a set of
:ref:`Actions <action_definition>` generated by a
:ref:`Strategy <strategy_definition>` (i.e., an algorithm) in order to achieve
the :ref:`Goal <goal_definition>` of an :ref:`Audit <audit_definition>`.
A :ref:`Solution <solution_definition>` is different from an
:ref:`Action Plan <action_plan_definition>` because it contains the
@@ -593,8 +617,8 @@ applied.
Two approaches to dealing with this can be envisaged:
- **fully automated mode**: only the :ref:`Solution <solution_definition>` with
the highest ranking (i.e., the highest
- **fully automated mode**: only the :ref:`Solution <solution_definition>`
with the highest ranking (i.e., the highest
:ref:`Optimization Efficiency <efficiency_definition>`)
will be sent to the :ref:`Watcher Planner <watcher_planner_definition>` and
translated into concrete :ref:`Actions <action_definition>`.
@@ -610,11 +634,13 @@ Strategy
========
A :ref:`Strategy <strategy_definition>` is an algorithm implementation which is
able to find a :ref:`Solution <solution_definition>` for a given :ref:`Goal <goal_definition>`.
able to find a :ref:`Solution <solution_definition>` for a given
:ref:`Goal <goal_definition>`.
There may be several potential strategies which are able to achieve the same
:ref:`Goal <goal_definition>`. This is why it is possible to configure which
specific :ref:`Strategy <strategy_definition>` should be used for each :ref:`Goal <goal_definition>`.
specific :ref:`Strategy <strategy_definition>` should be used for each
:ref:`Goal <goal_definition>`.
Some strategies may provide better optimization results but may take more time
to find an optimal :ref:`Solution <solution_definition>`.
@@ -628,8 +654,9 @@ provided as well.
Watcher Applier
===============
This component is in charge of executing the :ref:`Action Plan <action_plan_definition>`
built by the :ref:`Watcher Decision Engine <watcher_decision_engine_definition>`.
This component is in charge of executing the
:ref:`Action Plan <action_plan_definition>` built by the
:ref:`Watcher Decision Engine <watcher_decision_engine_definition>`.
See :doc:`architecture` for more details on this component.
@@ -658,8 +685,8 @@ Watcher Decision Engine
=======================
This component is responsible for computing a set of potential optimization
:ref:`Actions <action_definition>` in order to fulfill the :ref:`Goal <goal_definition>`
of an :ref:`Audit <audit_definition>`.
:ref:`Actions <action_definition>` in order to fulfill the
:ref:`Goal <goal_definition>` of an :ref:`Audit <audit_definition>`.
It first reads the parameters of the :ref:`Audit <audit_definition>` from the
associated :ref:`Audit Template <audit_template_definition>` and knows the

79
watcher/tests/test_doc_formatting.py Normal file
View File

@@ -0,0 +1,79 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import re
import testtools
class TestDocFormatting(testtools.TestCase):
def _check_lines_wrapping(self, tpl, raw):
code_block = False
for i, line in enumerate(raw.split("\n"), start=1):
# NOTE(ndipanov): Allow code block lines to be longer than 79 ch
if code_block:
if not line or line.startswith(" "):
continue
else:
code_block = False
if "::" in line:
code_block = True
if "http://" in line or "https://" in line:
continue
# Allow lines which do not contain any whitespace
if re.match("\s*[^\s]+$", line):
continue
if code_block is False:
self.assertTrue(
len(line) < 80,
msg="%s:%d: Line limited to a maximum of 79 characters." %
(tpl, i))
def _check_no_cr(self, tpl, raw):
cr = '\r'
matches = re.findall(cr, raw)
self.assertEqual(
len(matches), 0,
"Found %s literal carriage returns in file %s" %
(len(matches), tpl))
def _check_trailing_spaces(self, tpl, raw):
for i, line in enumerate(raw.split("\n"), start=1):
trailing_spaces = re.findall(" +$", line)
self.assertEqual(len(trailing_spaces), 0,
"Found trailing spaces on line %s of %s" % (
i, tpl))
def _check_tab(self, tpl, raw):
tab = '\t'
matches = re.findall(tab, raw)
self.assertEqual(
len(matches), 0,
"Found %s tab in file %s" %
(len(matches), tpl))
def test_template(self):
doc_path = os.path.join("doc", 'source')
for root, dirs, files in os.walk(top=doc_path):
for file in files:
absolute_path_file = os.path.join(root, file)
if not os.path.isdir(absolute_path_file):
if not absolute_path_file.endswith(".rst"):
continue
with open(absolute_path_file) as f:
data = f.read()
self._check_tab(absolute_path_file, data)
self._check_lines_wrapping(absolute_path_file, data)
self._check_no_cr(absolute_path_file, data)
self._check_trailing_spaces(absolute_path_file, data)