Arnike/watcher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial version

Browse Source 
Change-Id: I699e0ab082657880998d8618fe29eb7f56c6c661
This commit is contained in:
David TARDIVEL
2015-06-04 15:26:55 +02:00
parent 073c6e49cb
commit d14e057da1
316 changed files with 27260 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

213
doc/source/cmds/watcher-db-manage.rst Normal file
View File

@@ -0,0 +1,213 @@
.. _watcher-db-manage:
=============
watcher-db-manage
=============
The :command:`watcher-db-manage` utility is used to create the database schema
tables that the watcher services will use for storage. It can also be used to
upgrade (or downgrade) existing database tables when migrating between
different versions of watcher.
The `Alembic library <http://alembic.readthedocs.org>`_ is used to perform
the database migrations.
Options
=======
This is a partial list of the most useful options. To see the full list,
run the following::
watcher-db-manage --help
.. program:: watcher-db-manage
.. option:: -h, --help
Show help message and exit.
.. option:: --config-dir <DIR>
Path to a config directory with configuration files.
.. option:: --config-file <PATH>
Path to a configuration file to use.
.. option:: -d, --debug
Print debugging output.
.. option:: -v, --verbose
Print more verbose output.
.. option:: --version
Show the program's version number and exit.
.. option:: upgrade, downgrade, stamp, revision, version, create_schema
The :ref:`command <db-manage_cmds>` to run.
Usage
=====
Options for the various :ref:`commands <db-manage_cmds>` for
:command:`watcher-db-manage` are listed when the :option:`-h` or :option:`--help`
option is used after the command.
For example::
watcher-db-manage create_schema --help
Information about the database is read from the watcher configuration file
used by the API server and conductor services. This file must be specified
with the :option:`--config-file` option::
watcher-db-manage --config-file /path/to/watcher.conf create_schema
The configuration file defines the database backend to use with the
*connection* database option::
[database]
connection=mysql://root@localhost/watcher
If no configuration file is specified with the :option:`--config-file` option,
:command:`watcher-db-manage` assumes an SQLite database.
.. _db-manage_cmds:
Command Options
===============
:command:`watcher-db-manage` is given a command that tells the utility what actions
to perform. These commands can take arguments. Several commands are available:
.. _create_schema:
create_schema
-------------
.. program:: create_schema
.. option:: -h, --help
Show help for create_schema and exit.
This command will create database tables based on the most current version.
It assumes that there are no existing tables.
An example of creating database tables with the most recent version::
watcher-db-manage --config-file=/etc/watcher/watcher.conf create_schema
downgrade
---------
.. program:: downgrade
.. option:: -h, --help
Show help for downgrade and exit.
.. option:: --revision <REVISION>
The revision number you want to downgrade to.
This command will revert existing database tables to a previous version.
The version can be specified with the :option:`--revision` option.
An example of downgrading to table versions at revision 2581ebaf0cb2::
watcher-db-manage --config-file=/etc/watcher/watcher.conf downgrade --revision 2581ebaf0cb2
revision
--------
.. program:: revision
.. option:: -h, --help
Show help for revision and exit.
.. option:: -m <MESSAGE>, --message <MESSAGE>
The message to use with the revision file.
.. option:: --autogenerate
Compares table metadata in the application with the status of the database
and generates migrations based on this comparison.
This command will create a new revision file. You can use the
:option:`--message` option to comment the revision.
This is really only useful for watcher developers making changes that require
database changes. This revision file is used during database migration and
will specify the changes that need to be made to the database tables. Further
discussion is beyond the scope of this document.
stamp
-----
.. program:: stamp
.. option:: -h, --help
Show help for stamp and exit.
.. option:: --revision <REVISION>
The revision number.
This command will 'stamp' the revision table with the version specified with
the :option:`--revision` option. It will not run any migrations.
upgrade
-------
.. program:: upgrade
.. option:: -h, --help
Show help for upgrade and exit.
.. option:: --revision <REVISION>
The revision number to upgrade to.
This command will upgrade existing database tables to the most recent version,
or to the version specified with the :option:`--revision` option.
If there are no existing tables, then new tables are created, beginning
with the oldest known version, and successively upgraded using all of the
database migration files, until they are at the specified version. Note
that this behavior is different from the :ref:`create_schema` command
that creates the tables based on the most recent version.
An example of upgrading to the most recent table versions::
watcher-db-manage --config-file=/etc/watcher/watcher.conf upgrade
.. note::
This command is the default if no command is given to
:command:`watcher-db-manage`.
.. warning::
The upgrade command is not compatible with SQLite databases since it uses
ALTER TABLE commands to upgrade the database tables. SQLite supports only
a limited subset of ALTER TABLE.
version
-------
.. program:: version
.. option:: -h, --help
Show help for version and exit.
This command will output the current database version.

98
doc/source/conf.py Executable file
View File

@@ -0,0 +1,98 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
from watcher import version as watcher_version
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
# 'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
'sphinxcontrib.httpdomain',
'sphinxcontrib.pecanwsme.rest',
'wsmeext.sphinxext',
'oslosphinx'
]
wsme_protocols = ['restjson']
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'watcher'
copyright = u'2015, OpenStack Foundation'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
# The full version, including alpha/beta/rc tags.
release = watcher_version.version_info.release_string()
# The short X.Y version.
version = watcher_version.version_info.version_string()
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['watcher.']
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
html_theme_options = {'incubating': True}
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
]
# Example configuration for intersphinx: refer to the Python standard library.
# intersphinx_mapping = {'http://docs.python.org/': None}

222
doc/source/deploy/installation.rst Normal file
View File

@@ -0,0 +1,222 @@
.. _installation:
========================
Development Installation
========================
Watcher development uses virtualenv to track and manage Python dependencies while in development and testing. This allows you to install all of the Python package dependencies in a virtual environment or “virtualenv”, instead of installing the packages at the system level.
Linux Systems
-------------
Install the prerequisite packages.
On Ubuntu (tested on 12.04-64 and 14.04-64)::
sudo apt-get install python-dev libssl-dev python-pip git-core libmysqlclient-dev libffi-dev
On Fedora-based distributions e.g., Fedora/RHEL/CentOS/Scientific Linux (tested on CentOS 6.5)::
sudo yum install python-virtualenv openssl-devel python-pip git gcc libffi-devel mysql-devel postgresql-devel
On openSUSE-based distributions (SLES 12, openSUSE 13.1, Factory or Tumbleweed)::
sudo zypper install gcc git libmysqlclient-devel libopenssl-devel postgresql-devel python-devel python-pip
Manually installing and using the virtualenv
--------------------------------------------
If you have `virtualenvwrapper <https://virtualenvwrapper.readthedocs.org/en/latest/install.html>`_ installed::
$ mkvirtualenv watcher
$ git clone https://git.openstack.org/openstack/stackforge/watcher
$ cd watcher && python setup.py install
$ pip install -r ./requirements.txt
To run a specific test, use a positional argument for the unit tests::
# run a specific test for Python 2.7
tox -epy27 -- tests.api
You may pass options to the test programs using positional arguments::
# run all the Python 2.7 unit tests (in parallel!)
tox -epy27 -- --parallel
To run only the pep8/flake8 syntax and style checks::
tox -epep8
Configure Identity Service for Watcher
--------------------------------------
#. Create the Watcher service user (eg ``watcher``). The service uses this to
authenticate with the Identity Service. Use the ``service`` project and
give the user the ``admin`` role::
keystone user-create --name=watcher --pass=WATCHER_PASSWORD --email=watcher@example.com
keystone user-role-add --user=watcher --tenant=service --role=admin
or
openstack user create --password WATCHER_PASSWORD --enable --email watcher@example.com watcher
openstack role add --project services --user watcher admin
#. You must register the Watcher Service with the Identity Service so that
other OpenStack services can locate it. To register the service::
keystone service-create --name=watcher --type=infra-optim \
--description="Infrastructure Optimization service"
or
openstack service create --name watcher infra-optim
#. Create the endpoints by replacing YOUR_REGION and WATCHER_API_IP with your region and your Watcher Service's API node::
keystone endpoint-create \
--service-id=the_service_id_above \
--publicurl=http://WATCHER_API_IP:9322 \
--internalurl=http://WATCHER_API_IP:9322 \
--adminurl=http://WATCHER_API_IP:9322
or
openstack endpoint create --region YOUR_REGION watcher public http://WATCHER_API_IP:9322
openstack endpoint create --region YOUR_REGION watcher admin http://WATCHER_API_IP:9322
openstack endpoint create --region YOUR_REGION watcher internal http://WATCHER_API_IP:9322
Set up the Database for Watcher
-------------------------------
The Watcher Service stores information in a database. This guide uses the
MySQL database that is used by other OpenStack services.
#. In MySQL, create an ``watcher`` database that is accessible by the
``watcher`` user. Replace WATCHER_DBPASSWORD
with the actual password::
# mysql -u root -p
mysql> CREATE DATABASE watcher CHARACTER SET utf8;
mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'localhost' \
IDENTIFIED BY 'WATCHER_DBPASSWORD';
mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'%' \
IDENTIFIED BY 'WATCHER_DBPASSWORD';
Configure the Watcher Service
=============================
The Watcher Service is configured via its configuration file. This file
is typically located at ``/etc/watcher/watcher.conf``. You can copy the file ``etc/watcher/watcher.conf.sample`` from the GIT repo to your server and update it.
Although some configuration options are mentioned here, it is recommended that
you review all the available options so that the Watcher Service is
configured for your needs.
#. The Watcher Service stores information in a database. This guide uses the
MySQL database that is used by other OpenStack services.
Configure the location of the database via the ``connection`` option. In the
following, replace WATCHER_DBPASSWORD with the password of your ``watcher``
user, and replace DB_IP with the IP address where the DB server is located::
[database]
...
# The SQLAlchemy connection string used to connect to the
# database (string value)
#connection=<None>
connection = mysql://watcher:WATCHER_DBPASSWORD@DB_IP/watcher?charset=utf8
#. Configure the Watcher Service to use the RabbitMQ message broker by
setting one or more of these options. Replace RABBIT_HOST with the
address of the RabbitMQ server.::
[DEFAULT]
...
# The RabbitMQ broker address where a single node is used
# (string value)
rabbit_host=RABBIT_HOST
# The RabbitMQ userid (string value)
#rabbit_userid=guest
# The RabbitMQ password (string value)
#rabbit_password=guest
# The RabbitMQ virtual host (string value)
#rabbit_virtual_host=/
#. Configure the Watcher Service to use these credentials with the Identity
Service. Replace IDENTITY_IP with the IP of the Identity server, and
replace WATCHER_PASSWORD with the password you chose for the ``watcher``
user in the Identity Service::
[DEFAULT]
...
# Method to use for authentication: noauth or keystone.
# (string value)
auth_strategy=keystone
...
[keystone_authtoken]
# Complete public Identity API endpoint (string value)
#auth_uri=<None>
auth_uri=http://IDENTITY_IP:5000/v3
# Complete admin Identity API endpoint. This should specify the
# unversioned root endpoint e.g. https://localhost:35357/ (string
# value)
#identity_uri = <None>
identity_uri = http://IDENTITY_IP:5000
# Keystone account username (string value)
#admin_user=<None>
admin_user=watcher
# Keystone account password (string value)
#admin_password=<None>
admin_password=WATCHER_DBPASSWORD
# Keystone service account tenant name to validate user tokens
# (string value)
#admin_tenant_name=admin
admin_tenant_name=KEYSTONE_SERVICE_PROJECT_NAME
# Directory used to cache files related to PKI tokens (string
# value)
#signing_dir=<None>
#. Create the Watcher Service database tables::
watcher-db-manage --config-file /etc/watcher/watcher.conf create_schema
#. Start the Watcher Service::
watcher-api && watcher-decision-engine && watcher-applier
===============
Important notes
===============
#. Watcher must have admin role on supervized users' projects created on your IAAS, in order to be able to migrate project's instances if required by Watcher audits:
keystone user-role-add --user=watcher --tenant=<USER_PROJECT_NAME> --role=admin
or
openstack role add --project <USER_PROJECT_NAME> --user watcher admin
#. Please check also your hypervisor configuration to handle correctly instance migration:
`OpenStack - Configure Migrations <http://docs.openstack.org/admin-guide-cloud/content/section_configuring-compute-migrations.html>`_

75
doc/source/deploy/user-guide.rst Normal file
View File

@@ -0,0 +1,75 @@
.. _user-guide:
=================================
Welcome to the Watcher User Guide
=================================
In the `architecture <https://wiki.openstack.org/wiki/WatcherArchitecture>`_ you got information about how it works.
In this guide we're going to take you through the fundamentals of using Watcher.
Getting started with Watcher
----------------------------
This guide assumes you have a working installation of Watcher. If you get "watcher: command not found" you may have to verify your installation.
Please refer to installation guide.
In order to use Watcher, you have to configure your credentials suitable for watcher command-line tools.
I you need help on a specific command, you can use "watcher help COMMAND"
Seeing what the Watcher CLI can do ?
------------------------------------
We can see all of the commands available with Watcher CLI by running the watcher binary without options.
``watcher``
How do I run an audit of my cluster ?
-------------------------------------
First, you need to create an audit template. An audit template defines an optimization goal to achieve.
This goal should be declared in the Watcher service configuration file.
``$ watcher audit-template-create my_first_audit SERVERS_CONSOLIDATION``
If you get "You must provide a username via either --os-username or via env[OS_USERNAME]" you may have to verify your credentials
Then, you can create an audit. An audit is a request for optimizing your cluster depending on the specified goal.
You can launch an audit on your cluster by referencing the audit template (i.e. the goal) that you want to use.
- Get the audit template UUID::
``$ watcher audit-template-list``
- Start an audit based on this audit template settings::
``$ watcher audit-create -a <your_audit_template_uuid>``
Watcher service will compute an Action Plan composed of a list of potential optimization actions according to the goal to achieve.
You can see all of the goals available in the Watcher service configuration file, section ``[watcher_strategies]``.
- Wait until the Watcher audit has produced a new action plan, and get it::
``$ watcher action-plan-list --audit <the_audit_uuid>``
- Have a look on the list of optimization of this new action plan::
``$ watcher action-list --action-plan <the_action_plan_uuid>``
Once you've learnt how to create an Action Plan it's time to go further by applying it to your cluster :
- Execute the action plan::
``$ watcher action-plan-start <the_action_plan_uuid>``
You can follow the states of the actions by calling periodically ``watcher action-list``
Frequently Asked Questions
--------------------------
Under specific circumstances, you may encounter the following errors :
- Why do I get a 'Unable to establish connection to ....' error message ?
You typically get this error when one of the watcher services is not running.
You can make sure every Watcher service is running by launching the following command :
``
initctl list | grep watcher
watcher-api start/running, process 33062
watcher-decision-engine start/running, process 35511
watcher-applier start/running, process 47359
``

9
doc/source/dev/architecture.rst Normal file
View File

@@ -0,0 +1,9 @@
.. _architecture:
===================
System Architecture
===================
Please go to `Wiki Watcher Architecture <https://wiki.openstack.org/wiki/WatcherArchitecture>`_
.. _API service: ../webapi/v1.html

56
doc/source/dev/contributing.rst Normal file
View File

@@ -0,0 +1,56 @@
.. _contributing:
======================
Contributing to Watcher
======================
If you're interested in contributing to the Watcher project,
the following will help get you started.
Contributor License Agreement
-----------------------------
.. index::
single: license; agreement
In order to contribute to the Watcher project, you need to have
signed OpenStack's contributor's agreement.
.. seealso::
* http://docs.openstack.org/infra/manual/developers.html
* http://wiki.openstack.org/CLA
LaunchPad Project
-----------------
Most of the tools used for OpenStack depend on a launchpad.net ID for
authentication. After signing up for a launchpad account, join the
"openstack" team to have access to the mailing list and receive
notifications of important events.
.. seealso::
* http://launchpad.net
* http://launchpad.net/watcher
* http://launchpad.net/~openstack
Project Hosting Details
-------------------------
Bug tracker
http://launchpad.net/watcher
Mailing list (prefix subjects with ``[watcher]`` for faster responses)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Wiki
http://wiki.openstack.org/Watcher
Code Hosting
https://github.com/openstack/watcher
Code Review
https://review.openstack.org/#/q/status:open+project:stackforge/watcher,n,z

60
doc/source/index.rst Normal file
View File

@@ -0,0 +1,60 @@
============================================
Welcome to Watcher's developer documentation
============================================
Introduction
============
Watcher is an OpenStack project ...
The developer documentation provided here is continually kept up-to-date based
on the latest code, and may not represent the state of the project at any
specific prior release.
Developer Guide
===============
Introduction
------------
.. toctree::
:maxdepth: 1
dev/architecture
dev/contributing
API References
--------------
.. toctree::
:maxdepth: 1
webapi/v1
Admin Guide
===========
Overview
--------
.. toctree::
:maxdepth: 1
deploy/user-guide
deploy/installation
Commands
--------
.. toctree::
:maxdepth: 1
cmds/watcher-db-manage
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

1
doc/source/readme.rst Normal file
View File

@@ -0,0 +1 @@
.. include:: ../../README.rst

7
doc/source/usage.rst Normal file
View File

@@ -0,0 +1,7 @@
========
Usage
========
To use watcher in a project::
import watcher

61
doc/source/webapi/v1.rst Normal file
View File

@@ -0,0 +1,61 @@
=====================
RESTful Web API (v1)
=====================
Audit Templates
===============
.. rest-controller:: watcher.api.controllers.v1.audit_template:AuditTemplatesController
:webprefix: /v1/audit_template
.. autotype:: watcher.api.controllers.v1.audit_template.AuditTemplateCollection
:members:
.. autotype:: watcher.api.controllers.v1.audit_template.AuditTemplate
:members:
Audits
======
.. rest-controller:: watcher.api.controllers.v1.audit:AuditsController
:webprefix: /v1/audits
.. autotype:: watcher.api.controllers.v1.audit.AuditCollection
:members:
.. autotype:: watcher.api.controllers.v1.audit.Audit
:members:
Links
=====
.. autotype:: watcher.api.controllers.link.Link
:members:
ActionPlans
===========
.. rest-controller:: watcher.api.controllers.v1.action_plan:ActionPlansController
:webprefix: /v1/action_plans
.. autotype:: watcher.api.controllers.v1.action_plan.ActionPlan
:members:
.. autotype:: watcher.api.controllers.v1.action_plan.ActionPlanCollection
:members:
Actions
=======
.. rest-controller:: watcher.api.controllers.v1.action:ActionsController
:webprefix: /v1/actions
.. autotype:: watcher.api.controllers.v1.action.ActionCollection
:members:
.. autotype:: watcher.api.controllers.v1.action.Action
:members: