From f763d6c608df56116d158f40841e77589739aef9 Mon Sep 17 00:00:00 2001 From: David TARDIVEL Date: Thu, 29 Oct 2015 10:30:54 +0100 Subject: [PATCH] refactoring documentation Change-Id: I47ac167a3d96497fe4211e8a24ec97d0d32db35c --- doc/source/deploy/configuration.rst | 265 ++++++++++++++++++ doc/source/deploy/install-devstack-docker.rst | 170 ----------- doc/source/deploy/installation.rst | 200 ++++--------- doc/source/deploy/virtualenv.rst | 222 --------------- doc/source/dev/environment.rst | 248 ++++++++++++++++ 5 files changed, 569 insertions(+), 536 deletions(-) create mode 100644 doc/source/deploy/configuration.rst delete mode 100644 doc/source/deploy/install-devstack-docker.rst delete mode 100644 doc/source/deploy/virtualenv.rst create mode 100644 doc/source/dev/environment.rst diff --git a/doc/source/deploy/configuration.rst b/doc/source/deploy/configuration.rst new file mode 100644 index 000000000..907c2b0e4 --- /dev/null +++ b/doc/source/deploy/configuration.rst @@ -0,0 +1,265 @@ +.. + +=================== +Configuring Watcher +=================== + +This document is continually updated and reflects the latest +available code of the Watcher service. + +Service overview +================ + +The Watcher service is a collection of modules that provides support to +optimize your IAAS plateform. The Watcher service may, depending +upon configuration, interact with several other OpenStack services. This +includes: + +- the OpenStack Identity service (`keystone`_) for request authentication and to + locate other OpenStack services +- the OpenStack Telemetry module (`ceilometer`_) for consuming the resources metrics +- the OpenStack Compute service (`nova`_) works with the Watcher service and acts as + a user-facing API for instance migration. + +The Watcher service includes the following components: + +- ``watcher-decision-engine``: runs audit on part of your IAAS and return an action plan in order to optimize resource placement. +- ``watcher-api``: A RESTful API that processes application requests by sending + them to the watcher-decision-engine over RPC. +- ``watcher-applier``: applies the action plan. +- `python-watcherclient`_: A command-line interface (CLI) for interacting with + the Watcher service. + +Additionally, the Bare Metal service has certain external dependencies, which are +very similar to other OpenStack services: + +- A database to store audit and action plan information and state. You can set the database + back-end type and location. +- A queue. A central hub for passing messages, such as `RabbitMQ`_. + +Optionally, one may wish to utilize the following associated projects for +additional functionality: + +- `watcher metering`_: an alternative of Ceilometer project to collect real-time metering data. + +.. _`keystone`: https://github.com/openstack/keystone +.. _`ceilometer`: https://github.com/openstack/ceilometer +.. _`nova`: https://github.com/openstack/nova +.. _`python-watcherclient`: https://github.com/openstack/python-watcherclient +.. _`watcher metering`: https://github.com/b-com/watcher-metering +.. _`RabbitMQ`: https://www.rabbitmq.com/ + +Install and configure prerequisites +=================================== + +You can configure Watcher modules to run on separate nodes or the same node. +In this guide, the components run on one node, typically the Controller node. + +This section shows you how to install and configure the modules. + +It assumes that the Identity, Image, Compute, and Networking services +have already been set up. + + + +Configure the Identity service for the Watcher service +------------------------------------------------------ + +#. Create the Watcher service user (eg ``watcher``). The service uses this to + authenticate with the Identity Service. Use the ``service`` project and + give the user the ``admin`` role: + + .. code-block:: bash + + $ keystone user-create --name=watcher --pass=WATCHER_PASSWORD --email=watcher@example.com + $ keystone user-role-add --user=watcher --tenant=service --role=admin + $ keystone user-role-add --user=watcher --tenant=admin --role=admin + + or + + .. code-block:: bash + + $ openstack user create --password WATCHER_PASSWORD --enable --email watcher@example.com watcher + $ openstack role add --project services --user watcher admin + $ openstack role add --user watcher --project admin admin + + +#. You must register the Watcher Service with the Identity Service so that + other OpenStack services can locate it. To register the service: + + .. code-block:: bash + + $ keystone service-create --name=watcher --type=infra-optim \ + --description="Infrastructure Optimization service" + + or + + .. code-block:: bash + + $ openstack service create --name watcher infra-optim + +#. Create the endpoints by replacing YOUR_REGION and WATCHER_API_IP with your region and your + Watcher Service's API node IP address (or FQDN): + + .. code-block:: bash + + $ keystone endpoint-create \ + --service-id=the_service_id_above \ + --publicurl=http://WATCHER_API_IP:9322 \ + --internalurl=http://WATCHER_API_IP:9322 \ + --adminurl=http://WATCHER_API_IP:9322 + + or + + .. code-block:: bash + + $ openstack endpoint create --region YOUR_REGION watcher public http://WATCHER_API_IP:9322 + $ openstack endpoint create --region YOUR_REGION watcher admin http://WATCHER_API_IP:9322 + $ openstack endpoint create --region YOUR_REGION watcher internal http://WATCHER_API_IP:9322 + +Set up the database for Watcher +------------------------------- + +The Watcher service stores information in a database. This guide uses the +MySQL database that is used by other OpenStack services. + +#. In MySQL, create a ``watcher`` database that is accessible by the + ``watcher`` user. Replace WATCHER_DBPASSWORD + with the actual password:: + + $ mysql -u root -p + + mysql> CREATE DATABASE watcher CHARACTER SET utf8; + mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'localhost' \ + IDENTIFIED BY 'WATCHER_DBPASSWORD'; + mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'%' \ + IDENTIFIED BY 'WATCHER_DBPASSWORD'; + + +Configure the Watcher service +============================= + +The Watcher service is configured via its configuration file. This file +is typically located at ``/etc/watcher/watcher.conf``. + +The configuration file is organized into the following sections: + +* ``[DEFAULT]`` - General configuration +* ``[api]`` - API server configuration +* ``[database]`` - SQL driver configuration +* ``[keystone_authtoken]`` - Keystone Authentication plugin configuration +* ``[watcher_applier]`` - Watcher Applier module configuration +* ``[watcher_decision_engine]`` - Watcher Decision Engine module configuration +* ``[watcher_goals]`` - Goals mapping configuration +* ``[watcher_influxdb_collector]`` - influxDB driver configuration +* ``[watcher_messaging]`` -Messaging driver configuration +* ``[watcher_metrics_collector]`` - Metric collector driver configuration +* ``[watcher_metrics_collector]`` - Metric collector driver configuration +* ``[watcher_strategies]`` - Strategy configuration + + +The Watcher configuration file is expected to be named +``watcher.conf``. When starting Watcher, you can specify a different +configuration file to use with ``--config-file``. If you do **not** specify a +configuration file, Watcher will look in the following directories for a +configuration file, in order: + +* ``~/.watcher/`` +* ``~/`` +* ``/etc/watcher/`` +* ``/etc/`` + + +Although some configuration options are mentioned here, it is recommended that +you review all the `available options `_ +so that the watcher service is configured for your needs. + +#. The Watcher Service stores information in a database. This guide uses the + MySQL database that is used by other OpenStack services. + + Configure the location of the database via the ``connection`` option. In the + following, replace WATCHER_DBPASSWORD with the password of your ``watcher`` + user, and replace DB_IP with the IP address where the DB server is located:: + + [database] + ... + + # The SQLAlchemy connection string used to connect to the + # database (string value) + #connection= + connection = mysql://watcher:WATCHER_DBPASSWORD@DB_IP/watcher?charset=utf8 + +#. Configure the Watcher Service to use the RabbitMQ message broker by + setting one or more of these options. Replace RABBIT_HOST with the + address of the RabbitMQ server.:: + + [DEFAULT] + ... + # The RabbitMQ broker address where a single node is used + # (string value) + rabbit_host=RABBIT_HOST + + # The RabbitMQ userid (string value) + #rabbit_userid=guest + + # The RabbitMQ password (string value) + #rabbit_password=guest + + # The RabbitMQ virtual host (string value) + #rabbit_virtual_host=/ + +#. Configure the Watcher Service to use these credentials with the Identity + Service. Replace IDENTITY_IP with the IP of the Identity server, and + replace WATCHER_PASSWORD with the password you chose for the ``watcher`` + user in the Identity Service:: + + [DEFAULT] + ... + # Method to use for authentication: noauth or keystone. + # (string value) + auth_strategy=keystone + + ... + [keystone_authtoken] + + # Complete public Identity API endpoint (string value) + #auth_uri= + auth_uri=http://IDENTITY_IP:5000/v3 + + # Complete admin Identity API endpoint. This should specify the + # unversioned root endpoint e.g. https://localhost:35357/ (string + # value) + #identity_uri = + identity_uri = http://IDENTITY_IP:5000 + + # Keystone account username (string value) + #admin_user= + admin_user=watcher + + # Keystone account password (string value) + #admin_password= + admin_password=WATCHER_DBPASSWORD + + # Keystone service account tenant name to validate user tokens + # (string value) + #admin_tenant_name=admin + admin_tenant_name=KEYSTONE_SERVICE_PROJECT_NAME + + # Directory used to cache files related to PKI tokens (string + # value) + #signing_dir= + +#. Create the Watcher Service database tables:: + + $ watcher-db-manage --config-file /etc/watcher/watcher.conf create_schema + +#. Start the Watcher Service:: + + $ watcher-api && watcher-decision-engine && watcher-applier + +Configure Nova compute +====================== + +Please check your hypervisor configuration to correctly handle `instance migration`_. + +.. _`instance migration`: http://docs.openstack.org/admin-guide-cloud/compute-configuring-migrations.html diff --git a/doc/source/deploy/install-devstack-docker.rst b/doc/source/deploy/install-devstack-docker.rst deleted file mode 100644 index ddf420464..000000000 --- a/doc/source/deploy/install-devstack-docker.rst +++ /dev/null @@ -1,170 +0,0 @@ -.. _install-devstack-docker: - -Watcher Installation -==================== - -To test Watcher, you can deploy it next to DevStack and run Watcher in a Docker environment. Watcher image containers integrate automatic provisioning scripts to simplify installation and configuration. You can install DevStack and Docker tools on the same machine. - -You have to be familiar with Docker and its *docker-compose* orchestration tool. - - -Install devstack ----------------- - -If you need to install DevStack, you can refer to this page : -`DevStack - an OpenStack Community Production `_ - - :: - - $ git clone clone https://git.openstack.org/openstack-dev/devstack - $ cd devstack; ./stack.sh - -Install docker & docker compose -------------------------------- - -To install Docker, you can refer to this page : `Supported installation `_ and choose your distribution. - -1. On Ubuntu simply do:: - - $ sudo apt-get install curl - $ curl -sSL https://get.docker.com | sh - -2. To test your installation:: - - $ sudo docker run hello-world - - -To install Docker compose, you can refer to this page: `Install Docker Compose `_ - - :: - - $ curl -L https://github.com/docker/compose/releases/download/1.4.0/docker-compose-)\`uname -s\`-\`uname -m\` \> /usr/local/bin/docker-compose - $ chmod +x /usr/local/bin/docker-compose - - -Watcher containers ------------------- - -Containers orchestration templating is composed of 3 groups of containers, each group are independent: - -- SERVICE_DISCOVERY : containers providing an automatic service discovery tool, based on - - `consul `_ - - `registrator `_ - -- WATCHER : containers providing a basic OpenStack identity Service and the Watcher modules (including Horizon plugin) - - mariadb, - - rabbitmq, - - keystone, - - console, - - watcher-api, - - watcher-decision-engine, - - watcher-applier, - - horizon - -- METERING: containers providing a complete metering chain for Watcher. - - riemann, - - influxdb, - - nanoconfig-server, - - watcher-metering-publisher, - - watcher-metering-agent, - - grafana - - -Install the Service Discovery tool ----------------------------------- - -1. Upload the Watcher orchestration template files:: - - $ git clone https://github.com/b-com/watcher-tools.git - - -2. Deploy the containers:: - - $ cd docker/service-discovery - $ docker-compose up -d - -Install the Watcher module --------------------------- - -1. Customize the template file *docker-compose.yml*: - - comment the *keystone* service (you will use the DevStack one) - - comment the *mariadb* service if you want to use the DevStack db - - comment the *rabbitmq* service if you want to use the DevStack RabbitMQ service - - comment the *horizon* service if you don't want to use Horizon with the Watcher UI plugin - -2. For all non commented services, don't forget to set environment variables in the template file *docker-compose.yml*: - - - DevStack modules settings: - - - **KEYSTONE_NODE**: DevStack Keystone IP address - - **KEYSTONE_ADMIN_TOKEN**: DevStack Keystone admin token (default is *ADMIN_TOKEN*) - - **KEYSTONE_SERVICE_TENANT_NAME**: DevStack service project name (default is *service*) - - **MARIADB_NODE**: DevStack MySQL server IP address, if used. - - **RABBITMQ_NODE**: DevStack RabbitMQ server IP address, if used - - - Watcher modules: - - - **DEBUG_MODE**: Debug mode (true|false, default is *false*) - - **VERBOSE_MODE**: Verbose mode (true|false, default is *false*) - - - Watcher console: - - - **WATCHER_API_URL**: Watcher API endpoint (default is *http://watcher.service.watcher.b-com.com:9322*) - - **EXIT_AFTER_INIT**: Watcher API endpoint (default is *http://watcher.service.watcher.b-com.com:9322*) - - - *admin* user's credentials : - - - **OS_AUTH_URL**: DevStack Keystone Identity Auth URL (default is *http://keystone.service.watcher.b-com.com:5000/v3*) - - **OS_IDENTITY_API_VERSION**: DevStack Keystone Identity API Version (2.0|3, default is *3*) - - **OS_PROJECT_DOMAIN_ID**: DevStack Project Domain ID (default is *default*) - - **OS_USER_DOMAIN_ID**: DevStack User Domain ID (default is *default*) - - **OS_USERNAME**: DevStack User name (default is *admin*) - - **OS_PASSWORD**: DevStack User password (default is *1234*) - - **OS_PROJECT_NAME**: DevStack Project name (default is *admin*) - - -3. Deploy the containers:: - - $ cd docker/watcher - $ docker-compose up -d - -At startup the *console* container will automatically declare the watcher service into the DevStack Keystone Identity Service. - - -4. Deploy the metering chain containers:: - - $ cd docker/metering - $ docker-compose up -d - -Note: you can deploy complete metering chain for development purpose within containers, but reminder that the Watcher Metering Agent should be deployed on real Compute Node to collect real metric data. - -Run Watcher commands --------------------- - -You can use the *console* container to play with Watcher. - -1. Connect to the Watcher console container:: - - $ docker ps - - Get the *console* container ID and attach to it:: - - $ docker exec -it bash - - :: - - root@console:/# cd /root - root@console:/# source creds - root@console:~# watcher audit-list - +------+------+---------------------+-------+ - | UUID | Type | Audit Template Name | State | - +------+------+---------------------+-------+ - +------+------+---------------------+-------+ - - -Important notes -=============== - -Please check your Compute's hypervisor configuration to handle correctly instance migration: - - `OpenStack - Configure Migrations `_ \ No newline at end of file diff --git a/doc/source/deploy/installation.rst b/doc/source/deploy/installation.rst index 9d30ff653..2dc58573d 100644 --- a/doc/source/deploy/installation.rst +++ b/doc/source/deploy/installation.rst @@ -1,184 +1,96 @@ -.. _installation: - -Quick installation +================== +Installing Watcher ================== -#. Install the prerequisite packages. +This document describes how to install Watcher in order to use it. If you are +intending to develop on or with Watcher, please read :doc:`../dev/environment`. - On Ubuntu (tested on 14.04-64):: - - sudo apt-get install python-dev libssl-dev python-pip git-core libmysqlclient-dev libffi-dev - - On Fedora-based distributions e.g., Fedora/RHEL/CentOS/Scientific Linux (tested on CentOS 6.5):: - - sudo yum install python-virtualenv openssl-devel python-pip git gcc libffi-devel mysql-devel postgresql-devel - - On openSUSE-based distributions (SLES 12, openSUSE 13.1, Factory or Tumbleweed):: - - sudo zypper install gcc git libmysqlclient-devel libopenssl-devel postgresql-devel python-devel python-pip +Prerequisites +------------- -#. Install watcher modules from pip:: +The source install instructions specifically avoid using platform specific +packages, instead using the source for the code and the Python Package Index +(PyPi_). - sudo pip install python-watcher +.. _PyPi: http://pypi.python.org/pypi -Configure Watcher -================= +It's expected that your system already has python2.7_, latest version of pip_, and git_ available. -Configure Identity Service for Watcher --------------------------------------- +.. _python2.7: http://www.python.org +.. _pip: http://www.pip-installer.org/en/latest/installing.html +.. _git: http://git-scm.com/ -#. Create the Watcher service user (eg ``watcher``) and set a password. The service uses it to - authenticate with the Identity Service. Set KEYSTONE_SERVICE_PROJECT_NAME, your Keystone `service project name`_ and - give the user the ``admin`` role:: +Your system shall also have some additional system libraries: - keystone user-create --name=watcher --pass=WATCHER_PASSWORD --email=watcher@example.com - keystone user-role-add --user=watcher --tenant=KEYSTONE_SERVICE_PROJECT_NAME --role=admin + On Ubuntu (tested on 14.04LTS): - or:: + .. code-block:: bash - openstack user create --password WATCHER_PASSWORD --enable --email watcher@example.com watcher - openstack role add --project services --user watcher admin + $ sudo apt-get install python-dev libssl-dev libmysqlclient-dev libffi-dev -#. You must register the Watcher Service with the Identity Service so that - other OpenStack services can locate it. To register the service:: + On Fedora-based distributions e.g., Fedora/RHEL/CentOS/Scientific Linux (tested on CentOS 7.1): - keystone service-create --name=watcher --type=infra-optim --description="Infrastructure Optimization service" - - or:: - - openstack service create --name watcher infra-optim + .. code-block:: bash -#. Create the endpoints by replacing YOUR_REGION and WATCHER_API_IP with your region and your Watcher Service's API node URLs:: - - keystone endpoint-create \ - --service-id=the_service_id_above \ - --publicurl=http://WATCHER_API_IP:9322 \ - --internalurl=http://WATCHER_API_IP:9322 \ - --adminurl=http://WATCHER_API_IP:9322 - - or:: - - openstack endpoint create --region YOUR_REGION watcher public http://WATCHER_API_IP:9322 - openstack endpoint create --region YOUR_REGION watcher admin http://WATCHER_API_IP:9322 - openstack endpoint create --region YOUR_REGION watcher internal http://WATCHER_API_IP:9322 - -.. _`service project name`: http://docs.openstack.org/developer/keystone/configuringservices.html - -Set up the Database for Watcher -------------------------------- - -The Watcher Service stores information in a database. This guide uses the -MySQL database that is used by other OpenStack services. - -#. In MySQL, create a ``watcher`` database that is accessible by the - ``watcher`` user. Replace WATCHER_DBPASSWORD - with the real password:: - - # mysql -u root -p - mysql> CREATE DATABASE watcher CHARACTER SET utf8; - mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'localhost' \ - IDENTIFIED BY 'WATCHER_DBPASSWORD'; - mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'%' \ - IDENTIFIED BY 'WATCHER_DBPASSWORD'; + $ sudo yum install gcc python-devel openssl-devel libffi-devel mysql-devel -Configure the Watcher Service ------------------------------ +Installing from Source +---------------------- -The Watcher Service is configured via its configuration file. This file -is typically located at ``/etc/watcher/watcher.conf``. You can copy the file ``etc/watcher/watcher.conf.sample`` from the GIT repo to your server and update it. +Clone the Watcher repository: -Although some configuration options are mentioned here, it is recommended that -you review all the available options so that the Watcher Service is -configured for your needs. +.. code-block:: bash -#. The Watcher Service stores information in a database. This guide uses the - MySQL database that is used by other OpenStack services. + $ git clone https://git.openstack.org/openstack/watcher.git + $ cd watcher - Configure the location of the database via the ``connection`` option. In the - following, replace WATCHER_DBPASSWORD with the password of your ``watcher`` - user, and replace DB_IP with the IP address where the DB server is located:: +Install the Watcher modules: - [database] - ... +.. code-block:: bash - # The SQLAlchemy connection string used to connect to the - # database (string value) - #connection= - connection = mysql://watcher:WATCHER_DBPASSWORD@DB_IP/watcher?charset=utf8 + # python setup.py install -#. Configure the Watcher Service to use the RabbitMQ message broker by - setting one or more of these options. Replace RABBIT_HOST with the - address of the RabbitMQ server:: +The following commands should be available on the command-line path: - [DEFAULT] - ... - # The RabbitMQ broker address where a single node is used - # (string value) - rabbit_host=RABBIT_HOST +* ``watcher-api`` the Watcher Web service used to handle RESTful requests +* ``watcher-decision-engine`` the Watcher Decision Engine used to build action plans, according to optimization goals to achieve. +* ``watcher-applier`` the Watcher Applier module, used to apply action plan +* ``watcher-db-manage`` used to bootstrap Watcher data - # The RabbitMQ userid (string value) - #rabbit_userid=guest +You will find sample configuration files in ``etc/``: - # The RabbitMQ password (string value) - #rabbit_password=guest +* ``watcher.conf.sample`` - # The RabbitMQ virtual host (string value) - #rabbit_virtual_host=/ +Install the Watcher modules dependencies: -#. Configure the Watcher Service to use these credentials with the Identity Service. +.. code-block:: bash - Replace IDENTITY_IP with the address of the Keystone Identity server, KEYSTONE_SERVICE_PROJECT_NAME by the Keystone service project name, and WATCHER_PASSWORD with the password you chose for the ``watcher`` - user in the Identity Service:: + # pip install -r requirements.txt - [DEFAULT] - ... - # Method to use for authentication: noauth or keystone. - # (string value) - auth_strategy=keystone +From here, refer to :doc:`configuration` to declare Watcher as a new service into Keystone and to configure its different modules. Once configured, you should be able to run Watcher by issuing these commands: - ... - [keystone_authtoken] +.. code-block:: bash - # Complete public Identity API endpoint (string value) - #auth_uri= - auth_uri=http://IDENTITY_IP:5000/v3 + $ watcher-api + $ watcher-decision-engine + $ watcher-applier - # Complete admin Identity API endpoint. This should specify the - # unversioned root endpoint e.g. https://localhost:35357/ (string - # value) - #identity_uri = - identity_uri = http://IDENTITY_IP:5000 +By default, this will show logging on the console from which it was started. +Once started, you can use the `Watcher Client`_ to play with Watcher service. - # Keystone account username (string value) - #admin_user= - admin_user=watcher +.. _`Watcher Client`: https://git.openstack.org/openstack/python-watcherclient.git - # Keystone account password (string value) - #admin_password= - admin_password=WATCHER_PASSWORD +Installing from packages: PyPI +-------------------------------- - # Keystone service account tenant name to validate user tokens - # (string value) - #admin_tenant_name=admin - admin_tenant_name=KEYSTONE_SERVICE_PROJECT_NAME +Watcher package is available on PyPI repository. To install Watcher on your system +: - # Directory used to cache files related to PKI tokens (string - # value) - #signing_dir= +.. code-block:: bash + $ sudo pip install python-watcher -#. Create the Watcher Service database tables:: +Watcher modules and its dependencies will be automatically installed on your system. - watcher-db-manage --config-file /etc/watcher/watcher.conf create_schema - -Development installation -======================== - -We propose different ways to quickly install Watcher for development: - -* `DevStack and Docker`_ -* `Virtualenv`_ - -.. _DevStack and Docker: ./install-devstack-docker.rst -.. _Virtualenv: ./virtualenv.rst +Once installed, you still need to declare Watcher as a new service into Keystone and to configure its different modules, which you can find described in :doc:`configuration`. diff --git a/doc/source/deploy/virtualenv.rst b/doc/source/deploy/virtualenv.rst deleted file mode 100644 index b40e19382..000000000 --- a/doc/source/deploy/virtualenv.rst +++ /dev/null @@ -1,222 +0,0 @@ -.. _virtualenv: - -======================== -Development Installation -======================== - -Watcher development uses virtualenv to track and manage Python dependencies while in development and testing. This allows you to install all of the Python package dependencies in a virtual environment or “virtualenv”, instead of installing the packages at the system level. - - -Linux Systems -------------- - -Install the prerequisite packages. - -On Ubuntu (tested on 12.04-64 and 14.04-64):: - - sudo apt-get install python-dev libssl-dev python-pip git-core libmysqlclient-dev libffi-dev - -On Fedora-based distributions e.g., Fedora/RHEL/CentOS/Scientific Linux (tested on CentOS 6.5):: - - sudo yum install python-virtualenv openssl-devel python-pip git gcc libffi-devel mysql-devel postgresql-devel - -On openSUSE-based distributions (SLES 12, openSUSE 13.1, Factory or Tumbleweed):: - - sudo zypper install gcc git libmysqlclient-devel libopenssl-devel postgresql-devel python-devel python-pip - - - Manually installing and using the virtualenv - -------------------------------------------- - - If you have `virtualenvwrapper `_ installed:: - - $ mkvirtualenv watcher - $ git clone https://git.openstack.org/openstack/stackforge/watcher - $ cd watcher && python setup.py install - $ pip install -r ./requirements.txt - - To run a specific test, use a positional argument for the unit tests:: - - # run a specific test for Python 2.7 - tox -epy27 -- tests.api - - You may pass options to the test programs using positional arguments:: - - # run all the Python 2.7 unit tests (in parallel!) - tox -epy27 -- --parallel - - To run only the pep8/flake8 syntax and style checks:: - - tox -epep8 - - -Configure Identity Service for Watcher --------------------------------------- - -#. Create the Watcher service user (eg ``watcher``). The service uses this to - authenticate with the Identity Service. Use the ``service`` project and - give the user the ``admin`` role:: - - keystone user-create --name=watcher --pass=WATCHER_PASSWORD --email=watcher@example.com - keystone user-role-add --user=watcher --tenant=service --role=admin - - or - - openstack user create --password WATCHER_PASSWORD --enable --email watcher@example.com watcher - openstack role add --project services --user watcher admin - - -#. You must register the Watcher Service with the Identity Service so that - other OpenStack services can locate it. To register the service:: - - keystone service-create --name=watcher --type=infra-optim \ - --description="Infrastructure Optimization service" - - or - - openstack service create --name watcher infra-optim - -#. Create the endpoints by replacing YOUR_REGION and WATCHER_API_IP with your region and your Watcher Service's API node:: - - keystone endpoint-create \ - --service-id=the_service_id_above \ - --publicurl=http://WATCHER_API_IP:9322 \ - --internalurl=http://WATCHER_API_IP:9322 \ - --adminurl=http://WATCHER_API_IP:9322 - - or - - openstack endpoint create --region YOUR_REGION watcher public http://WATCHER_API_IP:9322 - openstack endpoint create --region YOUR_REGION watcher admin http://WATCHER_API_IP:9322 - openstack endpoint create --region YOUR_REGION watcher internal http://WATCHER_API_IP:9322 - - - -Set up the Database for Watcher -------------------------------- - -The Watcher Service stores information in a database. This guide uses the -MySQL database that is used by other OpenStack services. - -#. In MySQL, create an ``watcher`` database that is accessible by the - ``watcher`` user. Replace WATCHER_DBPASSWORD - with the actual password:: - - # mysql -u root -p - mysql> CREATE DATABASE watcher CHARACTER SET utf8; - mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'localhost' \ - IDENTIFIED BY 'WATCHER_DBPASSWORD'; - mysql> GRANT ALL PRIVILEGES ON watcher.* TO 'watcher'@'%' \ - IDENTIFIED BY 'WATCHER_DBPASSWORD'; - - -Configure the Watcher Service -============================= - -The Watcher Service is configured via its configuration file. This file -is typically located at ``/etc/watcher/watcher.conf``. You can copy the file ``etc/watcher/watcher.conf.sample`` from the GIT repo to your server and update it. - -Although some configuration options are mentioned here, it is recommended that -you review all the available options so that the Watcher Service is -configured for your needs. - -#. The Watcher Service stores information in a database. This guide uses the - MySQL database that is used by other OpenStack services. - - Configure the location of the database via the ``connection`` option. In the - following, replace WATCHER_DBPASSWORD with the password of your ``watcher`` - user, and replace DB_IP with the IP address where the DB server is located:: - - [database] - ... - - # The SQLAlchemy connection string used to connect to the - # database (string value) - #connection= - connection = mysql://watcher:WATCHER_DBPASSWORD@DB_IP/watcher?charset=utf8 - -#. Configure the Watcher Service to use the RabbitMQ message broker by - setting one or more of these options. Replace RABBIT_HOST with the - address of the RabbitMQ server.:: - - [DEFAULT] - ... - # The RabbitMQ broker address where a single node is used - # (string value) - rabbit_host=RABBIT_HOST - - # The RabbitMQ userid (string value) - #rabbit_userid=guest - - # The RabbitMQ password (string value) - #rabbit_password=guest - - # The RabbitMQ virtual host (string value) - #rabbit_virtual_host=/ - -#. Configure the Watcher Service to use these credentials with the Identity - Service. Replace IDENTITY_IP with the IP of the Identity server, and - replace WATCHER_PASSWORD with the password you chose for the ``watcher`` - user in the Identity Service:: - - [DEFAULT] - ... - # Method to use for authentication: noauth or keystone. - # (string value) - auth_strategy=keystone - - ... - [keystone_authtoken] - - # Complete public Identity API endpoint (string value) - #auth_uri= - auth_uri=http://IDENTITY_IP:5000/v3 - - # Complete admin Identity API endpoint. This should specify the - # unversioned root endpoint e.g. https://localhost:35357/ (string - # value) - #identity_uri = - identity_uri = http://IDENTITY_IP:5000 - - # Keystone account username (string value) - #admin_user= - admin_user=watcher - - # Keystone account password (string value) - #admin_password= - admin_password=WATCHER_DBPASSWORD - - # Keystone service account tenant name to validate user tokens - # (string value) - #admin_tenant_name=admin - admin_tenant_name=KEYSTONE_SERVICE_PROJECT_NAME - - # Directory used to cache files related to PKI tokens (string - # value) - #signing_dir= - - -#. Create the Watcher Service database tables:: - - watcher-db-manage --config-file /etc/watcher/watcher.conf create_schema - -#. Start the Watcher Service:: - - watcher-api && watcher-decision-engine && watcher-applier - -=============== -Important notes -=============== - - -#. Watcher must have admin role on supervized users' projects created on your IAAS, in order to be able to migrate project's instances if required by Watcher audits: - - keystone user-role-add --user=watcher --tenant= --role=admin - - or - - openstack role add --project --user watcher admin - -#. Please check also your hypervisor configuration to handle correctly instance migration: - - `OpenStack - Configure Migrations `_ \ No newline at end of file diff --git a/doc/source/dev/environment.rst b/doc/source/dev/environment.rst new file mode 100644 index 000000000..ae90f8ad4 --- /dev/null +++ b/doc/source/dev/environment.rst @@ -0,0 +1,248 @@ +.. + +============================================ +Setting up a Watcher development environment +============================================ + +This document describes getting the source from watcher `Git repository`_ +for development purposes. + +To install Watcher from packaging, refer instead to Watcher `User +Documentation`_. + +.. _`Git Repository`: http://git.openstack.org/cgit/openstack/watcher +.. _`User Documentation`: https://factory.b-com.com/www/watcher/doc/watcher/ + +Prerequisites +============= + +This document assumes you are using Ubuntu or Fedora, and that you have the following tools available on your system: + +- Python_ 2.7 and 3.4 +- git_ +- setuptools_ +- pip_ +- msgfmt (part of the gettext package) +- virtualenv and virtualenvwrapper_ + +**Reminder**: If you're successfully using a different platform, or a +different version of the above, please document your configuration here! + +.. _Python: http://www.python.org/ +.. _git: http://git-scm.com/ +.. _setuptools: http://pypi.python.org/pypi/setuptools +.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.org/en/latest/install.html + +Getting the latest code +======================= + +Make a clone of the code from our `Git repository`: + +.. code-block:: bash + + $ git clone https://git.openstack.org/openstack/watcher.git + +When that is complete, you can: + +.. code-block:: bash + + $ cd watcher + +Installing dependencies +======================= + +Watcher maintains two lists of dependencies:: + + requirements.txt + test-requirements.txt + +The first is the list of dependencies needed for running Watcher, the second list includes dependencies used for active development and testing of Watcher itself. + +These dependencies can be installed from PyPi_ using the Python tool pip_. + +.. _PyPi: http://pypi.python.org/ +.. _pip: http://pypi.python.org/pypi/pip + +However, your system *may* need additional dependencies that `pip` (and by +extension, PyPi) cannot satisfy. These dependencies should be installed +prior to using `pip`, and the installation method may vary depending on +your platform. + +* Ubuntu 14.04: + + .. code-block:: bash + + $ sudo apt-get install python-dev libssl-dev libmysqlclient-dev libffi-dev + +* Fedora 19+: + + .. code-block:: bash + + $ sudo yum install openssl-devel libffi-devel mysql-devel + + +PyPi Packages and VirtualEnv +---------------------------- + +We recommend establishing a virtualenv to run Watcher within. virtualenv +limits the Python environment to just what you're installing as dependencies, +useful to keep a clean environment for working on Watcher. + +.. code-block:: bash + + $ mkvirtualenv watcher + $ git clone https://git.openstack.org/openstack/watcher + + # Use 'python setup.py' to link Watcher into Python's site-packages + $ cd watcher && python setup.py install + + # Install the dependencies for running Watcher + $ pip install -r ./requirements.txt + + # Install the dependencies for developing, testing, and running Watcher + $ pip install -r ./test-requirements.txt + +This will create a local virtual environment in the directory ``$WORKON_HOME``. +The virtual environment can be disabled using the command: + +.. code-block:: bash + + $ deactivate + +You can re-activate this virtualenv for your current shell using: + +.. code-block:: bash + + $ workon watcher + +For more information on virtual environments, see virtualenv_. + +.. _virtualenv: http://www.virtualenv.org/ + + + +Verifying Watcher is set up +=========================== + +Once set up, either directly or within a virtualenv, you should be able to +invoke Python and import the libraries. If you're using a virtualenv, don't +forget to activate it: + +.. code-block:: bash + + $ workon watcher + +You should then be able to `import watcher` using Python without issue: + +.. code-block:: bash + + $ python -c "import watcher" + +If you can import watcher without a traceback, you should be ready to develop. + +Run Watcher unit tests +====================== +All unit tests should be run using tox. To run the unit tests under py27 and also run the pep8 tests: + +.. code-block:: bash + + $ workon watcher + (watcher) $ pip install tox + + (watcher) $ cd watcher + (watcher) $ tox -epep8 -epy27 + +You may pass options to the test programs using positional arguments. To run a specific unit test, this passes the -r option and desired test (regex string) to os-testr: + +.. code-block:: bash + + $ workon watcher + (watcher) $ tox -epy27 -- tests.api + +When you're done, deactivate the virtualenv: + +.. code-block:: bash + + $ deactivate + +Build the Watcher documentation +=============================== +you can easily build the HTML documentation from ``doc/source`` files, by using tox: + +.. code-block:: bash + + $ workon watcher + + (watcher) $ cd watcher + (watcher) $ tox -edocs + +The HTML files are available into ``doc/build`` directory. + + +Configure the Watcher modules +============================= +Watcher modules requires a configuration file. There is a sample configuration file that can be used to get started: + +.. code-block:: bash + + $ cp etc/watcher.conf.sample etc/watcher.conf + +The defaults are enough to get you going, but you can make any changes if needed. + +Create Watcher SQL database +=========================== +When initially getting set up, after you've configured which databases to use, you're probably going to need to run the following to your database schema in place: + +.. code-block:: bash + + $ workon watcher + + (watcher) $ watcher-db-manage --create_schema + +Running Watcher modules +======================= +To run Watcher API server instance, use: + +.. code-block:: bash + + $ workon watcher + + (watcher) $ watcher-api + +To run Watcher Decision Engine instance, use: + +.. code-block:: bash + + $ workon watcher + + (watcher) $ watcher-decision-engine + +To run Watcher Applier instance, use: + +.. code-block:: bash + + $ workon watcher + + (watcher) $ watcher-applier + +Default configuration of these modules are available into ``/etc/watcher`` directory. See :doc:`../deploy/configuration` for details on how Watcher is configured. By default, Watcher is configured with SQL backends. + + +Interact with Watcher +===================== + +You can also interact with Watcher through its REST API. There is a Python Watcher client library `python-watcherclient`_ which interacts exclusively through the REST API, and which Watcher itself uses to provide its command-line interface. + +When initially getting set up, after you've configured which databases to use, you're probably going to need to run the following to your database schema in place: + +.. _`python-watcherclient`: https://github.com/openstack/python-watcherclient + +Exercising the Watcher Services Locally +======================================= + +If you would like to exercise the Watcher services in isolation within a local virtual environment, you can do this without starting any other OpenStack services. For example, this is useful for rapidly prototyping and debugging interactions over the RPC channel, testing database migrations, and so forth. + +You will find in the `watcher-tools`_ project, Ansible playbooks and Docker template files to easily play with Watcher modules within a minimal OpenStack isolated environment (Identity, Message Bus, SQL database, Horizon, ...). + +.. _`watcher-tools`: https://github.com/b-com/watcher-tools +