RSserver module diagnostics

Important

All commands are executed only from superuser.

Mode superuser:

sudo -i

Services logging used by RSserver module

Logging happens using logging module.

By default, the file logs is located in the directory /var/log/aos/rs-server/.

The directory contains the following files:

  • api.log is aos-rs-server-api service message collection file;

  • broker-api.log is aos-rs-broker-api service message collection file;

  • keystone-listener.log is aos-rs-keystone-listener service message collection file;

  • project-syncer.log is project sync message collection file;

  • nova-listener.log is instances sync message collection file between Nova service database and RSserver database;

  • worker.log is file fr collecting messages of asynchronous module tasks;

  • trs_requests.log is file for collecting messages about user attempts to log in and get TRS instance via TRS web interface or RSclient. Both successful and unsuccessful attempts are logged. Enabling or disabling the logging process is determined by the parameter allow_getvm_log in the section broker in the configuration file rs_server.conf. Detailed description of the allow_getvm_log parameter is available in the section Configuration file. The file entries have the following format:

    [ system time of RSserver ] - request id - connection source (web/cli) - username (which attempts to connect) - status of obtaining IP address of the TRS instance (OK/Error and also error details) - ID of groups to which the user belongs - connection project - user's IP address

  • profiler.log is message collection file with API request IDs and aos-rs-worker at each stage:

    • receive request;
    • task creation;
    • task launch;
    • mutex waiting;
    • mutex starting;
    • sending a request to Nova to create instance;
    • receiving a response (profiling the receiving process instance).

    The total waiting of mutex time is also logged for each request. Messages are written to this file only in the DEBUG mode.

Note

Description of logging process provided by the OpenStack platform can be found in the corresponding section of the official documentation.

RSserver module debugging

In case of problems in operation of the module, there are following solutions:

  1. Set log level to value DEBUG, which will allow logging messages about events in log files with maximum detail for diagnostics and problem solving.

  2. Run openstack aos tests module self-diagnosis utility.

    Usage example:

    openstack aos tests --names rs_server --modules
Modules diagnostic started.
Running tests for: rs_server

+--------------------------+-----------------------------------------------------------------------------------------------+
| Start date and time      | Thu Dec 24 17:03:53 2020 GMT                                                                  |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Inactive services        | apache-htcacheclean, nova-novncproxy, nova-serialproxy, nova-xenvncproxy, rtslib-fb-targetctl |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Database names           | Failed to get database names. Please run command as root.                                     |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Rabbit users             | Failed to get RabbitMQ info. Please run command as root.                                      |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Rabbit virtual hosts     | Failed to get RabbitMQ info. Please run command as root.                                      |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Rabbit limits            | Failed to get RabbitMQ info. Please run command as root.                                      |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Apache config parameters | Failed to get Apache configs info. Please run command as root.                                |
+--------------------------+-----------------------------------------------------------------------------------------------+
| Apache MPM               | worker                                                                                        |
+--------------------------+-----------------------------------------------------------------------------------------------+


================
RSserver
Version: 2.2.0 (latest: 2.3.0)
License: 07-002-972fb12437f60c4a5411 (valid until 15.02.2021 03:00:00)
================
+--+----------------------------------------+--------+--------------------------------------------------------------+
|N |Test name                               | Status | Failure reason                                               |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|1 |test apache config enabled              |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|2 |test apache config existence            |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|3 |test config file existence              |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|4 |test connection to keystone             |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|5 |test connection to nova                 |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|6 |test license validity                   |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|7 |test migrations applied                 |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|8 |test profiler log file accesses         |SKIPPED |The test is relevant only if DEBUG is specified in LOG_LEVEL. |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|9 |test registered celery tasks            |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|10|test registered tasks to schedule       |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|11|test request vm log file accesses       |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|12|test rs api                             |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|13|test rs server api connection           |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|14|test rs server api service registration |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|15|test rs server systemd services         |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
|16|test rs web                             |OK      |                                                              |
+--+----------------------------------------+--------+--------------------------------------------------------------+
Ran 16 in 47.537s
OK (success=15, failures=0, errors=0)

======
TOTALS
======
Ran 16 in 47.537s
OK (success=15, failures=0, errors=0)

Note

Detailed description of the utility is available in the section “Self-diagnosis utility”. Detailed description of the module error codes is available in the section Description of errors in calls to RSserver.