Console utilities¶

Monitoring and management of compute nodes¶

The functionality of the monitoring and control package compute nodes is designed to ensure fault tolerance. It allows:

• to monitor compute nodes;
• to replace failed compute node with a reserved one.

It allows to assign reserved hypervisors from the list of available ones.

• You can assign hypervisor to fallback through the web interface Dashboard by applying to hypervisor action “Move to reserve” or by setting the flag “Reserved” (menu item Administrator-Hypervisors);
• If hypervisor fails, the one of the available reserved hypervisor will be enabled. If there are no reserved hypervisors, then the replacement will not occur which will be reported in log files module CloudManager.

Hypervisor applies to reserved if:

• it is exposed and marked as reserved;
• it has a power management port;
• it is powered off through power management device.

When changing the status hypervisor to down the following algorithm is executed:

1. The number of lost compute nodes is determined which is calculated as the number of all compute nodes in the down status except for the reserved ones;
2. If the number of lost compute nodes exceeds the number specified in the MAX_DOWN_HOSTS parameter, the algorithm is aborted;
3. If the parameter ALLOW_HOST_AUTO_POWER_OFF is not set, then auto evacuation is performed with the inclusion of reserved compute node:
   • Search is performed for the reserved compute node and its subsequent enabling;
   • The function of evacuation is performed for all instances located on compute node.
4. Otherwise, the following actions are performed:
   • Searching for all instances on hypervisor with status ACTIVE;
   • Restarting compute node with waiting for the timeout specified in the parameter HOST_RESTART_TIMEOUT;
   • Polling of compute node status, if the status is up, then all instances of compute node ACTIVE with are rebooted;
   • If the status is down, then auto evacuation is performed with the enabling of reserved compute node.

Synchronization of compute nodes¶

The utility allows you to synchronize information about compute nodes between Nova service database and CloudManager module database for the correct display of the list compute nodes in Dashboard.

Following actions are performed when the utility is running:

• Nova service database is periodically polled, and if changes are found in the list compute nodes then this list is synchronized with database of the CloudManager module;
• information about the power status of the ports of power management devices associated with compute nodes is updated;
• information about the number of free hypervisors on the node aggregators is updated and compared to the value of ‘free_hypervisor_count’ in the aggregator metadata. If the number of free hypervisors is exceeded, some of the hypervisors are transferred to the reserve and turned off. If there are not enough free hypervisors, the hypervisors from the reserve are turned on.

Automatic application of DRS audit results¶

Utility allows to automatically apply the results of successful audit whenever possible balancing instances on nodes within host aggregators.

Collecting information about Cinder block storages¶

Utility allows to collect information about Cinder block storage.

Filling in information about compute nodes¶

Utility allows to fill in additional information about compute node.

Sequential user poll is carried out for all compute nodes during the operation of the utility. You can either agree with the poll for the current compute node, and refuse it. In case of refusal to fill in additional information on the current node, additional information in database is not entered. In the case of choosing compute node, the user is given the opportunity to sequentially enter the necessary values ​​for the parameters of this compute node. Each parameter has a default value (displayed in brackets). If you press Enter without entering any information, its default value will be written as the parameter value. If the user selects the q option, the process will exit without writing the changes to database.

• Command:

  openstack aos nodes update
  

Example:

Do you want to edit this node 0 ([Y]/n/q)?
-Y
Datacenter number (Default: 444): (must be unsigned_int)?
-0
Datacenter Rack number (Default: 134): (must be unsigned_int)?
-0
Row number (Default: None): (must be unsigned_int)?
-0
Inventory code (Default: None): (must be limited_string)?
-123456
Rack Unit number (Default: 127): (must be unsigned_int)?
-0
Select image name (Default: 1):
1: Debian Stable
2: Zarafa Server
3: AD Server
4: DB Server
5: Windows Server with IIS
-2
Is reserved (Default: True): (must be boolean)?
-Yes
CPU Benchmark (Default: test): (must be limited_string)?
-test_upd

Additional information about compute nodes will be written to database at the final stage.

Initialization of compute nodes¶

Utility allows to map the port of the power management device to the IP and MAC-addresses compute node.

Sequential shutdown is started compute nodes by enumerating the cells of devices responsible for power management in the process of the utility operation. Thus, the correspondence is determined compute node and port (cells) of the device that controls power. For each compute node in database information about mapped ports of power management devices is entered. After the installation procedure all mapped between compute nodes and cells of power management devices form a “default” list and therefore do not need to be mapped on subsequent launches.

Command:

openstack aos nodes init

Installation process is performed by one of the options:

1. Manual mode with requesting permission to shutdown compute nodes;

   • Launching:

     openstack aos nodes init
     

   • Process description:

     • available IP addresses are mapped compute nodes and their MAC addresses;
     • process of configuring power management devices started;
     • node turned off on port 0 of one of the devices, the program enters a shutdown waiting cycle compute node, then, if turned off Compute node> is found, it is mapped to the port and device. Poll is cyclically repeated for all compute nodes. The user can either agree to the shutdown or refuse it. In case of failure, information about the mapping of the given node with the cell and device will not be saved and will not get into database;
     • final stage: writing to database information about ports of the power management device with reference to compute node. If the user selects the q option, the program will exit without writing the changes to database.

2. Automatic mode without requesting permission to shutdown compute nodes;

   • Launching:

     openstack aos nodes init --auto
     

   • Process description:

     • available IP addresses are mapped compute nodes and their MAC addresses;
     • node turned off on port 0 of one of the devices, the program enters a shutdown waiting cycle compute node, then, if turned off Compute node> is found, it is mapped to the port and device. Shutdown is cyclically repeated for all nodes;
     • there will be an entry in database information about the ports of the power management device with reference to compute node at the final stage.

3. Selective mode, selective shutdown of compute nodes.

   • Launching:

     openstack aos nodes init --partial-init
     

   • Process description:

     • In contrast to manual mode compute nodes that have already been initialized are skipped.

   • Launching excluding specific compute nodes:

     openstack aos nodes init --excluded-nodes node1.local node2.local ... nodeN.local
     

   • Process description:

     • Unlike the manual mode when specifying the list compute nodes the selected nodes will be excluded when defining the device port.

Decommissioning of a compute node¶

Utility allows to take out compute node from operation with the ability to transfer running instances using live migration, cold migration or evacuation.

It is necessary to specify one or more ways to transfer instances:

• Allow live migration of instance;
• Allow stopping and migration of instance;
• Allow instance evacuation.

If several methods are selected, transfer of instances will be carried out according to their priority, and if one succeeds, the rest will not be executed.

Command:

openstack aos nodes decommission <hostname>

Managing the dynamic configuration of node components¶

Utility allows to enable and disable the dynamic configuration of components compute nodes.

Command:

openstack aos nodes dcc

Managing SNMP on the node¶

Utility allows to enable and disable SNMP service on the compute nodes.

Command:

openstack aos nodes snmp

Managing SSH on the node¶

Utility allows to enable and disable SSH service on the compute nodes.

Command:

openstack aos nodes ssh

Displaying a list of infrastructure nodes¶

Utility lists the available infrastructure nodes.

Command:

openstack aos nodes list

Example:

openstack aos nodes list

+----+----------------------------------+-----------+------+--------------+
| ID | Name                             | Class     | Type | IP addresses |
+----+----------------------------------+-----------+------+--------------+
| 1  | node1.stand.loc                  | Hypervisor| Qemu | 10.35.222.11 |
|    |                                  |           |      | 192.168.4.21 |
+----+----------------------------------+-----------+------+--------------+
| 2  | node2.stand.loc                  | Hypervisor| Qemu | 10.35.222.12 |
+----+----------------------------------+-----------+------+--------------+
| 3  | -                                | Server    | -    | 10.35.115.3  |
+----+----------------------------------+-----------+------+--------------+

Power on the node¶

Command:

openstack aos nodes power-on

Example:

openstack aos nodes power-on node1.local

Power off the node¶

Command:

openstack aos nodes power-off

Example:

openstack aos nodes power-off node1.local

Restart node¶

Command:

openstack aos nodes reset

Example:

openstack aos nodes reset node1.local

PXE imaging¶

It allows to enter information into database about images from which you can boot by PXE.

Command:

openstack aos pxe-image create

Example:

openstack aos pxe-image create --name default --file /var/lib/tftpboot/pxelinux.cfg/default

Displaying information on the specified PXE image¶

Command:

openstack aos pxe-image show

Example:

openstack aos pxe-image show --id 1
+---------------+-----------------------------------------------------+
| Field         | Value                                               |
+---------------+-----------------------------------------------------+
| Image ID      | 1                                                   |
| Image Name    | default                                             |
| Image Content | default vesamenu.c32                                |
|               | aprompt 1                                           |
|               | timeout 30                                          |
|               | label ubuntu 14.04                                  |
|               | menu label CENTER COMPUTE NODE DEFAULT              |
|               | kernel vmlinuz                                      |
|               | append initrd=dracut root=192.168.100.1:/mnt/rootfs |
|               | biosdevname=0 ip=eth0:on ip=eth1:on                 |
|               | bond=bond0:eth0,eth1:mode=6 ip=bond0:dhcp ro        |
+---------------+-----------------------------------------------------+

Deleting image¶

Command:

openstack aos pxe-image delete

Example:

openstack aos pxe-image delete --id 1

Viewing a list of images¶

Command:

openstack aos pxe-image list

Initializing power management devices¶

Utility allows to add, delete and edit power management devices.

Command:

openstack aos power init

Example:

It prompts to initialize, delete or edit installed power management devices:

Do you want (a)dd device, (e)dit or (d)elete existing (a/e/d/[n])?

You can choose a for adding, e for editing, d for deleting or n for skipping this step.

Device adding.

Adding a new power management device. When choosing an option a questionnaire about a new device is launched:

1. Available to choose from a list of implemented devices:

   Select device name?
   

2. Provided to choose from a list of implemented clients for protocols:

   Select protocol name?
   

3. Enter the IP address or the host of the device to connect to it using the selected protocol:

   Ip address or host of device?
   

4. Enter the port of the device to connect to it using the selected protocol:

   Port number of device?
   

5. Enter the username (may be empty) to connect to him using the selected protocol:

   Username to access device?
   

6. Enter the password (may be empty) of the device to connect to it using the selected protocol:

   Password to access device?
   

Device editing.

If e edit mode is selected, there will be a search of available devices with a proposal to edit each:

Do you want edit this device: device_attributes_and_name ([y]/n/q)? and a questionnaire about the device has been launched.

Device deleting.

If d delete mode is selected, there will be a search of available devices with a proposal to delete each:

Do you want delete this device: device_attributes_and_name ([y]/n/q)?

Selecting y will restart device addition.

Displaying a list of computing devices¶

Utility displays a list of compute nodes and their state.

Command:

openstack aos power list -h

Example:

openstack aos power list

------------------------------------------------------------------------------------------------------------------
N | Hostname      | Node name         | Device name             | Port number | Power state       | Node state
------------------------------------------------------------------------------------------------------------------
1 | cn.7c48.local | fnc aмт21         | 10_10_15_21             | 1           | ВКЛ               | Включен
2 | cn.2dae.local | aмт13             | 10.10.15.13_13          | 1           | ВКЛ               | Включен
3 | cn.46f1.local | тестовая нода     | 10.10.15.17_17          | 3           | ВКЛ               | Включен
4 | cn.4b5c.local | baremetal_node    | 10.10.15.17_17          | 1           | ВЫКЛ              | Выключен
------------------------------------------------------------------------------------------------------------------

Compute node power management¶

Utility allows to receive on request the state of the power control device and to enable and disable compute node.

Command:

openstack aos power manage -h

Volume mirroring¶

Utility allows to convert volume to mirrored or standard.

Command:

openstack aos volume mirror

Volume migration between PVs¶

Utility allows to migrate volume between physical disks (PV).

Command:

openstack aos volume pvmigrate

Availability storage creating¶

Utility allows to create availability storage.

Command:

openstack aos storage create

Example:

openstack aos storage create test0 /var/ /home/

+-------------------+------+
|Field              |Value |
+-------------------+------+
|Storage ID         |44    |
+-------------------+------+
|Storage Name       |test0 |
+-------------------+------+
|Path for compute   |/var/ |
+-------------------+------+
|Path for controller|/home/|
+-------------------+------+

Availability storage editing¶

Utility allows to change availability storage name, path to availability storage on compute node and control node.

Command:

openstack aos storage update

Example:

openstack aos storage update 44 --compute-path /tmp/ --controller-path /tmp/

+-------------------+-----+
|Field              |Value|
+-------------------+-----+
|Storage ID         |44   |
+-------------------+-----+
|Storage Name       |test0|
+-------------------+-----+
|Path for compute   |/tmp/|
+-------------------+-----+
|Path for controller|/var/|
+-------------------+-----+

Availability storage deleting¶

Utility allows to delete previously created availability storage.

Command:

openstack aos storage delete

Example:

openstack aos storage delete 44

Assigning nodes to availability check storage¶

Utility allows to assign one or more nodes to availability storage.

Command:

openstack aos storage assign

Example:

openstack aos storage assign 1 --nodes 1 2 3

Unassigning node to availability storage¶

Utility allows to unassign node to availability storage.

Command:

openstack aos storage unassign

Example:

openstack aos storage unassign 1 --node 2

Assigning availability storages to node¶

Utility allows to assign one or more availability storages to node.

Command:

openstack aos storage assign

Example:

openstack aos storage assign 1 --storages 1 2 3

Unassigning availability storage to node¶

Utility allows to unassign availability storage to node.

Command:

openstack aos storage unassign

Example:

openstack aos storage unassign 1 --storage 3

Listing availability storages¶

Utility allows to display list of created availability storages.

Command:

openstack aos storage list

Example:

openstack aos storage list

+----------+------------+
|Storage ID|Storage Name|
+----------+------------+
|36        |test1       |
+----------+------------+
|43        |test2       |
+----------+------------+

openstack aos storage list --detail

+----------+------------+----------------+-------------------+
|Storage ID|Storage Name|Path for compute|Path for controller|
+----------+------------+----------------+-------------------+
|36        |test1       |/tmp/           |/tmp/              |
+----------+------------+----------------+-------------------+
|43        |test2       |/usr/           |/home/             |
+----------+------------+----------------+-------------------+

Database migration¶

Utility allows to change the structure database module. Necessary tables are created and the schemas are updated in database of the module during the operation of the utility.

Command:

openstack aos db migrate -n node_control