Console utilities¶
- CloudManager API service
- Running periodic tasks
- Events monitoring
- Module task queue processing
- OpenStack utility command
- Filling in information about compute nodes
- Initialization of compute nodes
- Decommissioning of a compute node
- Managing the dynamic configuration of node components
- Managing SNMP on the node
- Managing SSH on the node
- Displaying a list of infrastructure nodes
- Power on the node
- Power off the node
- Restart node
- PXE imaging
- Displaying information on the specified PXE image
- Deleting image
- Viewing a list of images
- Initializing power management devices
- Displaying a list of computing devices
- Compute node power management
- Volume mirroring
- Volume migration between PVs
- Availability storage creating
- Availability storage editing
- Availability storage deleting
- Assigning nodes to availability check storage
- Unassigning node to availability storage
- Assigning availability storages to node
- Unassigning availability storage to node
- Listing availability storages
- Bash scripts
CloudManager API service¶
CloudManager provides an API for managing nodes.
Command:
aos-cloud-manager-api
systemd service name:
aos-cloud-manager-api.service
Name | Description | Default value |
---|---|---|
--host |
IP-address on which the service will run. | 0.0.0.0 |
--port |
Port on which the service will run. | 9362 |
--certfile |
Path to file SSL certificate for secure connection. | |
--keyfile |
Path to private key SSL certificate for secure connection. | |
--debug |
Argument allows to start the service in DEBUG mode. |
False |
--logfile |
File path for logging. | /var/log/aos/cloud-manager/cloud-manager-api.log |
Running periodic tasks¶
Utility periodically runs tasks performed by aos-cloud-manager-worker
service:
- nodes_tracker
- Monitoring and management of compute nodes
- node_syncer
- Synchronization of compute nodes
- drs_trigger
- Automatic application of DRS audit results
- storage_syncer
- Collecting information about Cinder block storages
Command:
aos-cloud-manager-beat
systemd service name:
aos-cloud-manager-beat.service
Name | Description | Default value |
---|---|---|
--debug |
Argument allows to start the service in DEBUG mode. |
False |
--logfile |
File path for logging. | /var/log/aos/cloud-manager/beat.log |
Monitoring and management of compute nodes¶
The functionality of the monitoring and control package compute nodes is designed to ensure fault tolerance. It allows:
- to monitor compute nodes;
- to replace failed compute node with a reserved one.
It allows to assign reserved hypervisors from the list of available ones.
- You can assign hypervisor to fallback through the web interface Dashboard by applying to hypervisor action “Move to reserve” or by setting the flag “Reserved” (menu item Administrator-Hypervisors);
- If hypervisor fails, the one of the available reserved hypervisor will be enabled. If there are no reserved hypervisors, then the replacement will not occur which will be reported in log files module CloudManager.
Hypervisor applies to reserved if:
- it is exposed and marked as reserved;
- it has a power management port;
- it is powered off through power management device.
When changing the status hypervisor to down
the following algorithm is executed:
- The number of lost compute nodes is determined which is calculated as the number of all compute nodes in the
down
status except for the reserved ones; - If the number of lost compute nodes exceeds the number specified in the
MAX_DOWN_HOSTS
parameter, the algorithm is aborted; - If the parameter
ALLOW_HOST_AUTO_POWER_OFF
is not set, then auto evacuation is performed with the inclusion of reserved compute node:- Search is performed for the reserved compute node and its subsequent enabling;
- The function of evacuation is performed for all instances located on compute node.
- Otherwise, the following actions are performed:
- Searching for all instances on hypervisor with status
ACTIVE
; - Restarting compute node with waiting for the timeout specified in the parameter
HOST_RESTART_TIMEOUT
; - Polling of compute node status, if the status is
up
, then all instances of compute nodeACTIVE
with are rebooted; - If the status is
down
, then auto evacuation is performed with the enabling of reserved compute node.
- Searching for all instances on hypervisor with status
Synchronization of compute nodes¶
The utility allows you to synchronize information about compute nodes between Nova service database and CloudManager module database for the correct display of the list compute nodes in Dashboard.
- Following actions are performed when the utility is running:
- Nova service database is periodically polled, and if changes are found in the list compute nodes then this list is synchronized with database of the CloudManager module;
- information about the power status of the ports of power management devices associated with compute nodes is updated;
- information about the number of free hypervisors on the node aggregators is updated and compared to the value of ‘free_hypervisor_count’ in the aggregator metadata. If the number of free hypervisors is exceeded, some of the hypervisors are transferred to the reserve and turned off. If there are not enough free hypervisors, the hypervisors from the reserve are turned on.
Automatic application of DRS audit results¶
Utility allows to automatically apply the results of successful audit whenever possible balancing instances on nodes within host aggregators.
Collecting information about Cinder block storages¶
Note
It is need to start the agent service with the control type on nodes with the openstack-cinder-volume service for correct operation.
Utility allows to collect information about Cinder block storage.
Events monitoring¶
Utility allows to track events in Nova and keep the host information up to date in instance metadata. Metadata is updated when instance is created or when it is migrated to another compute node.
Command:
aos-cloud-manager-listener
systemd service name:
aos-cloud-manager-listener.service
Name | Description | Default value |
---|---|---|
--debug |
Argument allows to start service in the DEBUG mode. |
False |
--logfile |
File path for logging. | /var/log/aos/cloud-manager/listener.log |
Module task queue processing¶
Utility performs asynchronous module tasks.
Command:
aos-cloud-manager-worker
systemd service name:
aos-cloud-manager-worker.service
Name | Description | Default value |
---|---|---|
--debug |
Argument allows to start service in the DEBUG mode. |
False |
--logfile |
File path for logging. | /var/log/aos/cloud-manager/worker.log |
OpenStack utility command¶
Note
Output of messages with the level logging WARNING and below is disabled by default. Set log level and run the command with the --verbose
flag to enable verbose output.
Authentication mechanism is described in the section «OpenStackClient Authentication».
Filling in information about compute nodes¶
Utility allows to fill in additional information about compute node.
Sequential user poll is carried out for all compute nodes during the operation of the utility. You can either agree with the poll for the current compute node, and refuse it. In case of refusal to fill in additional information on the current node, additional information in database is not entered. In the case of choosing compute node, the user is given the opportunity to sequentially enter the necessary values for the parameters of this compute node. Each parameter has a default value (displayed in brackets). If you press Enter without entering any information, its default value will be written as the parameter value. If the user selects the q
option, the process will exit without writing the changes to database.
Note
The module assign_data
from the package node_assign_info
is used.
Command:
openstack aos nodes update
Name | Description |
---|---|
-h , --help |
Help output. |
Example:
Do you want to edit this node 0 ([Y]/n/q)?
-Y
Datacenter number (Default: 444): (must be unsigned_int)?
-0
Datacenter Rack number (Default: 134): (must be unsigned_int)?
-0
Row number (Default: None): (must be unsigned_int)?
-0
Inventory code (Default: None): (must be limited_string)?
-123456
Rack Unit number (Default: 127): (must be unsigned_int)?
-0
Select image name (Default: 1):
1: Debian Stable
2: Zarafa Server
3: AD Server
4: DB Server
5: Windows Server with IIS
-2
Is reserved (Default: True): (must be boolean)?
-Yes
CPU Benchmark (Default: test): (must be limited_string)?
-test_upd
Additional information about compute nodes will be written to database at the final stage.
Initialization of compute nodes¶
Utility allows to map the port of the power management device to the IP and MAC-addresses compute node.
Sequential shutdown is started compute nodes by enumerating the cells of devices responsible for power management in the process of the utility operation. Thus, the correspondence is determined compute node and port (cells) of the device that controls power. For each compute node in database information about mapped ports of power management devices is entered. After the installation procedure all mapped between compute nodes and cells of power management devices form a “default” list and therefore do not need to be mapped on subsequent launches.
Note
Module init_nodes
from package node_initialization
and module init_devices
from package power_control
are used.
Command:
openstack aos nodes init
Name | Description |
---|---|
-h , --help |
Help output. |
--auto |
Starting automatic mode. |
--partial-init |
Starting selective mode. |
--excluded-nodes |
Excluding of selected compute nodes. |
Installation process is performed by one of the options:
- Manual mode with requesting permission to shutdown compute nodes;
Launching:
openstack aos nodes init
- Process description:
- available IP addresses are mapped compute nodes and their MAC addresses;
- process of configuring power management devices started;
- node turned off on port 0 of one of the devices, the program enters a shutdown waiting cycle compute node, then, if turned off Compute node> is found, it is mapped to the port and device. Poll is cyclically repeated for all compute nodes. The user can either agree to the shutdown or refuse it. In case of failure, information about the mapping of the given node with the cell and device will not be saved and will not get into database;
- final stage: writing to database information about ports of the power management device with reference to compute node. If the user selects the
q
option, the program will exit without writing the changes to database.
- Automatic mode without requesting permission to shutdown compute nodes;
Launching:
openstack aos nodes init --auto
- Process description:
- available IP addresses are mapped compute nodes and their MAC addresses;
- node turned off on port 0 of one of the devices, the program enters a shutdown waiting cycle compute node, then, if turned off Compute node> is found, it is mapped to the port and device. Shutdown is cyclically repeated for all nodes;
- there will be an entry in database information about the ports of the power management device with reference to compute node at the final stage.
- Selective mode, selective shutdown of compute nodes.
Launching:
openstack aos nodes init --partial-init
- Process description:
- In contrast to manual mode compute nodes that have already been initialized are skipped.
Launching excluding specific compute nodes:
openstack aos nodes init --excluded-nodes node1.local node2.local ... nodeN.local
- Process description:
- Unlike the manual mode when specifying the list compute nodes the selected nodes will be excluded when defining the device port.
Hint
It is necessary to repeat the installation procedure in one of the ways when adding new compute nodes to the system:
- without saving data about compute nodes restart the installation process in manual, automatic or selective mode;
- with saving the data about previous initialization processes, specify the flag
--partial-init
. In this case the status of the already initialized nodes will not be taken into account, including the ports tied to a specific one will not be offered for shutdown compute node.
Decommissioning of a compute node¶
Utility allows to take out compute node from operation with the ability to transfer running instances using live migration, cold migration or evacuation.
It is necessary to specify one or more ways to transfer instances:
Important
If no transfer method is selected, then the hypervisor will be decommissioned only if there are no instances on it.
- Allow live migration of instance;
- Allow stopping and migration of instance;
- Allow instance evacuation.
If several methods are selected, transfer of instances will be carried out according to their priority, and if one succeeds, the rest will not be executed.
Note
Transfer execution priority instances if multiple methods are specified:
- Live migration of instance has high priority;
- Stopping and migration of instance has medium priority;
- Evacuation has low priority.
Command:
openstack aos nodes decommission <hostname>
Name | Description | Default value |
---|---|---|
-h , --help |
Help output. | |
<hostname> |
Hostname. | |
--allow-live-migrate |
Allow live migration instance. | False |
--allow-migrate |
Allow migration instance. | False |
--allow-evacuate |
Allow evacuation instance. | False |
Managing the dynamic configuration of node components¶
Utility allows to enable and disable the dynamic configuration of components compute nodes.
Command:
openstack aos nodes dcc
Name | Description |
---|---|
-h , --help |
Help output. |
--enable |
Enable dynamic configuration of components. When creating a node, the parameter is set by default. |
--disable |
Disable dynamic configuration of components. |
<hostname> |
Hostname or IP address. * is used to perform an action for all nodes. |
Managing SNMP on the node¶
Utility allows to enable and disable SNMP service on the compute nodes.
Command:
openstack aos nodes snmp
Name | Description |
---|---|
-h , --help |
Help output. |
--enable |
Enable SNMP on the node. When creating a node, the parameter is set by default. |
--disable |
Disable SNMP on the node. |
<hostname> |
Hostname or IP address. |
Managing SSH on the node¶
Utility allows to enable and disable SSH service on the compute nodes.
Command:
openstack aos nodes ssh
Name | Description |
---|---|
-h , --help |
Help output. |
--enable |
Enable SSH on the node. When creating a node, the parameter is set by default. |
--disable |
Disable SSH on the node. |
<hostname> |
Hostname or IP address. |
Displaying a list of infrastructure nodes¶
Utility lists the available infrastructure nodes.
Command:
openstack aos nodes list
Example:
openstack aos nodes list
+----+----------------------------------+-----------+------+--------------+
| ID | Name | Class | Type | IP addresses |
+----+----------------------------------+-----------+------+--------------+
| 1 | node1.stand.loc | Hypervisor| Qemu | 10.35.222.11 |
| | | | | 192.168.4.21 |
+----+----------------------------------+-----------+------+--------------+
| 2 | node2.stand.loc | Hypervisor| Qemu | 10.35.222.12 |
+----+----------------------------------+-----------+------+--------------+
| 3 | - | Server | - | 10.35.115.3 |
+----+----------------------------------+-----------+------+--------------+
Power on the node¶
Command:
openstack aos nodes power-on
Name | Description |
---|---|
<name> |
Hostname or ID. |
Example:
openstack aos nodes power-on node1.local
Power off the node¶
Command:
openstack aos nodes power-off
Name | Description |
---|---|
<name> |
Hostname or ID. |
Example:
openstack aos nodes power-off node1.local
Restart node¶
Command:
openstack aos nodes reset
Name | Description |
---|---|
<name> |
Hostname or ID. |
Example:
openstack aos nodes reset node1.local
PXE imaging¶
Note
Administrator rights are required to create an image.
It allows to enter information into database about images from which you can boot by PXE.
Command:
openstack aos pxe-image create
Name | Description |
---|---|
-h , --help |
Help output. |
-name |
Image name. |
-file |
Image address. |
Example:
openstack aos pxe-image create --name default --file /var/lib/tftpboot/pxelinux.cfg/default
Displaying information on the specified PXE image¶
Command:
openstack aos pxe-image show
Name | Description |
---|---|
-h , --help |
Help output. |
--id |
Image ID. |
Example:
openstack aos pxe-image show --id 1
+---------------+-----------------------------------------------------+
| Field | Value |
+---------------+-----------------------------------------------------+
| Image ID | 1 |
| Image Name | default |
| Image Content | default vesamenu.c32 |
| | aprompt 1 |
| | timeout 30 |
| | label ubuntu 14.04 |
| | menu label CENTER COMPUTE NODE DEFAULT |
| | kernel vmlinuz |
| | append initrd=dracut root=192.168.100.1:/mnt/rootfs |
| | biosdevname=0 ip=eth0:on ip=eth1:on |
| | bond=bond0:eth0,eth1:mode=6 ip=bond0:dhcp ro |
+---------------+-----------------------------------------------------+
Deleting image¶
Note
Administrator rights are required to remove an image.
Command:
openstack aos pxe-image delete
Name | Description |
---|---|
-h , --help |
Help output. |
--id |
Image ID. |
Example:
openstack aos pxe-image delete --id 1
Viewing a list of images¶
Command:
openstack aos pxe-image list
Name | Description |
---|---|
-h` , --help |
Help output. |
Initializing power management devices¶
Utility allows to add, delete and edit power management devices.
Command:
openstack aos power init
Name | Description |
---|---|
-h , --help |
Help output. |
Example:
It prompts to initialize, delete or edit installed power management devices:
Do you want (a)dd device, (e)dit or (d)elete existing (a/e/d/[n])?
You can choose a
for adding, e
for editing, d
for deleting or n
for skipping this step.
Device adding.
Adding a new power management device. When choosing an option a
questionnaire about a new device is launched:
Available to choose from a list of implemented devices:
Select device name?
Provided to choose from a list of implemented clients for protocols:
Select protocol name?
Enter the IP address or the host of the device to connect to it using the selected protocol:
Ip address or host of device?
Enter the port of the device to connect to it using the selected protocol:
Port number of device?
Enter the username (may be empty) to connect to him using the selected protocol:
Username to access device?
Enter the password (may be empty) of the device to connect to it using the selected protocol:
Password to access device?
Device editing.
If e
edit mode is selected, there will be a search of available devices with a proposal to edit each:
Do you want edit this device: device_attributes_and_name ([y]/n/q)? and a questionnaire about the device has been launched.
Device deleting.
If d
delete mode is selected, there will be a search of available devices with a proposal to delete each:
Do you want delete this device: device_attributes_and_name ([y]/n/q)?
Selecting y
will restart device addition.
Displaying a list of computing devices¶
Utility displays a list of compute nodes and their state.
Command:
openstack aos power list -h
Name | Description |
---|---|
-h , --help |
Help output. |
-a , --all |
Showing all device ports, only device ports associated with computational nodes are displayed without specifying the -a flag. |
Example:
openstack aos power list
------------------------------------------------------------------------------------------------------------------
N | Hostname | Node name | Device name | Port number | Power state | Node state
------------------------------------------------------------------------------------------------------------------
1 | cn.7c48.local | fnc aмт21 | 10_10_15_21 | 1 | ВКЛ | Включен
2 | cn.2dae.local | aмт13 | 10.10.15.13_13 | 1 | ВКЛ | Включен
3 | cn.46f1.local | тестовая нода | 10.10.15.17_17 | 3 | ВКЛ | Включен
4 | cn.4b5c.local | baremetal_node | 10.10.15.17_17 | 1 | ВЫКЛ | Выключен
------------------------------------------------------------------------------------------------------------------
Compute node power management¶
Utility allows to receive on request the state of the power control device and to enable and disable compute node.
Command:
openstack aos power manage -h
Name | Description |
---|---|
-h , --help |
Help output. |
-a , --all |
All device ports are displayed, flag only device ports associated with compute nodes are displayed without specifying the -a . |
Important
Package allows you to configure power devices including primary. If during the initial setup all ports of the device are disabled, it is need to enable them. Also it provides a custom console interface for adding or editing power management devices.
Volume mirroring¶
Utility allows to convert volume to mirrored or standard.
Command:
openstack aos volume mirror
Name | Description |
---|---|
-h , --help |
Help output. |
<volume> |
Volume name or UUID. |
--type |
Mirroring type (standard or mirrored ). |
Volume migration between PVs¶
Utility allows to migrate volume between physical disks (PV).
Command:
openstack aos volume pvmigrate
Name | Description |
---|---|
-h , --help |
Help output. |
<volume> |
Volume name or UUID. |
--source |
Original physical volume. |
--target |
Target physical volume. |
Availability storage creating¶
Utility allows to create availability storage.
Command:
openstack aos storage create
Name | Description |
---|---|
-h , --help |
Help output. |
<name> |
Availability storage name. |
<compute-path> |
Path to availability storage on compute node. |
<controller-path> |
Path to availability storage on control node. |
Example:
openstack aos storage create test0 /var/ /home/
+-------------------+------+
|Field |Value |
+-------------------+------+
|Storage ID |44 |
+-------------------+------+
|Storage Name |test0 |
+-------------------+------+
|Path for compute |/var/ |
+-------------------+------+
|Path for controller|/home/|
+-------------------+------+
Availability storage editing¶
Utility allows to change availability storage name, path to availability storage on compute node and control node.
Command:
openstack aos storage update
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Availability storage ID. |
--name |
New name of availability storage. |
--compute-path |
New path to availability storage on compute node. |
--controller-path |
New path to availability storage on control node. |
Example:
openstack aos storage update 44 --compute-path /tmp/ --controller-path /tmp/
+-------------------+-----+
|Field |Value|
+-------------------+-----+
|Storage ID |44 |
+-------------------+-----+
|Storage Name |test0|
+-------------------+-----+
|Path for compute |/tmp/|
+-------------------+-----+
|Path for controller|/var/|
+-------------------+-----+
Availability storage deleting¶
Utility allows to delete previously created availability storage.
Command:
openstack aos storage delete
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Availability storage ID. |
Example:
openstack aos storage delete 44
Assigning nodes to availability check storage¶
Utility allows to assign one or more nodes to availability storage.
Command:
openstack aos storage assign
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Availability storage ID. |
--nodes |
Assigned nodes IDs |
Example:
openstack aos storage assign 1 --nodes 1 2 3
Unassigning node to availability storage¶
Utility allows to unassign node to availability storage.
Command:
openstack aos storage unassign
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Storage ID. |
--node |
Node ID. |
Example:
openstack aos storage unassign 1 --node 2
Assigning availability storages to node¶
Utility allows to assign one or more availability storages to node.
Command:
openstack aos storage assign
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Node ID. |
--storages |
Assigned availability storage IDs. |
Example:
openstack aos storage assign 1 --storages 1 2 3
Unassigning availability storage to node¶
Utility allows to unassign availability storage to node.
Command:
openstack aos storage unassign
Name | Description |
---|---|
-h , --help |
Help output. |
<id> |
Node ID. |
--storage |
Availability storage ID. |
Example:
openstack aos storage unassign 1 --storage 3
Listing availability storages¶
Utility allows to display list of created availability storages.
Command:
openstack aos storage list
Name | Description |
---|---|
-h , --help |
Help output. |
--detail |
Display paths for storages on compute node and on control node. |
Example:
openstack aos storage list
+----------+------------+
|Storage ID|Storage Name|
+----------+------------+
|36 |test1 |
+----------+------------+
|43 |test2 |
+----------+------------+
openstack aos storage list --detail
+----------+------------+----------------+-------------------+
|Storage ID|Storage Name|Path for compute|Path for controller|
+----------+------------+----------------+-------------------+
|36 |test1 |/tmp/ |/tmp/ |
+----------+------------+----------------+-------------------+
|43 |test2 |/usr/ |/home/ |
+----------+------------+----------------+-------------------+
Bash scripts¶
Database migration¶
Utility allows to change the structure database module. Necessary tables are created and the schemas are updated in database of the module during the operation of the utility.
Command:
openstack aos db migrate -n node_control