Welcome to RTP ducomentation¶
Infrastructure Overview (INF)¶
This project implements a real time platform (rtp) to deploy the O-CU and O-DU.
In O-RAN architecture, the O-DU and O-CU could have different deployed scenarios. The could be container based or VM based, in this release, we only cover the container one. In general the performance sensitive parts of the 5G stack require real time platform, especially for O-DU, the L1 and L2 are requiring the real time feature, the platform should support the Preemptive Scheduling feature.
Following requirements are going to address the container based solution:
1.Support the real time kernel
2.Support Node Feature Discovery
3.Support CPU Affinity and Isolation
4.Support Dynamic HugePages Allocation
And for the network requirements, the following should be supported:
1.Multiple Networking Interface
2.High performance data plane including the DPDK based vswitch and PCI pass-through/SR-IOV.
In the Cherry release, besides the Bronze release features, it has the 2 server mode supported. And these 2 servers will be active and standby mode to provide the high availability (HA) capability. The controller functionality and storage functionality will be deployed at the 2 servers with standby-active mode managed by “service management”. If one server or one service in one server has error, it will be switched from active to standby one to maintain the service availability.
In the Bronze release, the following components and services are enabled:
Fault Management
Framework for infrastructure services to raise and persist alarm and event data.
Set, clear and query customer alarms
Generate customer logs for significant events
Maintains an Active Alarm List
Provides REST API to query alarms and events, also available through SNMP traps
Support for alarm suppression
Operator alarms
On platform nodes and resources
On hosted virtual resources
Operator logs - Event List
Logging of sets/clears of alarms
Related to platform nodes and resources
Related to hosted virtual resources
Configuration Management
Manages Installation and Commissioning
Auto-discover of new nodes
Full Infrastructure management
Manage installation parameters (i.e. console, root disks)
Nodal Configuration
Node role, role profiles
Core, memory (including huge page) assignments
Network Interfaces and storage assignments
Hardware Discovery
CPU/cores, SMT, processors, memory, huge pages
Storage, ports
GPUs, storage, Crypto/compression H/W
Software Management
Manages Installation and Commissioning
Auto-discover of new nodes
Full Infrastructure management
Manage installation parameters (i.e. console, root disks)
Nodal Configuration
Node role, role profiles
Core, memory (including huge page) assignments
Network Interfaces and storage assignments
Hardware Discovery
CPU/cores, SMT, processors, memory, huge pages
Storage, ports
GPUs, storage, Crypto/compression H/W
Host Management
Full life-cycle and availability management of the physical hosts
Detects and automatically handles host failures and initiates recovery
Monitoring and fault reporting for:
Cluster connectivity
Critical process failures
Resource utilization thresholds, interface states
H/W fault / sensors, host watchdog
Activity progress reporting
Interfaces with board management (BMC)
For out of band reset
Power-on/off
H/W sensor monitoring
Service Management
Manages high availability of critical infrastructure and cluster services
Supports many redundancy models: N, or N+M
Active or passive monitoring of services
Allows for specifying the impact of a service failure and escalation policy
Automatically recovers failed services
Uses multiple messaging paths to avoid split-brain communication failures
Up to 3 independent communication paths
LAG can also be configured for multi-link protection of each path
Messages are authenticated using HMAC
SHA-512 if configured / enabled on an interface by-interface basis
Support the ansible bootstrap to implement the zero touch provisioning
Enable the ansible configuration functions for infrastructure itself including the image installation and service configuration.
NOTE: These features leverage the StarlingX (www.starlingx.io). And in Bronze release, these features are only avalaible for IA platform.
NOTE: In Bronze release single server solution is supported only. All the functionalities include controller functions, storage functions and compute functions are integrated in the single server.
About Yocto and OpenEmbedded¶
The Yocto Project is an open source collaboration project that provides templates, tools and methods to help you create custom Linux-based systems for embedded and IOT products, regardless of the hardware architecture.
OpenEmbedded is a build automation framework and cross-compile environment used to create Linux distributions for embedded devices. The OpenEmbedded framework is developed by the OpenEmbedded community, which was formally established in 2003. OpenEmbedded is the recommended build system of the Yocto Project, which is a Linux Foundation workgroup that assists commercial companies in the development of Linux-based systems for embedded products.
About StarlingX¶
StarlingX is a complete cloud infrastructure software stack for the edge used by the most demanding applications in industrial IOT, telecom, video delivery and other ultra-low latency use cases. With deterministic low latency required by edge applications, and tools that make distributed edge manageable, StarlingX provides a container-based infrastructure for edge implementations in scalable solutions that is ready for production now.
Contact info¶
If you need support or add new features/components, please feel free to contact the following:
Jackie Huang <jackie.huang@windriver.com>
Xiaohua Zhang <xiaohua.zhang@windriver.com>
Developer-Guide¶
This project implements a real time platform to deploy the O-CU and O-DU and it’s based on Yocto/OpenEmbedded.
This includes a Yocto/OpenEmbedded compatible layer meta-stx-oran and wrapper scripts to pull all required Yocto/OE layers to build out the reference platform.
To contribute on this project, basic knowledge of Yocto/OpenEmbedded is needed, please refer to the following docs if you want to learn about how to develop with Yocto/OpenEmbedded:
1. Prerequisite for build environment¶
Your host need to meet the requirements for Yocto, please refer to:
The recommended and tested host is Ubuntu 16.04/18.04 and CentOS 7.
To install the required packages for Ubuntu 16.04/18.04:
$ sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib \
build-essential chrpath socat cpio python python3 python3-pip python3-pexpect \
xz-utils debianutils iputils-ping make xsltproc docbook-utils fop dblatex xmlto \
python-git
To install the required packages for CentOS 7:
$ sudo yum install -y epel-release
$ sudo yum makecache
$ sudo yum install gawk make wget tar bzip2 gzip python unzip perl patch \
diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat \
perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-Digest-SHA \
python34-pip xz which SDL-devel xterm
2. Use wrapper script build_inf.sh to setup build env and build the INF AIO x86 image¶
# Get the wrapper script with either curl or wget
$ curl -o build_inf.sh 'https://gerrit.o-ran-sc.org/r/gitweb?p=pti/rtp.git;a=blob_plain;f=scripts/build_inf.sh;hb=HEAD'
$ wget -O build_inf.sh 'https://gerrit.o-ran-sc.org/r/gitweb?p=pti/rtp.git;a=blob_plain;f=scripts/build_inf.sh;hb=HEAD'
$ chmod +x build_inf.sh
$ WORKSPACE=/path/to/workspace
$ ./build_inf.sh -w ${WORKSPACE}
If all go well, you will get the ISO image in: ${WORKSPACE}/prj_oran_inf_anaconda/tmp-glibc/deploy/images/intel-corei7-64/inf-image-aio-installer-intel-corei7-64.iso
Release-notes¶
This document provides the release notes for 6.0.0 of RTP.
Version history¶
Date |
Ver. |
Author |
Comment |
2019-11-02 |
1.0.0 |
Jackie Huang |
Initial Version Amber Release |
2020-06-14 |
2.0.0 |
Xiaohua Zhang |
Bronze Release |
2020-11-23 |
3.0.0 |
Xiaohua Zhang |
Cherry Release |
2021-06-29 |
4.0.0 |
Xiaohua Zhang |
Dawn Release |
2021-12-15 |
5.0.0 |
Jackie Huang |
E Release |
2022-06-15 |
6.0.0 |
Jackie Huang |
F Release |
Version 6.0.0, 2022-06-15¶
Sixth version (F release)
INF MultiOS support:
Add support for CentOS as the base OS
Two images will be provided:
Yocto based image
CentOS based image
Enable three deployment modes on Yocto based image:
AIO simplex mode
AIO duplex mode (2 servers with High Availabity)
AIO duplex mode (2 servers with High Availabity) with additional worker node
Enable four deployment modes on CentOS based image:
AIO simplex mode
AIO duplex mode (2 servers with High Availabity)
AIO duplex mode (2 servers with High Availabity) with additional worker node
Distributed Cloud
Version 5.0.0, 2021-12-15¶
Fifth version (E release)
Upgrade most components to align with StarlingX 5.0
Enable three deployment modes:
AIO simplex mode
AIO duplex mode (2 servers with High Availabity)
AIO duplex mode (2 servers with High Availabity) with additional worker node
Version 4.0.0, 2021-06-29¶
Fourth version (D release)
Enable the AIO duplex mode (2 servers with High Availabity) with additional worker node.
Reconstruct the repo to align the upstream projects include StarlingX and Yocto
Version 3.0.0, 2020-11-23¶
Third version (Cherry)
Based on version 2.0.0 (Bronze)
Add the AIO (all-in-one) 2 servers mode (High Availability)
Version 2.0.0, 2020-06-14¶
Second version (Bronze)
Based on Yocto version 2.7
Linux kernel 5.0 with preempt-rt patches
Leverage the StarlingX 3.0
Support the AIO (all-in-one) deployment scenario
With Software Management, Configuration Management, Host Management, Service Management, and Service Management enabled for IA platform
Support the Kubernetes Cluster for ARM platform (verified by NXP LX2160A)
With the ansbile bootstrap supported for IA platform
Version 1.0.0, 2019-11-02¶
Initial Version
Based on Yocto version 2.6 (‘thud’ branch)
Linux kernel 4.18.41 with preempt-rt patches
Add Docker-18.09.0, kubernetes-1.15.2
Add kubernetes plugins:
kubernetes-dashboard-1.8.3
flannel-0.11.0
multus-cni-3.3
node-feature-discovery-0.4.0
cpu-manager-for-kubernetes-1.3.1
Installation Guide¶
Abstract¶
This document describes how to install O-RAN INF image, example configuration (All-in-one Duplex) for better real time performance, and example deployment of Kubernetes cluster and plugins.
The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux and container technology.
Preface¶
Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
Hardware Requirements¶
Following minimum hardware requirements must be met for installation of O-RAN INF image with AIO-DX:
HW Aspect |
Requirement |
# of servers |
2 |
CPU |
Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket or Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) |
RAM |
32G |
Disk |
|
NICs |
|
BIOS settings |
|
ORAN INF E Release tested on HP ProLiant DL380p Gen8¶
1. Installation for the first server from the O-RAN INF ISO image¶
Please see the README.md file for how to build the image.
The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso
1.1 Burn the image to the USB device¶
Assume the the usb device is /dev/sdX here
$ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
1.2 Install the first server (controller-0)¶
Reboot the target from the USB device.
Select “All-in-one Graphics console” or “All-in-one Serial console install” and press ENTER
Start the auto installation
It will reboot aotumatically after installation
2. Configuration and initialize the bootstrap¶
2.1 First Login with sysadmin/sysadmin and change password¶
2.2 Set OAM network before bootstrap¶
export OAM_DEV=eno3
export CONTROLLER0_OAM_CIDR=128.224.210.110/24
export DEFAULT_OAM_GATEWAY=128.224.210.1
sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
sudo ip link set up dev $OAM_DEV
sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
2.3 Login the server through SSH with sysadmin¶
2.4 Prepare the localhost.yml for bootstrap¶
cat << EOF > localhost.yml
system_mode: duplex
management_subnet: 192.168.18.0/24
management_start_address: 192.168.18.2
management_end_address: 192.168.18.50
management_gateway_address: 192.168.18.1
external_oam_subnet: 128.224.210.0/24
external_oam_gateway_address: 128.224.210.1
external_oam_floating_address: 128.224.210.110
external_oam_node_0_address: 128.224.210.111
external_oam_node_1_address: 128.224.210.112
EOF
2.5 Run the ansible bootstrap¶
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
After the bootstrap successfully finish, it will show as following:
PLAY RECAP *************************************************************************************************************
localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
2.6 Congiure controller-0¶
Acquire admin credentials:
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$
Configure the OAM and MGMT interfaces of controller-0 and specify the attached networks:
OAM_IF=eno3
MGMT_IF=eno1
system host-if-modify controller-0 lo -c none
IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
for UUID in $IFNET_UUIDS; do
system interface-network-remove ${UUID}
done
system host-if-modify controller-0 $OAM_IF -n oam0
system host-if-modify controller-0 $MGMT_IF -n pxeboot0
system host-if-modify controller-0 oam0 -c platform
system interface-network-assign controller-0 oam0 oam
system host-if-modify controller-0 pxeboot0 -c platform
system interface-network-assign controller-0 pxeboot0 pxeboot
system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
system interface-network-assign controller-0 mgmt0 mgmt
system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
system interface-network-assign controller-0 cluster0 cluster-host
Example output:
[sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
[sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | lo |
| iftype | virtual |
| ports | [] |
| imac | 00:00:00:00:00:00 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-17T00:30:45.265032+00:00 |
| updated_at | 2021-11-17T01:03:39.031612+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
[sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
> system interface-network-remove ${UUID}
> done
Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:0c:b2 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-17T00:28:32.365863+00:00 |
| updated_at | 2021-11-17T01:03:45.090904+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | pxeboot0 |
| iftype | ethernet |
| ports | [u'eno1'] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-17T00:28:32.612230+00:00 |
| updated_at | 2021-11-17T01:03:47.341003+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:0c:b2 |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-17T00:28:32.365863+00:00 |
| updated_at | 2021-11-17T01:03:49.368879+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
| ifname | oam0 |
| network_name | oam |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | pxeboot0 |
| iftype | ethernet |
| ports | [u'eno1'] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-17T00:28:32.612230+00:00 |
| updated_at | 2021-11-17T01:03:53.143795+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
| ifname | pxeboot0 |
| network_name | pxeboot |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | mgmt0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | 18 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2021-11-17T01:03:57.303000+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
| ifname | mgmt0 |
| network_name | mgmt |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | cluster0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | 19 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2021-11-17T01:04:02.613518+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
| ifname | cluster0 |
| network_name | cluster-host |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
| 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
Configure NTP Servers for network time synchronization:
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
Output
[sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
| ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
| isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
| created_at | 2021-11-17T00:27:23.529571+00:00 |
| updated_at | None |
+--------------+--------------------------------------+
Add an OSD on controller-0 for Ceph:
system host-disk-list controller-0
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-stor-list controller-0
Output
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
| | de | num | type | gib | gib | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| | | | | 75 | | | | |
| | | | | | | | | |
| 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 0 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
| journal_node | /dev/sdb2 |
| uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 |
| tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
| tier_name | storage |
| created_at | 2021-11-17T01:05:04.063823+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 1 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
| journal_node | /dev/sdc2 |
| uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 |
| tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
| tier_name | storage |
| created_at | 2021-11-17T01:05:06.939798+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
| 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
2.7 Unlock controller-0¶
system host-unlock controller-0
Output:
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
+-----------------------+-------------------------------------------------+
| Property | Value |
+-----------------------+-------------------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | none |
| bm_username | None |
| boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| capabilities | {u'stor_function': u'monitor'} |
| clock_synchronization | ntp |
| config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 |
| config_status | None |
| config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 |
| console | ttyS0,115200 |
| created_at | 2021-11-17T00:28:01.983673+00:00 |
| hostname | controller-0 |
| id | 1 |
| install_output | graphical |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | provisioning |
| location | {} |
| mgmt_ip | 192.168.18.3 |
| mgmt_mac | 24:6e:96:5d:0c:92 |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| serialid | None |
| software_load | 21.05 |
| subfunction_avail | online |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | Unlocking |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2021-11-17T01:05:07.015414+00:00 |
| uptime | 3496 |
| uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vim_progress_status | None |
+-----------------------+-------------------------------------------------+
Controller-0 will reboot to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine.
Once the controller comes back up, check the status of controller-0
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
2. Installation for the second server (controller-1)¶
2.1 Power on the controller-1 server and force it to network boot¶
2.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node¶
2.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)¶
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | degraded |
| 2 | None | None | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+
2.4 Using the host id, set the personality of this host to ‘controller’:¶
[sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| action | none |
| administrative | locked |
| availability | offline |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {} |
| clock_synchronization | ntp |
| config_applied | None |
| config_status | None |
| config_target | None |
| console | ttyS0,115200 |
| created_at | 2021-11-17T10:17:44.387813+00:00 |
| hostname | controller-1 |
| id | 2 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | None |
| invprovision | None |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 21.05 |
| subfunction_avail | not-installed |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | None |
| tboot | false |
| ttys_dcd | None |
| updated_at | None |
| uptime | 0 |
| uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 |
| vim_progress_status | None |
+-----------------------+--------------------------------------+
2.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in ‘system host-list’.¶
This can take 5-10 minutes, depending on the performance of the host machine.
[root@controller-0 hieradata(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | locked | disabled | online |
+----+--------------+-------------+----------------+-------------+--------------+
2.6 Configure controller-1¶
OAM_IF=eno3
MGMT_IF=eno1
system host-if-modify controller-1 $OAM_IF -n oam0
system host-if-modify controller-1 oam0 -c platform
system interface-network-assign controller-1 oam0 oam
system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
system interface-network-assign controller-1 cluster0 cluster-host
system host-if-list controller-1
system host-disk-list controller-1
system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
system host-stor-list controller-1
Output:
[sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
[sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:39:0e |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-22T14:01:32.365863+00:00 |
| updated_at | 2021-11-22T15:04:45.090904+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:39:0e |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2021-11-22T14:05:16.052229+00:00 |
| updated_at | 2021-11-22T15:08:35.324634+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-1 |
| uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
| ifname | oam0 |
| network_name | oam |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | cluster0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:38:ee |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | 19 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2021-11-22T15:08:43.932209+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-1 |
| uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
| ifname | cluster0 |
| network_name | cluster-host |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
| c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
| | de | num | type | gib | gib | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| | | | | 75 | | | | |
| | | | | | | | | |
| 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 |
| | | | | 57 | | | | |
| | | | | | | | | |
| 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 |
| | | | | 57 | | | | |
| | | | | | | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 2 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
| journal_node | /dev/sdb2 |
| uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 |
| tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
| tier_name | storage |
| created_at | 2021-11-22T15:11:55.641193+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 3 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
| journal_node | /dev/sdc2 |
| uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae |
| tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
| tier_name | storage |
| created_at | 2021-11-22T15:12:04.274839+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
| 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
2.7 Unlock controller-1¶
Unlock controller-1 in order to bring it into service:
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {u'stor_function': u'monitor'} |
| clock_synchronization | ntp |
| config_applied | None |
| config_status | Config out-of-date |
| config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| console | ttyS0,115200 |
| created_at | 2021-11-22T12:58:11.630526+00:00 |
| hostname | controller-1 |
| id | 3 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | unprovisioned |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 21.05 |
| subfunction_avail | online |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | Unlocking |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2021-11-22T15:13:09.716324+00:00 |
| uptime | 752 |
| uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vim_progress_status | None |
+-----------------------+--------------------------------------+
Controller-1 will reboot in order to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine.
[root@controller-0 hieradata(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
+-----------------------+-----------------------------------------------------------------------+
| Property | Value |
+-----------------------+-----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
| clock_synchronization | ntp |
| config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| config_status | None |
| config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| console | ttyS0,115200 |
| created_at | 2021-11-22T12:58:11.630526+00:00 |
| hostname | controller-1 |
| id | 2 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | provisioned |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | enabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 21.05 |
| subfunction_avail | available |
| subfunction_oper | enabled |
| subfunctions | controller,worker,lowlatency |
| task | |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2021-11-22T23:59:07.787759+00:00 |
| uptime | 31008 |
| uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vim_progress_status | services-enabled |
+-----------------------+-----------------------------------------------------------------------+
3. Simple use case for sriov¶
3.1 After controller-0 is rebooted and up running, download the DPDK¶
[sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
[sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Password:
--2021-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10251680 (9.8M) [application/octet-stream]
Saving to: ‘dpdk-17.11.10.tar.xz’
dpdk-17.11.10.tar.xz 100%
[========================================================================================>] 9.78M
1.48MB/s in 6.8s
2021-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
sudo tar xvf dpdk-17.11.10.tar.xz
sudo ln -s dpdk-stable-17.11.10 dpdk-stable
3.2 Prepare the yaml file for the network assignment container¶
The following the exmaple of the yaml file:
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-0
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-0"
> }'
>
> ---
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-1
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-1"
> }'
> EOF
3.3 Run the network assignent container for the 2 VFs¶
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
3.4 Prepare the VF container yaml file¶
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
> apiVersion: v1
> kind: Pod
metadata:
> metadata:
> name: pod-with-dpdk-vfs-0
> annotations:
> k8s.v1.cni.cncf.io/networks: '[
> { "name": "sriov-data-dpdk-0" },
{ "name": "sriov-data-dpdk-1" }
> { "name": "sriov-data-dpdk-1" }
> ]'
> spec:
> restartPolicy: Never
> containers:
> - name: pod-with-dpdk-vfs-0
> image: wrsnfv/ubuntu-dpdk-build:v0.3
> env:
> - name: RTE_SDK
> value: "/usr/src/dpdk"
> command:
> - sleep
> - infinity
> stdin: true
> tty: true
> securityContext:
> privileged: true
> capabilities:
> add:
> - ALL
> resources:
> requests:
> cpu: 4
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> limits:
> cpu: 4
> hugepages-1Gi: 2Gi
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> volumeMounts:
> - mountPath: /mnt/huge-1048576kB
> name: hugepage
> - name: dpdk-volume
> mountPath: /usr/src/dpdk
> - name: lib-volume
> mountPath: /lib/modules
> - name: src-volume
> mountPath: /usr/src/
> volumes:
> - name: hugepage
> emptyDir:
> medium: HugePages
> - name: dpdk-volume
> hostPath:
> path: /opt/dpdk-stable/
> - name: lib-volume
> hostPath:
> path: /lib/modules
> - name: src-volume
> hostPath:
> path: /usr/src/
> EOF
3.5 Run the VF container¶
Start the VF container.
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
pod/pod-with-dpdk-vfs-0 created
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
Login the VF container
kubectl exec -it pod-with-dpdk-vfs-0 -- bash
Build the DPDK
cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/zconf.lex.o
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --syncconfig Kconfig
HOSTCC arch/x86/tools/relocs_32.o
HOSTCC arch/x86/tools/relocs_64.o
HOSTCC arch/x86/tools/relocs_common.o
HOSTLD arch/x86/tools/relocs
HOSTCC scripts/genksyms/genksyms.o
YACC scripts/genksyms/parse.tab.c
HOSTCC scripts/genksyms/parse.tab.o
LEX scripts/genksyms/lex.lex.c
YACC scripts/genksyms/parse.tab.h
HOSTCC scripts/genksyms/lex.lex.o
HOSTLD scripts/genksyms/genksyms
HOSTCC scripts/bin2c
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
HOSTCC scripts/recordmcount
HOSTCC scripts/sortextable
HOSTCC scripts/asn1_compiler
HOSTCC scripts/sign-file
HOSTCC scripts/extract-cert
CC scripts/mod/empty.o
HOSTCC scripts/mod/mk_elfconfig
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/modpost.o
CC scripts/mod/devicetable-offsets.s
UPD scripts/mod/devicetable-offsets.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
CC kernel/bounds.s
CC arch/x86/kernel/asm-offsets.s
CALL scripts/checksyscalls.sh
Build the test_pmd application
cd $RTE_SDK
./usertools/dpdk-setup.sh
Option: 14
CC config.o
CC iofwd.o
CC macfwd.o
CC macswap.o
CC flowgen.o
CC rxonly.o
CC txonly.o
CC csumonly.o
CC icmpecho.o
CC tm.o
LD testpmd
INSTALL-APP testpmd
INSTALL-MAP testpmd.map
== Build app/proc_info
CC main.o
LD dpdk-procinfo
INSTALL-APP dpdk-procinfo
INSTALL-MAP dpdk-procinfo.map
== Build app/pdump
CC main.o
LD dpdk-pdump
INSTALL-APP dpdk-pdump
INSTALL-MAP dpdk-pdump.map
== Build app/test-crypto-perf
CC main.o
CC cperf_ops.o
CC cperf_options_parsing.o
CC cperf_test_vectors.o
CC cperf_test_throughput.o
CC cperf_test_latency.o
CC cperf_test_pmd_cyclecount.o
CC cperf_test_verify.o
CC cperf_test_vector_parsing.o
CC cperf_test_common.o
LD dpdk-test-crypto-perf
INSTALL-APP dpdk-test-crypto-perf
INSTALL-MAP dpdk-test-crypto-perf.map
== Build app/test-eventdev
CC evt_main.o
CC evt_options.o
CC evt_test.o
CC parser.o
CC test_order_common.o
CC test_order_queue.o
CC test_order_atq.o
CC test_perf_common.o
CC test_perf_queue.o
CC test_perf_atq.o
LD dpdk-test-eventdev
INSTALL-APP dpdk-test-eventdev
INSTALL-MAP dpdk-test-eventdev.map
Build complete [x86_64-native-linuxapp-gcc]
Installation cannot run with T defined and DESTDIR undefined
------------------------------------------------------------------------------
RTE_TARGET exported as x86_64-native-linuxapp-gcc
------------------------------------------------------------------------------
Press enter to continue ...
Check the VF PCI information:
root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
Get back to pod and run the test_pmd
cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
taskset -p --cpu-list 1
pid 1's current affinity list: 2,3,18,19
./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
peer=1,"9e:fd:e6:dd:c1:01"
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:11.0 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:05:11.1 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
Auto-start selected
Ports to start sending a burst of packets first
Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
Configuring Port 0 (socket 0)
Port 0: 9E:FD:E6:DD:C1:01
Configuring Port 1 (socket 0)
Port 1: 9E:FD:E6:DD:C1:02
Checking link statuses...
Port0 Link Up. speed 10000 Mbps- full-duplex
Port1 Link Up. speed 10000 Mbps- full-duplex
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
disabled
Logical Core 3 (socket 0) forwards packets on 2 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
port 1:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 56 RX-missed: 0 RX-bytes: 4096
RX-errors: 0
RX-nombuf: 0
TX-packets: 64 TX-errors: 0 TX-bytes: 4096
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 432 RX-missed: 0 RX-bytes: 27712
RX-errors: 0
RX-nombuf: 0
TX-packets: 461 TX-errors: 0 TX-bytes: 30080
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
RX-errors: 0
RX-nombuf: 0
TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
Throughput (since last show)
Rx-pps: 7068409
Tx-pps: 7091206
############################################################################