.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 .. Copyright (C) 2019 Wind River Systems, Inc. Installation Guide ================== .. contents:: :depth: 3 :local: Abstract -------- This document describes how to install O-RAN INF image, example configuration for better real time performance, and example deployment of Kubernetes cluster and plugins. The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux and container technology. Version history +--------------------+--------------------+--------------------+--------------------+ | **Date** | **Ver.** | **Author** | **Comment** | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2020-11-23 | 3.0.0 (Cherry) | Xiaohua Zhang | Cherry version | | | | | | +--------------------+--------------------+--------------------+--------------------+ Preface ------- Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide. Hardware Requirements --------------------- Following minimum hardware requirements must be met for installation of O-RAN INF image: +--------------------+----------------------------------------------------+ | **HW Aspect** | **Requirement** | | | | +--------------------+----------------------------------------------------+ | **# of servers** | 1 | +--------------------+----------------------------------------------------+ | **CPU** | 4 | | | | +--------------------+----------------------------------------------------+ | **RAM** | 16G | | | | +--------------------+----------------------------------------------------+ | **Disk** | 500G * 2 | | | | +--------------------+----------------------------------------------------+ | **NICs** | 2 | | | | +--------------------+----------------------------------------------------+ If want to enable the HA mode which is available in Cherry release, 2 servers are needed. ORAN INF Cherry test on Dell R740 ================================= 1. Installation for the first server from the O-RAN INF ISO image ----------------------------------------------------------------- - Please see the README.md file for how to build the image. - The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso 1.1 Burn the image to the USB device ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Assume the the usb device is /dev/sdX here :: $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M 1.2 Install the first Dell-R740 server (controller-0) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Reboot the target from the USB device. - Select “All-in-one Graphics console” or “All-in-one Serial console install” and press ENTER - Start the auto installation - It will reboot aotumatically after installation 2. Configuration and initialize the bootstrap --------------------------------------------- 2.1 First Login with sysadmin/sysadmin and change password ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2.2 Set OAM network before bootstrap ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: export OAM_DEV=eno3 export CONTROLLER0_OAM_CIDR=128.224.210.110/24 export DEFAULT_OAM_GATEWAY=128.224.210.1 sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV sudo ip link set up dev $OAM_DEV sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV 2.3 Login the server through SSH with sysadmin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2.4 Prepare the localhost.yml for bootstrap ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: cat << EOF > localhost.yml system_mode: duplex management_subnet: 192.168.18.0/24 management_start_address: 192.168.18.2 management_end_address: 192.168.18.50 management_gateway_address: 192.168.18.1 external_oam_subnet: 128.224.210.0/24 external_oam_gateway_address: 128.224.210.1 external_oam_floating_address: 128.224.210.110 external_oam_node_0_address: 128.224.210.111 external_oam_node_1_address: 128.224.210.112 EOF 2.5 Run the ansible bootstrap ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv After the bootstrap successfully finish, it will show as following: :: PLAY RECAP ************************************************************************************************************* localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0 2.6 Congiure controller-0 ~~~~~~~~~~~~~~~~~~~~~~~~~ Acquire admin credentials: :: controller-0:~$ source /etc/platform/openrc [sysadmin@controller-0 ~(keystone_admin)]$ Configure the OAM and MGMT interfaces of controller-0 and specify the attached networks: :: OAM_IF=eno3 MGMT_IF=eno1 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $OAM_IF -n oam0 system host-if-modify controller-0 $MGMT_IF -n pxeboot0 system host-if-modify controller-0 oam0 -c platform system interface-network-assign controller-0 oam0 oam system host-if-modify controller-0 pxeboot0 -c platform system interface-network-assign controller-0 pxeboot0 pxeboot system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0 system interface-network-assign controller-0 mgmt0 mgmt system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0 system interface-network-assign controller-0 cluster0 cluster-host Example output: :: [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | lo | | iftype | virtual | | ports | [] | | imac | 00:00:00:00:00:00 | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-17T00:30:45.265032+00:00 | | updated_at | 2020-11-17T01:03:39.031612+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do > system interface-network-remove ${UUID} > done Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | oam0 | | iftype | ethernet | | ports | [u'eno3'] | | imac | 24:6e:96:5d:0c:b2 | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-17T00:28:32.365863+00:00 | | updated_at | 2020-11-17T01:03:45.090904+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | pxeboot0 | | iftype | ethernet | | ports | [u'eno1'] | | imac | 24:6e:96:5d:0c:92 | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-17T00:28:32.612230+00:00 | | updated_at | 2020-11-17T01:03:47.341003+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | oam0 | | iftype | ethernet | | ports | [u'eno3'] | | imac | 24:6e:96:5d:0c:b2 | | imtu | 1500 | | ifclass | platform | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-17T00:28:32.365863+00:00 | | updated_at | 2020-11-17T01:03:49.368879+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-0 | | uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 | | ifname | oam0 | | network_name | oam | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | pxeboot0 | | iftype | ethernet | | ports | [u'eno1'] | | imac | 24:6e:96:5d:0c:92 | | imtu | 1500 | | ifclass | platform | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-17T00:28:32.612230+00:00 | | updated_at | 2020-11-17T01:03:53.143795+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-0 | | uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c | | ifname | pxeboot0 | | network_name | pxeboot | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | mgmt0 | | iftype | vlan | | ports | [] | | imac | 24:6e:96:5d:0c:92 | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | 18 | | uses | [u'pxeboot0'] | | used_by | [] | | created_at | 2020-11-17T01:03:57.303000+00:00 | | updated_at | None | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-0 | | uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 | | ifname | mgmt0 | | network_name | mgmt | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | cluster0 | | iftype | vlan | | ports | [] | | imac | 24:6e:96:5d:0c:92 | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vlan_id | 19 | | uses | [u'pxeboot0'] | | used_by | [] | | created_at | 2020-11-17T01:04:02.613518+00:00 | | updated_at | None | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-0 | | uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 | | ifname | cluster0 | | network_name | cluster-host | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 | | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 | | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 | | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 | +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ Configure NTP Servers for network time synchronization: :: system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org Output :: [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 | | ntpservers | 0.pool.ntp.org,1.pool.ntp.org | | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a | | created_at | 2020-11-17T00:27:23.529571+00:00 | | updated_at | None | +--------------+--------------------------------------+ Add an OSD on controller-0 for Ceph: :: system host-disk-list controller-0 system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {} system host-stor-list controller-0 Output :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | | | de | num | type | gib | gib | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | | | | | | 75 | | | | | | | | | | | | | | | | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 | | | | | | 25 | | | | | | | | | | | | | | | | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 | | | | | | 25 | | | | | | | | | | | | | | | | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 | | | | | | 25 | | | | | | | | | | | | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} +------------------+-------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------+ | osdid | 0 | | function | osd | | state | configuring-on-unlock | | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c | | journal_size_gib | 1024 | | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | | journal_node | /dev/sdb2 | | uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 | | tier_name | storage | | created_at | 2020-11-17T01:05:04.063823+00:00 | | updated_at | None | +------------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {} +------------------+-------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------+ | osdid | 1 | | function | osd | | state | configuring-on-unlock | | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | | journal_size_gib | 1024 | | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | | journal_node | /dev/sdc2 | | uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 | | tier_name | storage | | created_at | 2020-11-17T01:05:06.939798+00:00 | | updated_at | None | +------------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage | | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ 2.7 Unlock controller-0 ~~~~~~~~~~~~~~~~~~~~~~~ :: system host-unlock controller-0 Output: :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0 +-----------------------+-------------------------------------------------+ | Property | Value | +-----------------------+-------------------------------------------------+ | action | none | | administrative | locked | | availability | online | | bm_ip | None | | bm_type | none | | bm_username | None | | boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | | capabilities | {u'stor_function': u'monitor'} | | clock_synchronization | ntp | | config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 | | config_status | None | | config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 | | console | ttyS0,115200 | | created_at | 2020-11-17T00:28:01.983673+00:00 | | hostname | controller-0 | | id | 1 | | install_output | graphical | | install_state | None | | install_state_info | None | | inv_state | inventoried | | invprovision | provisioning | | location | {} | | mgmt_ip | 192.168.18.3 | | mgmt_mac | 24:6e:96:5d:0c:92 | | operational | disabled | | personality | controller | | reserved | False | | rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | | serialid | None | | software_load | 19.12 | | subfunction_avail | online | | subfunction_oper | disabled | | subfunctions | controller,worker,lowlatency | | task | Unlocking | | tboot | false | | ttys_dcd | None | | updated_at | 2020-11-17T01:05:07.015414+00:00 | | uptime | 3496 | | uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | | vim_progress_status | None | +-----------------------+-------------------------------------------------+ Controller-0 will reboot to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine. Once the controller comes back up, check the status of controller-0 :: controller-0:~$ source /etc/platform/openrc [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 2. Installation for the second server (controller-1) ---------------------------------------------------- 2.1 Power on the controller-1 server and force it to network boot ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | degraded | | 2 | None | None | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ 2.4 Using the host id, set the personality of this host to 'controller’: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | action | none | | administrative | locked | | availability | offline | | bm_ip | None | | bm_type | None | | bm_username | None | | boot_device | /dev/sda | | capabilities | {} | | clock_synchronization | ntp | | config_applied | None | | config_status | None | | config_target | None | | console | ttyS0,115200 | | created_at | 2020-11-17T10:17:44.387813+00:00 | | hostname | controller-1 | | id | 2 | | install_output | text | | install_state | None | | install_state_info | None | | inv_state | None | | invprovision | None | | location | {} | | mgmt_ip | 192.168.18.4 | | mgmt_mac | 24:6e:96:5d:38:ee | | operational | disabled | | personality | controller | | reserved | False | | rootfs_device | /dev/sda | | serialid | None | | software_load | 19.12 | | subfunction_avail | not-installed | | subfunction_oper | disabled | | subfunctions | controller,worker,lowlatency | | task | None | | tboot | false | | ttys_dcd | None | | updated_at | None | | uptime | 0 | | uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 | | vim_progress_status | None | +-----------------------+--------------------------------------+ 2.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This can take 5-10 minutes, depending on the performance of the host machine. :: [root@controller-0 hieradata(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | online | +----+--------------+-------------+----------------+-------------+--------------+ 2.6 Configure controller-1 ~~~~~~~~~~~~~~~~~~~~~~~~~~ :: OAM_IF=eno3 MGMT_IF=eno1 system host-if-modify controller-1 $OAM_IF -n oam0 system host-if-modify controller-1 oam0 -c platform system interface-network-assign controller-1 oam0 oam system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0 system interface-network-assign controller-1 cluster0 cluster-host system host-if-list controller-1 system host-disk-list controller-1 system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {} system host-stor-list controller-1 Output: :: [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | oam0 | | iftype | ethernet | | ports | [u'eno3'] | | imac | 24:6e:96:5d:39:0e | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 | | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-22T14:01:32.365863+00:00 | | updated_at | 2020-11-22T15:04:45.090904+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | oam0 | | iftype | ethernet | | ports | [u'eno3'] | | imac | 24:6e:96:5d:39:0e | | imtu | 1500 | | ifclass | platform | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 | | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-11-22T14:05:16.052229+00:00 | | updated_at | 2020-11-22T15:08:35.324634+00:00 | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-1 | | uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 | | ifname | oam0 | | network_name | oam | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0 +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | cluster0 | | iftype | vlan | | ports | [] | | imac | 24:6e:96:5d:38:ee | | imtu | 1500 | | ifclass | None | | aemode | None | | schedpolicy | None | | txhashpolicy | None | | uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 | | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | vlan_id | 19 | | uses | [u'pxeboot0'] | | used_by | [] | | created_at | 2020-11-22T15:08:43.932209+00:00 | | updated_at | None | | sriov_numvfs | 0 | | sriov_vf_driver | None | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-1 | | uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 | | ifname | cluster0 | | network_name | cluster-host | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 | | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 | | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 | | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 | +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | | | de | num | type | gib | gib | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | | | | | | 75 | | | | | | | | | | | | | | | | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 | | | | | | 25 | | | | | | | | | | | | | | | | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 | | | | | | 25 | | | | | | | | | | | | | | | | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 | | | | | | 25 | | | | | | | | | | | | | | | | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 | | | | | | 57 | | | | | | | | | | | | | | | | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 | | | | | | 57 | | | | | | | | | | | | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} +------------------+-------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------+ | osdid | 2 | | function | osd | | state | configuring-on-unlock | | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | | journal_size_gib | 1024 | | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | | journal_node | /dev/sdb2 | | uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c | | tier_name | storage | | created_at | 2020-11-22T15:11:55.641193+00:00 | | updated_at | None | +------------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {} +------------------+-------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------+ | osdid | 3 | | function | osd | | state | configuring-on-unlock | | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c | | journal_size_gib | 1024 | | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | | journal_node | /dev/sdc2 | | uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c | | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae | | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c | | tier_name | storage | | created_at | 2020-11-22T15:12:04.274839+00:00 | | updated_at | None | +------------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage | | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ 2.7 Unlock controller-1 ~~~~~~~~~~~~~~~~~~~~~~~ Unlock controller-1 in order to bring it into service: :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1 +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | action | none | | administrative | locked | | availability | online | | bm_ip | None | | bm_type | None | | bm_username | None | | boot_device | /dev/sda | | capabilities | {u'stor_function': u'monitor'} | | clock_synchronization | ntp | | config_applied | None | | config_status | Config out-of-date | | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac | | console | ttyS0,115200 | | created_at | 2020-11-22T12:58:11.630526+00:00 | | hostname | controller-1 | | id | 3 | | install_output | text | | install_state | None | | install_state_info | None | | inv_state | inventoried | | invprovision | unprovisioned | | location | {} | | mgmt_ip | 192.168.18.4 | | mgmt_mac | 24:6e:96:5d:38:ee | | operational | disabled | | personality | controller | | reserved | False | | rootfs_device | /dev/sda | | serialid | None | | software_load | 19.12 | | subfunction_avail | online | | subfunction_oper | disabled | | subfunctions | controller,worker,lowlatency | | task | Unlocking | | tboot | false | | ttys_dcd | None | | updated_at | 2020-11-22T15:13:09.716324+00:00 | | uptime | 752 | | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | vim_progress_status | None | +-----------------------+--------------------------------------+ Controller-1 will reboot in order to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine. :: [root@controller-0 hieradata(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1 +-----------------------+-----------------------------------------------------------------------+ | Property | Value | +-----------------------+-----------------------------------------------------------------------+ | action | none | | administrative | unlocked | | availability | available | | bm_ip | None | | bm_type | None | | bm_username | None | | boot_device | /dev/sda | | capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} | | clock_synchronization | ntp | | config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac | | config_status | None | | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac | | console | ttyS0,115200 | | created_at | 2020-11-22T12:58:11.630526+00:00 | | hostname | controller-1 | | id | 2 | | install_output | text | | install_state | None | | install_state_info | None | | inv_state | inventoried | | invprovision | provisioned | | location | {} | | mgmt_ip | 192.168.18.4 | | mgmt_mac | 24:6e:96:5d:38:ee | | operational | enabled | | personality | controller | | reserved | False | | rootfs_device | /dev/sda | | serialid | None | | software_load | 19.12 | | subfunction_avail | available | | subfunction_oper | enabled | | subfunctions | controller,worker,lowlatency | | task | | | tboot | false | | ttys_dcd | None | | updated_at | 2020-11-22T23:59:07.787759+00:00 | | uptime | 31008 | | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | | vim_progress_status | services-enabled | +-----------------------+-----------------------------------------------------------------------+ 3. Simple use case for sriov ```````````````````````````` 3.1 After controller-0 is rebooted and up running, download the DPDK '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz Password: --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ... Connecting to fast.dpdk.org|151.101.2.49|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10251680 (9.8M) [application/octet-stream] Saving to: ‘dpdk-17.11.10.tar.xz’ dpdk-17.11.10.tar.xz 100% [========================================================================================>] 9.78M 1.48MB/s in 6.8s 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680] sudo tar xvf dpdk-17.11.10.tar.xz sudo ln -s dpdk-stable-17.11.10 dpdk-stable 3.2 Prepare the yaml file for the network assignment container '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The following the exmaple of the yaml file: :: [sysadmin@controller-0 sriov(keystone_admin)]$ cat < netdef-data-dpdk.yaml > apiVersion: "k8s.cni.cncf.io/v1" > kind: NetworkAttachmentDefinition > metadata: > name: sriov-data-dpdk-0 > annotations: > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0 > spec: > config: '{ > "type": "sriov", > "name": "sriov-data-dpdk-0" > }' > > --- > apiVersion: "k8s.cni.cncf.io/v1" > kind: NetworkAttachmentDefinition > metadata: > name: sriov-data-dpdk-1 > annotations: > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1 > spec: > config: '{ > "type": "sriov", > "name": "sriov-data-dpdk-1" > }' > EOF 3.3 Run the network assignent container for the 2 VFs ''''''''''''''''''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created 3.4 Prepare the VF container yaml file '''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 sriov(keystone_admin)]$ cat < pod-with-dpdk-vfs-0.yaml > apiVersion: v1 > kind: Pod metadata: > metadata: > name: pod-with-dpdk-vfs-0 > annotations: > k8s.v1.cni.cncf.io/networks: '[ > { "name": "sriov-data-dpdk-0" }, { "name": "sriov-data-dpdk-1" } > { "name": "sriov-data-dpdk-1" } > ]' > spec: > restartPolicy: Never > containers: > - name: pod-with-dpdk-vfs-0 > image: wrsnfv/ubuntu-dpdk-build:v0.3 > env: > - name: RTE_SDK > value: "/usr/src/dpdk" > command: > - sleep > - infinity > stdin: true > tty: true > securityContext: > privileged: true > capabilities: > add: > - ALL > resources: > requests: > cpu: 4 > memory: 4Gi > intel.com/pci_sriov_net_physnet0: '1' > intel.com/pci_sriov_net_physnet1: '1' > limits: > cpu: 4 > hugepages-1Gi: 2Gi > memory: 4Gi > intel.com/pci_sriov_net_physnet0: '1' > intel.com/pci_sriov_net_physnet1: '1' > volumeMounts: > - mountPath: /mnt/huge-1048576kB > name: hugepage > - name: dpdk-volume > mountPath: /usr/src/dpdk > - name: lib-volume > mountPath: /lib/modules > - name: src-volume > mountPath: /usr/src/ > volumes: > - name: hugepage > emptyDir: > medium: HugePages > - name: dpdk-volume > hostPath: > path: /opt/dpdk-stable/ > - name: lib-volume > hostPath: > path: /lib/modules > - name: src-volume > hostPath: > path: /usr/src/ > EOF 3.5 Run the VF container '''''''''''''''''''''''' Start the VF container. :: [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml pod/pod-with-dpdk-vfs-0 created [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod NAME READY STATUS RESTARTS AGE pod-with-dpdk-vfs-0 1/1 Running 0 6m40s Login the VF container :: kubectl exec -it pod-with-dpdk-vfs-0 -- bash Build the DPDK :: cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/conf.o HOSTCC scripts/kconfig/confdata.o HOSTCC scripts/kconfig/expr.o HOSTCC scripts/kconfig/symbol.o HOSTCC scripts/kconfig/preprocess.o HOSTCC scripts/kconfig/zconf.lex.o HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf --syncconfig Kconfig HOSTCC arch/x86/tools/relocs_32.o HOSTCC arch/x86/tools/relocs_64.o HOSTCC arch/x86/tools/relocs_common.o HOSTLD arch/x86/tools/relocs HOSTCC scripts/genksyms/genksyms.o YACC scripts/genksyms/parse.tab.c HOSTCC scripts/genksyms/parse.tab.o LEX scripts/genksyms/lex.lex.c YACC scripts/genksyms/parse.tab.h HOSTCC scripts/genksyms/lex.lex.o HOSTLD scripts/genksyms/genksyms HOSTCC scripts/bin2c HOSTCC scripts/kallsyms HOSTCC scripts/conmakehash HOSTCC scripts/recordmcount HOSTCC scripts/sortextable HOSTCC scripts/asn1_compiler HOSTCC scripts/sign-file HOSTCC scripts/extract-cert CC scripts/mod/empty.o HOSTCC scripts/mod/mk_elfconfig MKELF scripts/mod/elfconfig.h HOSTCC scripts/mod/modpost.o CC scripts/mod/devicetable-offsets.s UPD scripts/mod/devicetable-offsets.h HOSTCC scripts/mod/file2alias.o HOSTCC scripts/mod/sumversion.o HOSTLD scripts/mod/modpost CC kernel/bounds.s CC arch/x86/kernel/asm-offsets.s CALL scripts/checksyscalls.sh Build the test_pmd application :: cd $RTE_SDK ./usertools/dpdk-setup.sh Option: 14 CC config.o CC iofwd.o CC macfwd.o CC macswap.o CC flowgen.o CC rxonly.o CC txonly.o CC csumonly.o CC icmpecho.o CC tm.o LD testpmd INSTALL-APP testpmd INSTALL-MAP testpmd.map == Build app/proc_info CC main.o LD dpdk-procinfo INSTALL-APP dpdk-procinfo INSTALL-MAP dpdk-procinfo.map == Build app/pdump CC main.o LD dpdk-pdump INSTALL-APP dpdk-pdump INSTALL-MAP dpdk-pdump.map == Build app/test-crypto-perf CC main.o CC cperf_ops.o CC cperf_options_parsing.o CC cperf_test_vectors.o CC cperf_test_throughput.o CC cperf_test_latency.o CC cperf_test_pmd_cyclecount.o CC cperf_test_verify.o CC cperf_test_vector_parsing.o CC cperf_test_common.o LD dpdk-test-crypto-perf INSTALL-APP dpdk-test-crypto-perf INSTALL-MAP dpdk-test-crypto-perf.map == Build app/test-eventdev CC evt_main.o CC evt_options.o CC evt_test.o CC parser.o CC test_order_common.o CC test_order_queue.o CC test_order_atq.o CC test_perf_common.o CC test_perf_queue.o CC test_perf_atq.o LD dpdk-test-eventdev INSTALL-APP dpdk-test-eventdev INSTALL-MAP dpdk-test-eventdev.map Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined ------------------------------------------------------------------------------ RTE_TARGET exported as x86_64-native-linuxapp-gcc ------------------------------------------------------------------------------ Press enter to continue ... Check the VF PCI information: :: root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address: :: [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn* lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn* lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02 Get back to pod and run the test_pmd :: cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/ taskset -p --cpu-list 1 pid 1's current affinity list: 2,3,18,19 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto- start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth- peer=1,"9e:fd:e6:dd:c1:01" EAL: Detected 32 lcore(s) EAL: No free hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:05:11.0 on NUMA socket 0 EAL: probe driver: 8086:10ed net_ixgbe_vf EAL: using IOMMU type 1 (Type 1) EAL: PCI device 0000:05:11.1 on NUMA socket 0 EAL: probe driver: 8086:10ed net_ixgbe_vf Auto-start selected Ports to start sending a burst of packets first Warning: lsc_interrupt needs to be off when using tx_first. Disabling. USER1: create a new mbuf pool : n=155456, size=2176, socket=0 Configuring Port 0 (socket 0) Port 0: 9E:FD:E6:DD:C1:01 Configuring Port 1 (socket 0) Port 1: 9E:FD:E6:DD:C1:02 Checking link statuses... Port0 Link Up. speed 10000 Mbps- full-duplex Port1 Link Up. speed 10000 Mbps- full-duplex Done No commandline core given, start packet forwarding io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages disabled Logical Core 3 (socket 0) forwards packets on 2 streams: RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02 io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=2 port 0: CRC stripping enabled RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 1: CRC stripping enabled RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 Port statistics ==================================== ######################## NIC statistics for port 0 ######################## RX-packets: 56 RX-missed: 0 RX-bytes: 4096 RX-errors: 0 RX-nombuf: 0 TX-packets: 64 TX-errors: 0 TX-bytes: 4096 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 432 RX-missed: 0 RX-bytes: 27712 RX-errors: 0 RX-nombuf: 0 TX-packets: 461 TX-errors: 0 TX-bytes: 30080 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ Port statistics ==================================== ######################## NIC statistics for port 0 ######################## RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344 RX-errors: 0 RX-nombuf: 0 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376 Throughput (since last show) Rx-pps: 7068409 Tx-pps: 7091206 ############################################################################ References ---------- - `StarlingX`_ .. _`StarlingX`: https://docs.starlingx.io/