.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 .. Copyright (C) 2019 Wind River Systems, Inc. Installation Guide ================== .. contents:: :depth: 3 :local: Abstract -------- This document describes how to install O-RAN INF image, example configuration for better real time performance, and example deployment of Kubernetes cluster and plugins. The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux and container technology. Version history +--------------------+--------------------+--------------------+--------------------+ | **Date** | **Ver.** | **Author** | **Comment** | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | | | | | | | | | | +--------------------+--------------------+--------------------+--------------------+ Preface ------- Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide. Hardware Requirements --------------------- Following minimum hardware requirements must be met for installation of O-RAN INF image: +--------------------+----------------------------------------------------+ | **HW Aspect** | **Requirement** | | | | +--------------------+----------------------------------------------------+ | **# of servers** | 1 | +--------------------+----------------------------------------------------+ | **CPU** | 4 | | | | +--------------------+----------------------------------------------------+ | **RAM** | 16G | | | | +--------------------+----------------------------------------------------+ | **Disk** | 500G * 2 | | | | +--------------------+----------------------------------------------------+ | **NICs** | 2 | | | | +--------------------+----------------------------------------------------+ Software Installation and Deployment ------------------------------------ 1. Installation from the O-RAN INF ISO image ```````````````````````````````````````````` - Please see the README.md file for how to build the image. - The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso 1.1 Burn the image to the USB device '''''''''''''''''''''''''''''''''''' - Assume the the usb device is /dev/sdX here :: $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M 1.2 Install the server '''''''''''''''''''''' - Reboot the target from the USB device. - Select "Graphics console install" or "Serial console install" and press ENTER - Remove the USB device and press ENTER to reboot 2. Configuration and initialize the bootstrap ````````````````````````````````````````````` 2.1 First Login ''''''''''''''' Example: :: ifconfig eno1 128.224.180.14/24 up ip route add default via 128.224.180.1 ping 8.8.8.8 2.3 Login the server through SSH '''''''''''''''''''''''''''''''' 2.4 Prepare the localhost.yml for bootstrap ''''''''''''''''''''''''''''''''''''''''''' Example: :: system_mode: simplex external_oam_subnet: 128.224.180.0/24 external_oam_gateway_address: 128.224.180.1 external_oam_floating_address: 128.224.180.14 admin_username: admin admin_password: Wind123! ansible_become_pass: Wind123! docker_http_proxy: http://xxx.xxx.xxx.xxx:8080 docker_https_proxy: https://xxx.xxx.xxx.xxx:8080 2.5 Run the bootstrap ''''''''''''''''''''' Use the following command: :: ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv After the bootstrap successfully finish, it will show as following: :: PLAY RECAP ****************************************************************************************************************************************** localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0 2.6 Configure the OAM network ''''''''''''''''''''''''''''' Example: :: controller-0:~$ source /etc/platform/openrc [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | eno1 | +-----------------+--------------------------------------+ | iftype | ethernet | +-----------------+--------------------------------------+ | ports | [u'eno1'] | +-----------------+--------------------------------------+ | imac | 3c:a8:2a:1d:31:d0 | +-----------------+--------------------------------------+ | imtu | 1500 | +-----------------+--------------------------------------+ | ifclass | platform | +-----------------+--------------------------------------+ | aemode | None | +-----------------+--------------------------------------+ | schedpolicy | None | +-----------------+--------------------------------------+ | txhashpolicy | None | +-----------------+--------------------------------------+ | uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 | +-----------------+--------------------------------------+ | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-----------------+--------------------------------------+ | vlan_id | None | +-----------------+--------------------------------------+ | uses | [] | +-----------------+--------------------------------------+ | used_by | [] | +-----------------+--------------------------------------+ | created_at | 2020-06-03T08:21:17.526303+00:00 | +-----------------+--------------------------------------+ | updated_at | 2020-06-03T10:02:54.682501+00:00 | +-----------------+--------------------------------------+ | sriov_numvfs | 0 | +-----------------+--------------------------------------+ | sriov_vf_driver | None | +-----------------+--------------------------------------+ | accelerated | [False] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | hostname | controller-0 | +--------------+--------------------------------------+ | uuid | 32592cc3-9d66-4961-a51e-23686da00290 | +--------------+--------------------------------------+ | ifname | eno1 | +--------------+--------------------------------------+ | network_name | oam | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0 +--------------+--------------------------------------+--------+--------------+ | hostname | uuid | ifname | network_name | +--------------+--------------------------------------+--------+--------------+ | controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt | +--------------+--------------------------------------+--------+--------------+ | controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam | +--------------+--------------------------------------+--------+--------------+ | controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host | +--------------+--------------------------------------+--------+--------------+ 2.7 Configure the NTP server '''''''''''''''''''''''''''' Example: :: [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 | +--------------+--------------------------------------+ | ntpservers | 0.pool.ntp.org,1.pool.ntp.org | +--------------+--------------------------------------+ | isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 | +--------------+--------------------------------------+ | created_at | 2020-06-03T08:19:47.812249+00:00 | +--------------+--------------------------------------+ | updated_at | None | +--------------+--------------------------------------+ 2.8 Configure the Storage by using the second hard drive as ceph backend '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Example: :: [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path | | | de | num | type | gib | gib | | id | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | | | | | | | | | | | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ | | | | | | | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} +------------------+-------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------+ | osdid | 0 | +------------------+-------------------------------------------------------+ | function | osd | +------------------+-------------------------------------------------------+ | state | configuring-on-unlock | +------------------+-------------------------------------------------------+ | journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | +------------------+-------------------------------------------------------+ | journal_size_gib | 1024 | +------------------+-------------------------------------------------------+ | journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | +------------------+-------------------------------------------------------+ | journal_node | /dev/sdb2 | +------------------+-------------------------------------------------------+ | uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | +------------------+-------------------------------------------------------+ | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | +------------------+-------------------------------------------------------+ | idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | +------------------+-------------------------------------------------------+ | tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d | +------------------+-------------------------------------------------------+ | tier_name | storage | +------------------+-------------------------------------------------------+ | created_at | 2020-06-03T10:03:55.645744+00:00 | +------------------+-------------------------------------------------------+ | updated_at | None | +------------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage | +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ 2.9 Configure the SR-IOV enabled network '''''''''''''''''''''''''''''''''''''''' It needs : - assign the label of SR-IOV - change the CPU policy to static - change the hugepage setting - create the data network - assign the SR-IOV interface to data network Example: :: [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0 [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b | +-------------+--------------------------------------+ | host_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-------------+--------------------------------------+ | label_key | sriovdp | +-------------+--------------------------------------+ | label_value | enabled | +-------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | uuid | 4936f32b-6939-4115-b643-846bd50e7738 | +-------------+--------------------------------------+ | host_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-------------+--------------------------------------+ | label_key | kube-cpu-mgr-policy | +-------------+--------------------------------------+ | label_value | static | +-------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0 +-------------------------------------+--------------------------------------+ | Property | Value | +-------------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 0 | +-------------------------------------+--------------------------------------+ | Platform (MiB) | 7000 | +-------------------------------------+--------------------------------------+ | Available (MiB) | 0 | +-------------------------------------+--------------------------------------+ | Huge Pages Configured | True | +-------------------------------------+--------------------------------------+ | vSwitch Huge Pages: Size (MiB) | 1024 | +-------------------------------------+--------------------------------------+ | Total | 0 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | Required | None | +-------------------------------------+--------------------------------------+ | Application Pages (4K): Total | None | +-------------------------------------+--------------------------------------+ | Application Huge Pages (2M): Total | 0 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | Application Huge Pages (1G): Total | 0 | +-------------------------------------+--------------------------------------+ | Total Pending | 7 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 | +-------------------------------------+--------------------------------------+ | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-------------------------------------+--------------------------------------+ | inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 | +-------------------------------------+--------------------------------------+ | created_at | 2020-06-03T08:21:27.468357+00:00 | +-------------------------------------+--------------------------------------+ | updated_at | 2020-06-03T10:10:49.159080+00:00 | +-------------------------------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1 +-------------------------------------+--------------------------------------+ | Property | Value | +-------------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 0 | +-------------------------------------+--------------------------------------+ | Platform (MiB) | 1000 | +-------------------------------------+--------------------------------------+ | Available (MiB) | 0 | +-------------------------------------+--------------------------------------+ | Huge Pages Configured | True | +-------------------------------------+--------------------------------------+ | vSwitch Huge Pages: Size (MiB) | 1024 | +-------------------------------------+--------------------------------------+ | Total | 0 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | Required | None | +-------------------------------------+--------------------------------------+ | Application Pages (4K): Total | None | +-------------------------------------+--------------------------------------+ | Application Huge Pages (2M): Total | 0 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | Application Huge Pages (1G): Total | 0 | +-------------------------------------+--------------------------------------+ | Total Pending | 7 | +-------------------------------------+--------------------------------------+ | Available | 0 | +-------------------------------------+--------------------------------------+ | uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 | +-------------------------------------+--------------------------------------+ | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-------------------------------------+--------------------------------------+ | inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f | +-------------------------------------+--------------------------------------+ | created_at | 2020-06-03T08:21:27.566279+00:00 | +-------------------------------------+--------------------------------------+ | updated_at | 2020-06-03T10:10:49.220907+00:00 | +-------------------------------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0' [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1' [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0 [sysadmin@controller-0 ~(keystone_admin)]$ [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | id | 1 | +--------------+--------------------------------------+ | uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b | +--------------+--------------------------------------+ | name | physnet0 | +--------------+--------------------------------------+ | network_type | flat | +--------------+--------------------------------------+ | mtu | 1500 | +--------------+--------------------------------------+ | description | None | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | id | 2 | +--------------+--------------------------------------+ | uuid | 899ca8ba-6800-4d62-aed4-3770878da738 | +--------------+--------------------------------------+ | name | physnet1 | +--------------+--------------------------------------+ | network_type | flat | +--------------+--------------------------------------+ | mtu | 1500 | +--------------+--------------------------------------+ | description | None | +--------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ | 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] | +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | sriov0 | +-----------------+--------------------------------------+ | iftype | ethernet | +-----------------+--------------------------------------+ | ports | [u'ens2f0'] | +-----------------+--------------------------------------+ | imac | 00:1b:21:b3:46:2c | +-----------------+--------------------------------------+ | imtu | 1500 | +-----------------+--------------------------------------+ | ifclass | pci-sriov | +-----------------+--------------------------------------+ | aemode | None | +-----------------+--------------------------------------+ | schedpolicy | None | +-----------------+--------------------------------------+ | txhashpolicy | None | +-----------------+--------------------------------------+ | uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 | +-----------------+--------------------------------------+ | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | +-----------------+--------------------------------------+ | vlan_id | None | +-----------------+--------------------------------------+ | uses | [] | +-----------------+--------------------------------------+ | used_by | [] | +-----------------+--------------------------------------+ | created_at | 2020-06-03T12:34:11.633254+00:00 | +-----------------+--------------------------------------+ | updated_at | 2020-06-04T02:01:28.168127+00:00 | +-----------------+--------------------------------------+ | sriov_numvfs | 6 | +-----------------+--------------------------------------+ | sriov_vf_driver | vfio | +-----------------+--------------------------------------+ | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0} +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | hostname | controller-0 | +------------------+--------------------------------------+ | uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 | +------------------+--------------------------------------+ | ifname | sriov0 | +------------------+--------------------------------------+ | datanetwork_name | physnet0 | +------------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF +-----------------+--------------------------------------+ | Property | Value | +-----------------+--------------------------------------+ | ifname | sriov1 | +-----------------+--------------------------------------+ | iftype | ethernet | +-----------------+--------------------------------------+ | ports | [u'ens2f1'] | +-----------------+--------------------------------------+ | imac | 00:1b:21:b3:46:2d | +-----------------+--------------------------------------+ | imtu | 1500 | +-----------------+--------------------------------------+ | ifclass | pci-sriov | +-----------------+--------------------------------------+ | aemode | None | +-----------------+--------------------------------------+ | schedpolicy | None | +-----------------+--------------------------------------+ | txhashpolicy | None | +-----------------+--------------------------------------+ | uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 | | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | | vlan_id | None | | uses | [] | | used_by | [] | | created_at | 2020-06-03T12:34:12.072624+00:00 | | updated_at | 2020-06-04T02:03:51.405322+00:00 | | sriov_numvfs | 6 | | sriov_vf_driver | vfio | | accelerated | [True] | +-----------------+--------------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1} +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | hostname | controller-0 | +------------------+--------------------------------------+ | uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f | +------------------+--------------------------------------+ | ifname | sriov1 | +------------------+--------------------------------------+ | datanetwork_name | physnet1 | +------------------+--------------------------------------+ 2.10 Unlock the server '''''''''''''''''''''' Example: :: [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0 +-----------------------+-------------------------------------------------+ | Property | Value | +-----------------------+-------------------------------------------------+ | action | none | +-----------------------+-------------------------------------------------+ | administrative | locked | +-----------------------+-------------------------------------------------+ | availability | online | +-----------------------+-------------------------------------------------+ | bm_ip | None | +-----------------------+-------------------------------------------------+ | bm_type | none | +-----------------------+-------------------------------------------------+ | bm_username | None | +-----------------------+-------------------------------------------------+ | boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 | +-----------------------+-------------------------------------------------+ | capabilities | {u'stor_function': u'monitor'} | +-----------------------+-------------------------------------------------+ | clock_synchronization | ntp | +-----------------------+-------------------------------------------------+ | config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea | +-----------------------+-------------------------------------------------+ | config_status | Config out-of-date | +-----------------------+-------------------------------------------------+ | config_target | bdaf20c7-c032-4aa2-839d-93da92051bea | +-----------------------+-------------------------------------------------+ | console | tty0 | +-----------------------+-------------------------------------------------+ | created_at | 2020-06-03T08:20:59.372602+00:00 | +-----------------------+-------------------------------------------------+ | hostname | controller-0 | +-----------------------+-------------------------------------------------+ | id | 1 | +-----------------------+-------------------------------------------------+ | install_output | graphical | +-----------------------+-------------------------------------------------+ | install_state | None | +-----------------------+-------------------------------------------------+ | install_state_info | None | +-----------------------+-------------------------------------------------+ | inv_state | inventoried | +-----------------------+-------------------------------------------------+ | invprovision | provisioning | +-----------------------+-------------------------------------------------+ | location | {} | +-----------------------+-------------------------------------------------+ | mgmt_ip | 192.168.204.2 | +-----------------------+-------------------------------------------------+ | mgmt_mac | 00:00:00:00:00:00 | +-----------------------+-------------------------------------------------+ | operational | disabled | +-----------------------+-------------------------------------------------+ | personality | controller | +-----------------------+-------------------------------------------------+ | reserved | False | +-----------------------+-------------------------------------------------+ | rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 | +-----------------------+-------------------------------------------------+ | serialid | None | +-----------------------+-------------------------------------------------+ | software_load | 19.12 | +-----------------------+-------------------------------------------------+ | subfunction_avail | online | +-----------------------+-------------------------------------------------+ | subfunction_oper | disabled | +-----------------------+-------------------------------------------------+ | subfunctions | controller,worker | +-----------------------+-------------------------------------------------+ | task | Unlocking | +-----------------------+-------------------------------------------------+ | tboot | false | +-----------------------+-------------------------------------------------+ | ttys_dcd | None | +-----------------------+-------------------------------------------------+ | updated_at | 2020-06-04T02:04:59.514854+00:00 | +-----------------------+-------------------------------------------------+ | uptime | 49475 | +-----------------------+-------------------------------------------------+ | uuid | 08c23843-afa4-4b80-a285-be126365276e | +-----------------------+-------------------------------------------------+ | vim_progress_status | None | +-----------------------+-------------------------------------------------+ 3. Simple use case for sriov ```````````````````````````` 3.1 After controller-0 is rebooted and up running, download the DPDK '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz Password: --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ... Connecting to fast.dpdk.org|151.101.2.49|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10251680 (9.8M) [application/octet-stream] Saving to: ‘dpdk-17.11.10.tar.xz’ dpdk-17.11.10.tar.xz 100% [========================================================================================>] 9.78M 1.48MB/s in 6.8s 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680] sudo tar xvf dpdk-17.11.10.tar.xz sudo ln -s dpdk-stable-17.11.10 dpdk-stable 3.2 Prepare the yaml file for the network assignment container '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The following the exmaple of the yaml file: :: [sysadmin@controller-0 sriov(keystone_admin)]$ cat < netdef-data-dpdk.yaml > apiVersion: "k8s.cni.cncf.io/v1" > kind: NetworkAttachmentDefinition > metadata: > name: sriov-data-dpdk-0 > annotations: > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0 > spec: > config: '{ > "type": "sriov", > "name": "sriov-data-dpdk-0" > }' > > --- > apiVersion: "k8s.cni.cncf.io/v1" > kind: NetworkAttachmentDefinition > metadata: > name: sriov-data-dpdk-1 > annotations: > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1 > spec: > config: '{ > "type": "sriov", > "name": "sriov-data-dpdk-1" > }' > EOF 3.3 Run the network assignent container for the 2 VFs ''''''''''''''''''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created 3.4 Prepare the VF container yaml file '''''''''''''''''''''''''''''''''''''' :: [sysadmin@controller-0 sriov(keystone_admin)]$ cat < pod-with-dpdk-vfs-0.yaml > apiVersion: v1 > kind: Pod metadata: > metadata: > name: pod-with-dpdk-vfs-0 > annotations: > k8s.v1.cni.cncf.io/networks: '[ > { "name": "sriov-data-dpdk-0" }, { "name": "sriov-data-dpdk-1" } > { "name": "sriov-data-dpdk-1" } > ]' > spec: > restartPolicy: Never > containers: > - name: pod-with-dpdk-vfs-0 > image: wrsnfv/ubuntu-dpdk-build:v0.3 > env: > - name: RTE_SDK > value: "/usr/src/dpdk" > command: > - sleep > - infinity > stdin: true > tty: true > securityContext: > privileged: true > capabilities: > add: > - ALL > resources: > requests: > cpu: 4 > memory: 4Gi > intel.com/pci_sriov_net_physnet0: '1' > intel.com/pci_sriov_net_physnet1: '1' > limits: > cpu: 4 > hugepages-1Gi: 2Gi > memory: 4Gi > intel.com/pci_sriov_net_physnet0: '1' > intel.com/pci_sriov_net_physnet1: '1' > volumeMounts: > - mountPath: /mnt/huge-1048576kB > name: hugepage > - name: dpdk-volume > mountPath: /usr/src/dpdk > - name: lib-volume > mountPath: /lib/modules > - name: src-volume > mountPath: /usr/src/ > volumes: > - name: hugepage > emptyDir: > medium: HugePages > - name: dpdk-volume > hostPath: > path: /opt/dpdk-stable/ > - name: lib-volume > hostPath: > path: /lib/modules > - name: src-volume > hostPath: > path: /usr/src/ > EOF 3.5 Run the VF container '''''''''''''''''''''''' Start the VF container. :: [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml pod/pod-with-dpdk-vfs-0 created [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod NAME READY STATUS RESTARTS AGE pod-with-dpdk-vfs-0 1/1 Running 0 6m40s Login the VF container :: kubectl exec -it pod-with-dpdk-vfs-0 -- bash Build the DPDK :: cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/conf.o HOSTCC scripts/kconfig/confdata.o HOSTCC scripts/kconfig/expr.o HOSTCC scripts/kconfig/symbol.o HOSTCC scripts/kconfig/preprocess.o HOSTCC scripts/kconfig/zconf.lex.o HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf --syncconfig Kconfig HOSTCC arch/x86/tools/relocs_32.o HOSTCC arch/x86/tools/relocs_64.o HOSTCC arch/x86/tools/relocs_common.o HOSTLD arch/x86/tools/relocs HOSTCC scripts/genksyms/genksyms.o YACC scripts/genksyms/parse.tab.c HOSTCC scripts/genksyms/parse.tab.o LEX scripts/genksyms/lex.lex.c YACC scripts/genksyms/parse.tab.h HOSTCC scripts/genksyms/lex.lex.o HOSTLD scripts/genksyms/genksyms HOSTCC scripts/bin2c HOSTCC scripts/kallsyms HOSTCC scripts/conmakehash HOSTCC scripts/recordmcount HOSTCC scripts/sortextable HOSTCC scripts/asn1_compiler HOSTCC scripts/sign-file HOSTCC scripts/extract-cert CC scripts/mod/empty.o HOSTCC scripts/mod/mk_elfconfig MKELF scripts/mod/elfconfig.h HOSTCC scripts/mod/modpost.o CC scripts/mod/devicetable-offsets.s UPD scripts/mod/devicetable-offsets.h HOSTCC scripts/mod/file2alias.o HOSTCC scripts/mod/sumversion.o HOSTLD scripts/mod/modpost CC kernel/bounds.s CC arch/x86/kernel/asm-offsets.s CALL scripts/checksyscalls.sh Build the test_pmd application :: cd $RTE_SDK ./usertools/dpdk-setup.sh Option: 14 CC config.o CC iofwd.o CC macfwd.o CC macswap.o CC flowgen.o CC rxonly.o CC txonly.o CC csumonly.o CC icmpecho.o CC tm.o LD testpmd INSTALL-APP testpmd INSTALL-MAP testpmd.map == Build app/proc_info CC main.o LD dpdk-procinfo INSTALL-APP dpdk-procinfo INSTALL-MAP dpdk-procinfo.map == Build app/pdump CC main.o LD dpdk-pdump INSTALL-APP dpdk-pdump INSTALL-MAP dpdk-pdump.map == Build app/test-crypto-perf CC main.o CC cperf_ops.o CC cperf_options_parsing.o CC cperf_test_vectors.o CC cperf_test_throughput.o CC cperf_test_latency.o CC cperf_test_pmd_cyclecount.o CC cperf_test_verify.o CC cperf_test_vector_parsing.o CC cperf_test_common.o LD dpdk-test-crypto-perf INSTALL-APP dpdk-test-crypto-perf INSTALL-MAP dpdk-test-crypto-perf.map == Build app/test-eventdev CC evt_main.o CC evt_options.o CC evt_test.o CC parser.o CC test_order_common.o CC test_order_queue.o CC test_order_atq.o CC test_perf_common.o CC test_perf_queue.o CC test_perf_atq.o LD dpdk-test-eventdev INSTALL-APP dpdk-test-eventdev INSTALL-MAP dpdk-test-eventdev.map Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined ------------------------------------------------------------------------------ RTE_TARGET exported as x86_64-native-linuxapp-gcc ------------------------------------------------------------------------------ Press enter to continue ... Check the VF PCI information: :: root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address: :: [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn* lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn* lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02 Get back to pod and run the test_pmd :: cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/ taskset -p --cpu-list 1 pid 1's current affinity list: 2,3,18,19 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto- start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth- peer=1,"9e:fd:e6:dd:c1:01" EAL: Detected 32 lcore(s) EAL: No free hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:05:11.0 on NUMA socket 0 EAL: probe driver: 8086:10ed net_ixgbe_vf EAL: using IOMMU type 1 (Type 1) EAL: PCI device 0000:05:11.1 on NUMA socket 0 EAL: probe driver: 8086:10ed net_ixgbe_vf Auto-start selected Ports to start sending a burst of packets first Warning: lsc_interrupt needs to be off when using tx_first. Disabling. USER1: create a new mbuf pool : n=155456, size=2176, socket=0 Configuring Port 0 (socket 0) Port 0: 9E:FD:E6:DD:C1:01 Configuring Port 1 (socket 0) Port 1: 9E:FD:E6:DD:C1:02 Checking link statuses... Port0 Link Up. speed 10000 Mbps- full-duplex Port1 Link Up. speed 10000 Mbps- full-duplex Done No commandline core given, start packet forwarding io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages disabled Logical Core 3 (socket 0) forwards packets on 2 streams: RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02 io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=2 port 0: CRC stripping enabled RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 1: CRC stripping enabled RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 Port statistics ==================================== ######################## NIC statistics for port 0 ######################## RX-packets: 56 RX-missed: 0 RX-bytes: 4096 RX-errors: 0 RX-nombuf: 0 TX-packets: 64 TX-errors: 0 TX-bytes: 4096 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 432 RX-missed: 0 RX-bytes: 27712 RX-errors: 0 RX-nombuf: 0 TX-packets: 461 TX-errors: 0 TX-bytes: 30080 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ Port statistics ==================================== ######################## NIC statistics for port 0 ######################## RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344 RX-errors: 0 RX-nombuf: 0 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376 Throughput (since last show) Rx-pps: 7068409 Tx-pps: 7091206 ############################################################################ References ---------- - `StarlingX`_ .. _`StarlingX`: https://docs.starlingx.io/