Installation Guide¶
Abstract¶
This document describes how to install O-RAN INF image, example configuration for better real time performance, and example deployment of Kubernetes cluster and plugins.
The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux and container technology.
Version history
Date |
Ver. |
Author |
Comment |
2019-11-02 |
1.0.0 |
Jackie Huang |
Initail version |
2020-06-03 |
2.0.0 (Bronze) |
Xiaohua Zhang |
Bronze version |
Preface¶
Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
Hardware Requirements¶
Following minimum hardware requirements must be met for installation of O-RAN INF image:
HW Aspect |
Requirement |
# of servers |
1 |
CPU |
4 |
RAM |
16G |
Disk |
500G * 2 |
NICs |
2 |
Software Installation and Deployment¶
1. Installation from the O-RAN INF ISO image¶
Please see the README.md file for how to build the image.
The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso
1.1 Burn the image to the USB device¶
Assume the the usb device is /dev/sdX here
$ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
1.2 Install the server¶
Reboot the target from the USB device.
Select “Graphics console install” or “Serial console install” and press ENTER
Remove the USB device and press ENTER to reboot
2. Configuration and initialize the bootstrap¶
2.1 First Login¶
Example:
ifconfig eno1 128.224.180.14/24 up
ip route add default via 128.224.180.1
ping 8.8.8.8
2.4 Prepare the localhost.yml for bootstrap¶
Example:
system_mode: simplex
external_oam_subnet: 128.224.180.0/24
external_oam_gateway_address: 128.224.180.1
external_oam_floating_address: 128.224.180.14
admin_username: admin
admin_password: Wind123!
ansible_become_pass: Wind123!
docker_http_proxy: http://xxx.xxx.xxx.xxx:8080
docker_https_proxy: https://xxx.xxx.xxx.xxx:8080
2.5 Run the bootstrap¶
Use the following command:
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
After the bootstrap successfully finish, it will show as following:
PLAY RECAP ******************************************************************************************************************************************
localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0
2.6 Configure the OAM network¶
Example:
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | eno1 |
+-----------------+--------------------------------------+
| iftype | ethernet |
+-----------------+--------------------------------------+
| ports | [u'eno1'] |
+-----------------+--------------------------------------+
| imac | 3c:a8:2a:1d:31:d0 |
+-----------------+--------------------------------------+
| imtu | 1500 |
+-----------------+--------------------------------------+
| ifclass | platform |
+-----------------+--------------------------------------+
| aemode | None |
+-----------------+--------------------------------------+
| schedpolicy | None |
+-----------------+--------------------------------------+
| txhashpolicy | None |
+-----------------+--------------------------------------+
| uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 |
+-----------------+--------------------------------------+
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-----------------+--------------------------------------+
| vlan_id | None |
+-----------------+--------------------------------------+
| uses | [] |
+-----------------+--------------------------------------+
| used_by | [] |
+-----------------+--------------------------------------+
| created_at | 2020-06-03T08:21:17.526303+00:00 |
+-----------------+--------------------------------------+
| updated_at | 2020-06-03T10:02:54.682501+00:00 |
+-----------------+--------------------------------------+
| sriov_numvfs | 0 |
+-----------------+--------------------------------------+
| sriov_vf_driver | None |
+-----------------+--------------------------------------+
| accelerated | [False] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
+--------------+--------------------------------------+
| uuid | 32592cc3-9d66-4961-a51e-23686da00290 |
+--------------+--------------------------------------+
| ifname | eno1 |
+--------------+--------------------------------------+
| network_name | oam |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+--------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+--------+--------------+
| controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt |
+--------------+--------------------------------------+--------+--------------+
| controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam |
+--------------+--------------------------------------+--------+--------------+
| controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host |
+--------------+--------------------------------------+--------+--------------+
2.7 Configure the NTP server¶
Example:
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 |
+--------------+--------------------------------------+
| ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
+--------------+--------------------------------------+
| isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 |
+--------------+--------------------------------------+
| created_at | 2020-06-03T08:19:47.812249+00:00 |
+--------------+--------------------------------------+
| updated_at | None |
+--------------+--------------------------------------+
2.8 Configure the Storage by using the second hard drive as ceph backend¶
Example:
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path |
| | de | num | type | gib | gib | | id | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| | | | | | | | | |
| 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
| | | | | | | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 0 |
+------------------+-------------------------------------------------------+
| function | osd |
+------------------+-------------------------------------------------------+
| state | configuring-on-unlock |
+------------------+-------------------------------------------------------+
| journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
+------------------+-------------------------------------------------------+
| journal_size_gib | 1024 |
+------------------+-------------------------------------------------------+
| journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 |
+------------------+-------------------------------------------------------+
| journal_node | /dev/sdb2 |
+------------------+-------------------------------------------------------+
| uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
+------------------+-------------------------------------------------------+
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+------------------+-------------------------------------------------------+
| idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 |
+------------------+-------------------------------------------------------+
| tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d |
+------------------+-------------------------------------------------------+
| tier_name | storage |
+------------------+-------------------------------------------------------+
| created_at | 2020-06-03T10:03:55.645744+00:00 |
+------------------+-------------------------------------------------------+
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
2.9 Configure the SR-IOV enabled network¶
- It needs :
assign the label of SR-IOV
change the CPU policy to static
change the hugepage setting
create the data network
assign the SR-IOV interface to data network
Example:
[sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b |
+-------------+--------------------------------------+
| host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-------------+--------------------------------------+
| label_key | sriovdp |
+-------------+--------------------------------------+
| label_value | enabled |
+-------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | 4936f32b-6939-4115-b643-846bd50e7738 |
+-------------+--------------------------------------+
| host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-------------+--------------------------------------+
| label_key | kube-cpu-mgr-policy |
+-------------+--------------------------------------+
| label_value | static |
+-------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| Memory: Usable Total (MiB) | 0 |
+-------------------------------------+--------------------------------------+
| Platform (MiB) | 7000 |
+-------------------------------------+--------------------------------------+
| Available (MiB) | 0 |
+-------------------------------------+--------------------------------------+
| Huge Pages Configured | True |
+-------------------------------------+--------------------------------------+
| vSwitch Huge Pages: Size (MiB) | 1024 |
+-------------------------------------+--------------------------------------+
| Total | 0 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| Required | None |
+-------------------------------------+--------------------------------------+
| Application Pages (4K): Total | None |
+-------------------------------------+--------------------------------------+
| Application Huge Pages (2M): Total | 0 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| Application Huge Pages (1G): Total | 0 |
+-------------------------------------+--------------------------------------+
| Total Pending | 7 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 |
+-------------------------------------+--------------------------------------+
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-------------------------------------+--------------------------------------+
| inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 |
+-------------------------------------+--------------------------------------+
| created_at | 2020-06-03T08:21:27.468357+00:00 |
+-------------------------------------+--------------------------------------+
| updated_at | 2020-06-03T10:10:49.159080+00:00 |
+-------------------------------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| Memory: Usable Total (MiB) | 0 |
+-------------------------------------+--------------------------------------+
| Platform (MiB) | 1000 |
+-------------------------------------+--------------------------------------+
| Available (MiB) | 0 |
+-------------------------------------+--------------------------------------+
| Huge Pages Configured | True |
+-------------------------------------+--------------------------------------+
| vSwitch Huge Pages: Size (MiB) | 1024 |
+-------------------------------------+--------------------------------------+
| Total | 0 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| Required | None |
+-------------------------------------+--------------------------------------+
| Application Pages (4K): Total | None |
+-------------------------------------+--------------------------------------+
| Application Huge Pages (2M): Total | 0 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| Application Huge Pages (1G): Total | 0 |
+-------------------------------------+--------------------------------------+
| Total Pending | 7 |
+-------------------------------------+--------------------------------------+
| Available | 0 |
+-------------------------------------+--------------------------------------+
| uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 |
+-------------------------------------+--------------------------------------+
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-------------------------------------+--------------------------------------+
| inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f |
+-------------------------------------+--------------------------------------+
| created_at | 2020-06-03T08:21:27.566279+00:00 |
+-------------------------------------+--------------------------------------+
| updated_at | 2020-06-03T10:10:49.220907+00:00 |
+-------------------------------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0'
[sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1'
[sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| id | 1 |
+--------------+--------------------------------------+
| uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b |
+--------------+--------------------------------------+
| name | physnet0 |
+--------------+--------------------------------------+
| network_type | flat |
+--------------+--------------------------------------+
| mtu | 1500 |
+--------------+--------------------------------------+
| description | None |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| id | 2 |
+--------------+--------------------------------------+
| uuid | 899ca8ba-6800-4d62-aed4-3770878da738 |
+--------------+--------------------------------------+
| name | physnet1 |
+--------------+--------------------------------------+
| network_type | flat |
+--------------+--------------------------------------+
| mtu | 1500 |
+--------------+--------------------------------------+
| description | None |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| uuid | name | type | pci address | device | processor | accelerated | device type |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
| 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | sriov0 |
+-----------------+--------------------------------------+
| iftype | ethernet |
+-----------------+--------------------------------------+
| ports | [u'ens2f0'] |
+-----------------+--------------------------------------+
| imac | 00:1b:21:b3:46:2c |
+-----------------+--------------------------------------+
| imtu | 1500 |
+-----------------+--------------------------------------+
| ifclass | pci-sriov |
+-----------------+--------------------------------------+
| aemode | None |
+-----------------+--------------------------------------+
| schedpolicy | None |
+-----------------+--------------------------------------+
| txhashpolicy | None |
+-----------------+--------------------------------------+
| uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 |
+-----------------+--------------------------------------+
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-----------------+--------------------------------------+
| vlan_id | None |
+-----------------+--------------------------------------+
| uses | [] |
+-----------------+--------------------------------------+
| used_by | [] |
+-----------------+--------------------------------------+
| created_at | 2020-06-03T12:34:11.633254+00:00 |
+-----------------+--------------------------------------+
| updated_at | 2020-06-04T02:01:28.168127+00:00 |
+-----------------+--------------------------------------+
| sriov_numvfs | 6 |
+-----------------+--------------------------------------+
| sriov_vf_driver | vfio |
+-----------------+--------------------------------------+
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0}
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| hostname | controller-0 |
+------------------+--------------------------------------+
| uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 |
+------------------+--------------------------------------+
| ifname | sriov0 |
+------------------+--------------------------------------+
| datanetwork_name | physnet0 |
+------------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | sriov1 |
+-----------------+--------------------------------------+
| iftype | ethernet |
+-----------------+--------------------------------------+
| ports | [u'ens2f1'] |
+-----------------+--------------------------------------+
| imac | 00:1b:21:b3:46:2d |
+-----------------+--------------------------------------+
| imtu | 1500 |
+-----------------+--------------------------------------+
| ifclass | pci-sriov |
+-----------------+--------------------------------------+
| aemode | None |
+-----------------+--------------------------------------+
| schedpolicy | None |
+-----------------+--------------------------------------+
| txhashpolicy | None |
+-----------------+--------------------------------------+
| uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 |
| ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-06-03T12:34:12.072624+00:00 |
| updated_at | 2020-06-04T02:03:51.405322+00:00 |
| sriov_numvfs | 6 |
| sriov_vf_driver | vfio |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1}
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| hostname | controller-0 |
+------------------+--------------------------------------+
| uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f |
+------------------+--------------------------------------+
| ifname | sriov1 |
+------------------+--------------------------------------+
| datanetwork_name | physnet1 |
+------------------+--------------------------------------+
2.10 Unlock the server¶
Example:
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
+-----------------------+-------------------------------------------------+
| Property | Value |
+-----------------------+-------------------------------------------------+
| action | none |
+-----------------------+-------------------------------------------------+
| administrative | locked |
+-----------------------+-------------------------------------------------+
| availability | online |
+-----------------------+-------------------------------------------------+
| bm_ip | None |
+-----------------------+-------------------------------------------------+
| bm_type | none |
+-----------------------+-------------------------------------------------+
| bm_username | None |
+-----------------------+-------------------------------------------------+
| boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
+-----------------------+-------------------------------------------------+
| capabilities | {u'stor_function': u'monitor'} |
+-----------------------+-------------------------------------------------+
| clock_synchronization | ntp |
+-----------------------+-------------------------------------------------+
| config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea |
+-----------------------+-------------------------------------------------+
| config_status | Config out-of-date |
+-----------------------+-------------------------------------------------+
| config_target | bdaf20c7-c032-4aa2-839d-93da92051bea |
+-----------------------+-------------------------------------------------+
| console | tty0 |
+-----------------------+-------------------------------------------------+
| created_at | 2020-06-03T08:20:59.372602+00:00 |
+-----------------------+-------------------------------------------------+
| hostname | controller-0 |
+-----------------------+-------------------------------------------------+
| id | 1 |
+-----------------------+-------------------------------------------------+
| install_output | graphical |
+-----------------------+-------------------------------------------------+
| install_state | None |
+-----------------------+-------------------------------------------------+
| install_state_info | None |
+-----------------------+-------------------------------------------------+
| inv_state | inventoried |
+-----------------------+-------------------------------------------------+
| invprovision | provisioning |
+-----------------------+-------------------------------------------------+
| location | {} |
+-----------------------+-------------------------------------------------+
| mgmt_ip | 192.168.204.2 |
+-----------------------+-------------------------------------------------+
| mgmt_mac | 00:00:00:00:00:00 |
+-----------------------+-------------------------------------------------+
| operational | disabled |
+-----------------------+-------------------------------------------------+
| personality | controller |
+-----------------------+-------------------------------------------------+
| reserved | False |
+-----------------------+-------------------------------------------------+
| rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
+-----------------------+-------------------------------------------------+
| serialid | None |
+-----------------------+-------------------------------------------------+
| software_load | 19.12 |
+-----------------------+-------------------------------------------------+
| subfunction_avail | online |
+-----------------------+-------------------------------------------------+
| subfunction_oper | disabled |
+-----------------------+-------------------------------------------------+
| subfunctions | controller,worker |
+-----------------------+-------------------------------------------------+
| task | Unlocking |
+-----------------------+-------------------------------------------------+
| tboot | false |
+-----------------------+-------------------------------------------------+
| ttys_dcd | None |
+-----------------------+-------------------------------------------------+
| updated_at | 2020-06-04T02:04:59.514854+00:00 |
+-----------------------+-------------------------------------------------+
| uptime | 49475 |
+-----------------------+-------------------------------------------------+
| uuid | 08c23843-afa4-4b80-a285-be126365276e |
+-----------------------+-------------------------------------------------+
| vim_progress_status | None |
+-----------------------+-------------------------------------------------+
3. Simple use case for sriov¶
3.1 After controller-0 is rebooted and up running, download the DPDK¶
[sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
[sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Password:
--2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10251680 (9.8M) [application/octet-stream]
Saving to: ‘dpdk-17.11.10.tar.xz’
dpdk-17.11.10.tar.xz 100%
[========================================================================================>] 9.78M
1.48MB/s in 6.8s
2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
sudo tar xvf dpdk-17.11.10.tar.xz
sudo ln -s dpdk-stable-17.11.10 dpdk-stable
3.2 Prepare the yaml file for the network assignment container¶
The following the exmaple of the yaml file:
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-0
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-0"
> }'
>
> ---
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-1
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-1"
> }'
> EOF
3.3 Run the network assignent container for the 2 VFs¶
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
3.4 Prepare the VF container yaml file¶
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
> apiVersion: v1
> kind: Pod
metadata:
> metadata:
> name: pod-with-dpdk-vfs-0
> annotations:
> k8s.v1.cni.cncf.io/networks: '[
> { "name": "sriov-data-dpdk-0" },
{ "name": "sriov-data-dpdk-1" }
> { "name": "sriov-data-dpdk-1" }
> ]'
> spec:
> restartPolicy: Never
> containers:
> - name: pod-with-dpdk-vfs-0
> image: wrsnfv/ubuntu-dpdk-build:v0.3
> env:
> - name: RTE_SDK
> value: "/usr/src/dpdk"
> command:
> - sleep
> - infinity
> stdin: true
> tty: true
> securityContext:
> privileged: true
> capabilities:
> add:
> - ALL
> resources:
> requests:
> cpu: 4
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> limits:
> cpu: 4
> hugepages-1Gi: 2Gi
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> volumeMounts:
> - mountPath: /mnt/huge-1048576kB
> name: hugepage
> - name: dpdk-volume
> mountPath: /usr/src/dpdk
> - name: lib-volume
> mountPath: /lib/modules
> - name: src-volume
> mountPath: /usr/src/
> volumes:
> - name: hugepage
> emptyDir:
> medium: HugePages
> - name: dpdk-volume
> hostPath:
> path: /opt/dpdk-stable/
> - name: lib-volume
> hostPath:
> path: /lib/modules
> - name: src-volume
> hostPath:
> path: /usr/src/
> EOF
3.5 Run the VF container¶
Start the VF container.
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
pod/pod-with-dpdk-vfs-0 created
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
Login the VF container
kubectl exec -it pod-with-dpdk-vfs-0 -- bash
Build the DPDK
cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/zconf.lex.o
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --syncconfig Kconfig
HOSTCC arch/x86/tools/relocs_32.o
HOSTCC arch/x86/tools/relocs_64.o
HOSTCC arch/x86/tools/relocs_common.o
HOSTLD arch/x86/tools/relocs
HOSTCC scripts/genksyms/genksyms.o
YACC scripts/genksyms/parse.tab.c
HOSTCC scripts/genksyms/parse.tab.o
LEX scripts/genksyms/lex.lex.c
YACC scripts/genksyms/parse.tab.h
HOSTCC scripts/genksyms/lex.lex.o
HOSTLD scripts/genksyms/genksyms
HOSTCC scripts/bin2c
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
HOSTCC scripts/recordmcount
HOSTCC scripts/sortextable
HOSTCC scripts/asn1_compiler
HOSTCC scripts/sign-file
HOSTCC scripts/extract-cert
CC scripts/mod/empty.o
HOSTCC scripts/mod/mk_elfconfig
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/modpost.o
CC scripts/mod/devicetable-offsets.s
UPD scripts/mod/devicetable-offsets.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
CC kernel/bounds.s
CC arch/x86/kernel/asm-offsets.s
CALL scripts/checksyscalls.sh
Build the test_pmd application
cd $RTE_SDK
./usertools/dpdk-setup.sh
Option: 14
CC config.o
CC iofwd.o
CC macfwd.o
CC macswap.o
CC flowgen.o
CC rxonly.o
CC txonly.o
CC csumonly.o
CC icmpecho.o
CC tm.o
LD testpmd
INSTALL-APP testpmd
INSTALL-MAP testpmd.map
== Build app/proc_info
CC main.o
LD dpdk-procinfo
INSTALL-APP dpdk-procinfo
INSTALL-MAP dpdk-procinfo.map
== Build app/pdump
CC main.o
LD dpdk-pdump
INSTALL-APP dpdk-pdump
INSTALL-MAP dpdk-pdump.map
== Build app/test-crypto-perf
CC main.o
CC cperf_ops.o
CC cperf_options_parsing.o
CC cperf_test_vectors.o
CC cperf_test_throughput.o
CC cperf_test_latency.o
CC cperf_test_pmd_cyclecount.o
CC cperf_test_verify.o
CC cperf_test_vector_parsing.o
CC cperf_test_common.o
LD dpdk-test-crypto-perf
INSTALL-APP dpdk-test-crypto-perf
INSTALL-MAP dpdk-test-crypto-perf.map
== Build app/test-eventdev
CC evt_main.o
CC evt_options.o
CC evt_test.o
CC parser.o
CC test_order_common.o
CC test_order_queue.o
CC test_order_atq.o
CC test_perf_common.o
CC test_perf_queue.o
CC test_perf_atq.o
LD dpdk-test-eventdev
INSTALL-APP dpdk-test-eventdev
INSTALL-MAP dpdk-test-eventdev.map
Build complete [x86_64-native-linuxapp-gcc]
Installation cannot run with T defined and DESTDIR undefined
------------------------------------------------------------------------------
RTE_TARGET exported as x86_64-native-linuxapp-gcc
------------------------------------------------------------------------------
Press enter to continue ...
Check the VF PCI information:
root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
Get back to pod and run the test_pmd
cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
taskset -p --cpu-list 1
pid 1's current affinity list: 2,3,18,19
./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
peer=1,"9e:fd:e6:dd:c1:01"
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:11.0 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:05:11.1 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
Auto-start selected
Ports to start sending a burst of packets first
Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
Configuring Port 0 (socket 0)
Port 0: 9E:FD:E6:DD:C1:01
Configuring Port 1 (socket 0)
Port 1: 9E:FD:E6:DD:C1:02
Checking link statuses...
Port0 Link Up. speed 10000 Mbps- full-duplex
Port1 Link Up. speed 10000 Mbps- full-duplex
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
disabled
Logical Core 3 (socket 0) forwards packets on 2 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
port 1:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 56 RX-missed: 0 RX-bytes: 4096
RX-errors: 0
RX-nombuf: 0
TX-packets: 64 TX-errors: 0 TX-bytes: 4096
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 432 RX-missed: 0 RX-bytes: 27712
RX-errors: 0
RX-nombuf: 0
TX-packets: 461 TX-errors: 0 TX-bytes: 30080
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
RX-errors: 0
RX-nombuf: 0
TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
Throughput (since last show)
Rx-pps: 7068409
Tx-pps: 7091206
############################################################################