Setup Configuration

A.1 Setup Configuration

The configuration shown in Figure 26 shows how to set up a test environment to execute xRAN scenarios where O-DU and 0-RU are simulated using the sample application. This setup allows development and prototyping as well as testing of xRAN specific functionality. The O-DU side can be instantiated with a full 5G NR L1 reference as well. The configuration differences of the 5G NR l1app configuration are provided below. Steps for running the sample application on the O-DU side and 0-RU side are the same, except configuration file options may be different.

Figure 26. Setup for xRAN Testing

Figure 26. Setup for xRAN Testing

Figure 27. Setup for xRAN Testing with PHY and Configuration C3

Figure 27. Setup for xRAN Testing with PHY and Configuration C3

A.2 Prerequisites

Each server in Figure 26 requires the following:

  • Wolfpass server according to recommended BOM for FlexRAN such as Intel® Xeon® Skylake Gold 6148 FC-LGA3647 2.4 GHz 27.5 MB 150W 20 cores (two sockets)
  • BIOS settings:
  • Intel(R) Virtualization Technology Enabled
  • Intel(R) VT for Directed I/O - Enabled
  • ACS Control - Enabled
  • Coherency Support - Disabled
  • Front Haul networking cards:
  • Intel® Ethernet Converged Network Adapter XL710-QDA2
  • Intel® Ethernet Converged Network Adapter XXV710-DA2
  • Intel® FPGA Programmable Acceleration Card (Intel® FPGA PAC) N3000

The Front Haul NIC requires support for PTP HW timestamping.

The recommended configuration for NICs is:

ethtool -i enp33s0f0
driver: i40e
version: 2.10.19.82
firmware-version: 7.20 0x80007949 1.2585.0
expansion-rom-version:
bus-info: 0000:21:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
ethtool -T enp33s0f0
Time stamping parameters for enp33s0f0:
Capabilities:
    hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
    software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
    hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
    hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 4
Hardware Transmit Timestamp Modes:
    off                   (HWTSTAMP_TX_OFF)
    on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
    none                  (HWTSTAMP_FILTER_NONE)
    ptpv1-l4-sync         (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
    ptpv1-l4-delay-req    (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
    ptpv2-l4-event        (HWTSTAMP_FILTER_PTP_V2_L4_EVENT)
    ptpv2-l4-sync         (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
    ptpv2-l4-delay-req    (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
    ptpv2-l2-event        (HWTSTAMP_FILTER_PTP_V2_L2_EVENT)
    ptpv2-l2-sync         (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
    ptpv2-l2-delay-req    (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
    ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)
    ptpv2-sync            (HWTSTAMP_FILTER_PTP_V2_SYNC)
    ptpv2-delay-req       (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)

A PTP Main Reference Clock is required to be available in the network to provide synchronization of both O-DU and RU to GPS time.

1.Installing Intel® C++ Compiler v19.0.3 is preferred. or you could get Intel® C++ Compiler through below link with community license, however the version you could get is always latest version, the verification for that version might not be performed yet, please feedback through O-DU Low project WIKI page if you meet an issue.
https://software.intel.com/en-us/system-studio/choose-download

2.Download DPDK 19.11.

3.Change DPDK files according to below diff information which relevant to O-RAN FH:

diff --git a/drivers/net/i40e/i40e_ethdev.c
b/drivers/net/i40e/i40e_ethdev.c

index 85a6a86..236fbe0 100644

--- a/drivers/net/i40e/i40e_ethdev.c

+++ b/drivers/net/i40e/i40e_ethdev.c

@@ -2207,7 +2207,7 @@ void i40e_flex_payload_reg_set_default(struct
i40e_hw \*hw)

/\* Map queues with MSIX interrupt \*/

main_vsi->nb_used_qps = dev->data->nb_rx_queues -

pf->nb_cfg_vmdq_vsi \* RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM;

- i40e_vsi_queues_bind_intr(main_vsi, I40E_ITR_INDEX_DEFAULT);

+ i40e_vsi_queues_bind_intr(main_vsi, I40E_ITR_INDEX_NONE);

i40e_vsi_enable_queues_intr(main_vsi);

/\* Map VMDQ VSI queues with MSIX interrupt \*/

@@ -2218,6 +2218,10 @@ void i40e_flex_payload_reg_set_default(struct
i40e_hw \*hw)

i40e_vsi_enable_queues_intr(pf->vmdq[i].vsi);

}

+ i40e_aq_debug_write_global_register(hw,

+ 0x0012A504,

+ 0, NULL);

+

/\* enable FDIR MSIX interrupt \*/

if (pf->fdir.fdir_vsi) {

i40e_vsi_queues_bind_intr(pf->fdir.fdir_vsi,

diff --git a/drivers/net/i40e/i40e_ethdev_vf.c
b/drivers/net/i40e/i40e_ethdev_vf.c

index 001c301..6f9ffdb 100644

--- a/drivers/net/i40e/i40e_ethdev_vf.c

+++ b/drivers/net/i40e/i40e_ethdev_vf.c

@@ -640,7 +640,7 @@ struct rte_i40evf_xstats_name_off {

map_info = (struct virtchnl_irq_map_info \*)cmd_buffer;

map_info->num_vectors = 1;

- map_info->vecmap[0].rxitr_idx = I40E_ITR_INDEX_DEFAULT;

+ map_info->vecmap[0].rxitr_idx = I40E_ITR_INDEX_NONE;

map_info->vecmap[0].vsi_id = vf->vsi_res->vsi_id;

/\* Alway use default dynamic MSIX interrupt \*/

map_info->vecmap[0].vector_id = vector_id;

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
b/drivers/net/ixgbe/ixgbe_ethdev.c

index 26b1927..018eb8f 100644

--- a/drivers/net/ixgbe/ixgbe_ethdev.c

+++ b/drivers/net/ixgbe/ixgbe_ethdev.c

@@ -3705,7 +3705,7 @@ static int
ixgbevf_dev_xstats_get_names(__rte_unused struct rte_eth_dev \*dev,

\* except for 82598EB, which remains constant.

\*/

if (dev_conf->txmode.mq_mode == ETH_MQ_TX_NONE &&

- hw->mac.type != ixgbe_mac_82598EB)

+ hw->mac.type != ixgbe_mac_82598EB && hw->mac.type !=
ixgbe_mac_82599EB)

dev_info->max_tx_queues = IXGBE_NONE_MODE_TX_NB_QUEUES;

}

dev_info->min_rx_bufsize = 1024; /\* cf BSIZEPACKET in SRRCTL register
\*/

diff --git a/lib/librte_eal/common/include/rte_dev.h
b/lib/librte_eal/common/include/rte_dev.h

old mode 100644

new mode 100755

5.Build and install DPDK:

[root@xran dpdk]# ./usertools/dpdk-setup.sh

select [39] x86_64-native-linuxapp-icc

select [46] Insert VFIO module

exit [62] Exit Script

6.Make below file changes in dpdk that assure i40e to get best latency of packet processing:

--- i40e.h 2018-11-30 11:27:00.000000000 +0000

+++ i40e_patched.h 2019-03-06 15:49:06.877522427 +0000

@@ -451,7 +451,7 @@

#define I40E_QINT_RQCTL_VAL(qp, vector, nextq_type) \\

(I40E_QINT_RQCTL_CAUSE_ENA_MASK \| \\

- (I40E_RX_ITR << I40E_QINT_RQCTL_ITR_INDX_SHIFT) \| \\

+ (I40E_ITR_NONE << I40E_QINT_RQCTL_ITR_INDX_SHIFT) \| \\

((vector) << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) \| \\

((qp) << I40E_QINT_RQCTL_NEXTQ_INDX_SHIFT) \| \\

(I40E_QUEUE_TYPE_##nextq_type << I40E_QINT_RQCTL_NEXTQ_TYPE_SHIFT))

--- i40e_main.c 2018-11-30 11:27:00.000000000 +0000

+++ i40e_main_patched.c 2019-03-06 15:46:13.521518062 +0000

@@ -15296,6 +15296,9 @@

pf->hw_features \|= I40E_HW_HAVE_CRT_RETIMER;

/\* print a string summarizing features \*/

i40e_print_features(pf);

+

+ /\* write to this register to clear rx descriptor \*/

+ i40e_aq_debug_write_register(hw, 0x0012A504, 0, NULL);

return 0;

A.3 Configuration of System

1.Boot Linux with the following arguments:

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-1062.12.1.rt56.1042.el7.x86_64 root=/dev/mapper/centos-root ro
crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap intel_iommu=on iommu=pt
usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 audit=0
intel_pstate=disable cgroup_memory=1 cgroup_enable=memory mce=off idle=poll
hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=0 default_hugepagesz=1G
isolcpus=1-19,21-39 rcu_nocbs=1-19,21-39 kthread_cpus=0,20 irqaffinity=0,20
nohz_full=1-19,21-39

2.Download from Intel Website and install updated version of i40e driver if needed. The current recommended version of i40e is 2.10.19.82. However, any latest version of i40e after x2.9.21 expected to be functional for ORAN FH.

3.Identify PCIe Bus address of the Front Haul NIC:

lspci |grep Eth
19:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking (rev 02)
19:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking (rev 02)
1d:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking (rev 02)
1d:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking (rev 02)
21:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
21:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
67:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)

4.Identify the Ethernet device name:

ethtool -i enp33s0f0
driver: i40e
version: 2.10.19.82
firmware-version: 7.20 0x80007949 1.2585.0
expansion-rom-version:
bus-info: 0000:21:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Enable

5.Enable two virtual functions (VF) on the device:

echo 2 > /sys/class/net/enp33s0f0/device/sriov_numvfs

More information about VFs supported by Intel NICs can be found at https://doc.dpdk.org/guides/nics/intel_vf.html.

The resulting configuration can look like the listing below, where two new VFs were added:

lspci|grep Eth

21:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
21:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
21:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
21:02.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)

6.Configure MAC address and VLAN settings for VFs for XRAN, based on requirements for xRAN scenario and assignment of VLAN ID using IP tool perform configuration of VF.

Example where O-DU and O-RU simulation run on the same sytem:

#!/bin/bash

echo 2 > /sys/bus/pci/devices/0000:21:00.0/sriov_numvfs ip link set enp33s0f0 vf 1 mac 00:11:22:33:44:66 vlan 1 ip link set enp33s0f0 vf 0 mac 00:11:22:33:44:66 vlan 2 echo 2 > /sys/bus/pci/devices/0000:21:00.1/sriov_numvfs ip link set enp33s0f1 vf 1 mac 00:11:22:33:44:55 vlan 1 ip link set enp33s0f1 vf 0 mac 00:11:22:33:44:55 vlan 2

where output is next:

[root@xran app]# ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: enp25s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 64:4c:36:10:1f:30 brd ff:ff:ff:ff:ff:ff

3: enp25s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 64:4c:36:10:1f:31 brd ff:ff:ff:ff:ff:ff

4: enp29s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 64:4c:36:10:1f:34 brd ff:ff:ff:ff:ff:ff

5: enp29s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 64:4c:36:10:1f:35 brd ff:ff:ff:ff:ff:ff

6: enp33s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 3c:fd:fe:b9:f8:b4 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:11:22:33:44:66, vlan 2, spoof checking on, link-state auto, trust off

vf 1 MAC 00:11:22:33:44:66, vlan 1, spoof checking on, link-state auto, trust off

7: enp33s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 3c:fd:fe:b9:f8:b5 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:11:22:33:44:55, vlan 2, spoof checking on, link-state auto, trust off

vf 1 MAC 00:11:22:33:44:55, vlan 1, spoof checking on, link-state auto, trust off

8: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether a4:bf:01:3e:1f:be brd ff:ff:ff:ff:ff:ff

9: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether a4:bf:01:3e:1f:bf brd ff:ff:ff:ff:ff:ff

10: npacf0g0l0: <LOWER_UP> mtu 9600 qdisc noop state UNKNOWN mode DEFAULT group default qlen 1000

link/generic

11: npacf0g0l1: <LOWER_UP> mtu 9600 qdisc noop state UNKNOWN mode DEFAULT group default qlen 1000

link/generic

12: npacf0g0l2: <LOWER_UP> mtu 9600 qdisc noop state UNKNOWN mode DEFAULT group default qlen 1000

link/generic

13: npacf0g0l3: <LOWER_UP> mtu 9600 qdisc noop state UNKNOWN mode DEFAULT group default qlen 1000

link/generic

After this step FH NIC is configured.

O-DU

VF for C-plane is VF1 on PFH enp33s0f0enp216s0f0, it has ETH mac address 00:11:22:33:44:66 and VLAN tag 1. PCIe Bus address is VF1 is 21d8:02.1

VF for U-plane is VF0 on PFH enp33s0f0enp216s0f0, it has ETH mac address 00:11:22:33:44:66 and VLAN tag 2. PCIe Bus address is VF1 is 21d8:02.0

O-RU

VF for C-plane is VF1 on PF enp33s0f1, it has ETH mac address 00:11:22:33:44:55 and VLAN tag 1. PCIe Bus address is VF1 is 21:0a.1

VF for U-plane is VF0 on PF enp33s0f1, it has ETH mac address 00:11:22:33:44:55 and VLAN tag 2. PCIe Bus address is VF1 is 21:0a.0

A.4 Install and Configure Sample Application

To install and configure the sample application:

  1. Set up the environment:

    export GTEST_ROOT=`pwd`/gtest-1.7.0

    export RTE_SDK=`pwd`/dpdk-19.11

    export RTE_TARGET=x86_64-native-linuxapp-icc

    export MLOG_DIR=`pwd`/flexran_l1_sw/libs/mlog

    export XRAN_DIR=`pwd`/flexran_xran

  2. Compile xRAN library and test the application:

    [turner@xran home]$ cd $XRAN_DIR

    [turner@xran xran]$ ./build.sh

  3. Configure the sample app.

IQ samples can be generated using Octave* and script libs/xran/app/gen_test.m. (CentOS* has octave-3.8.2-20.el7.x86_64 compatible with get_test.m)

Other IQ sample test vectors can be used as well. The format of IQ samples is binary int16_t I and Q for N slots of the OTA RF signal. For example, for mmWave, it corresponds to 792RE*2*14symbol*8slots*10 ms = 3548160 bytes per antenna. Refer to comments in gen_test.m to correctly specify the configuration for IQ test vector generation.

Update config_file_o_du.dat (or config_file_o_ru.dat) with a suitable configuration for your scenario.

Update run_o_du.sh (run_o_ru.sh) with PCIe bus address of VF0 and VF1 used for U-plane and C-plane correspondingly:

./build/sample-app -c ./usecase/mu0_10mhz/config_file_o_du.dat -p 2 0000:21d8:02.0 0000:21d8:02.1
  1. Run application using run_o_du.sh (run_o_ru.sh).