Installation Guide

Abstract
Introduction
Hardware Requirements
Software Installation and Deployment
Software Uninstallation
Install Influx DB as datalake (Optional)
Prepare Non-RT RIC DME as data source for AIMLFW (optional)
Feature group creation
Register Model (optional)
Training job creation with DME as data source
Training job creation with standalone Influx DB as data source
Obtain Model URL for deploying trained models
Install only Kserve for deploying models
Uninstall only Kserve
Deploy trained qoe prediction model on Kserve
Test predictions using model deployed on Kserve
Install both Kserve and Kserve adapter for deploying models
Uninstall both Kserve and Kserve adapter for deploying models
Steps to deploy model using Kserve adapter

Abstract 

This document describes how to install AIMLFW, demo scenarios, it’s dependencies and required system resources.

Version history

Date	Ver.	Author	Comment
2022-11-30	0.1.0		First draft
2023-06-06	1.0.0	Joseph Thaliath	H Release
2023-08-10	1.0.1	Joseph Thaliath	H Maintenance release
2023-12-14	1.1.0	Joseph Thaliath	I release

Introduction 

This document describes the supported software and hardware configurations for the reference component as well as providing guidelines on how to install and configure such reference system.

The audience of this document is assumed to have good knowledge in AI/ML tools, Kubernetes and Linux system.

Hardware Requirements 

Below are the minimum requirements for installing the AIMLFW

OS: Ubuntu 22.04 server
16 cpu cores
32 GB RAM
60 GB harddisk

Software Installation and Deployment 

git clone "https://gerrit.o-ran-sc.org/r/aiml-fw/aimlfw-dep"
cd aimlfw-dep

Update recipe file RECIPE_EXAMPLE/example_recipe_latest_stable.yaml which includes update of VM IP and datalake details.

Note: In case the Influx DB datalake is not available, this can be skipped at this stage and can be updated after installing datalake.

bin/install_traininghost.sh

Check running state of all pods and services using below command

kubectl get pods --all-namespaces
kubectl get svc --all-namespaces

Check the AIMLFW dashboard by using the following url

http://localhost:32005/

In case of any change required in the RECIPE_EXAMPLE/example_recipe_latest_stable.yaml file after installation, the following steps can be followed to reinstall with new changes.

bin/uninstall.sh
bin/install.sh -f RECIPE_EXAMPLE/example_recipe_latest_stable.yaml

Software Uninstallation 

bin/uninstall_traininghost.sh

Install Influx DB as datalake (Optional)

Standalone Influx DB installation can be used if DME is not used as a data source.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/influxdb
kubectl exec -it <pod name> bash

From below command we can get username, org name, org id and access token

cat bitnami/influxdb/influxd.bolt | tr -cd "[:print:]"

eg: {“id”:”0a576f4ba82db000”,”token”:”xJVlOom1GRUxDNkldo1v”,”status”:”active”,”description”:”admin’s Token”,”orgID”:”783d5882c44b34f0”,”userID”:”0a576f4b91edb000”,”permissions” …

Use the tokens further in the below configurations and in the recipe file.

Following are the steps to add qoe data to Influx DB.

Execute below from inside Influx DB container to create a bucket:

influx bucket create -n UEData -o primary -t <token>

Install the following dependencies

sudo pip3 install pandas
sudo pip3 install influxdb_client

Use the insert.py in ric-app/qp repository to upload the qoe data in Influx DB

git clone -b f-release https://gerrit.o-ran-sc.org/r/ric-app/qp
cd qp/qp

Update insert.py file with the following content:

import pandas as pd
from influxdb_client import InfluxDBClient
from influxdb_client.client.write_api import SYNCHRONOUS
import datetime


class INSERTDATA:

   def __init__(self):
        self.client = InfluxDBClient(url = "http://localhost:8086", token="<token>")


def explode(df):
     for col in df.columns:
             if isinstance(df.iloc[0][col], list):
                     df = df.explode(col)
             d = df[col].apply(pd.Series)
             df[d.columns] = d
             df = df.drop(col, axis=1)
     return df


def jsonToTable(df):
     df.index = range(len(df))
     cols = [col for col in df.columns if isinstance(df.iloc[0][col], dict) or isinstance(df.iloc[0][col], list)]
     if len(cols) == 0:
             return df
     for col in cols:
             d = explode(pd.DataFrame(df[col], columns=[col]))
             d = d.dropna(axis=1, how='all')
             df = pd.concat([df, d], axis=1)
             df = df.drop(col, axis=1).dropna()
     return jsonToTable(df)


def time(df):
     df.index = pd.date_range(start=datetime.datetime.now(), freq='10ms', periods=len(df))
     df['measTimeStampRf'] = df['measTimeStampRf'].apply(lambda x: str(x))
     return df


def populatedb():
     df = pd.read_json('cell.json.gz', lines=True)
     df = df[['cellMeasReport']].dropna()
     df = jsonToTable(df)
     df = time(df)
     db = INSERTDATA()
     write_api = db.client.write_api(write_options=SYNCHRONOUS)
     write_api.write(bucket="UEData",record=df, data_frame_measurement_name="liveCell",org="primary")

populatedb()

Update <token> in insert.py file

Follow below command to port forward to access Influx DB

kubectl port-forward svc/my-release-influxdb 8086:8086

To insert data:

python3 insert.py

To check inserted data in Influx DB , execute below command inside the Influx DB container:

influx query  'from(bucket: "UEData") |> range(start: -1000d)' -o primary -t <token>

Prepare Non-RT RIC DME as data source for AIMLFW (optional)

Bring up the RANPM setup by following the steps mentioned in the file install/README.md present in the repository RANPM repository

Once all the pods are in running state, follow the below steps to prepare ranpm setup for AIMLFW qoe usecase data access

The scripts files are present in the folder demos/hrelease/scripts of repository AIMLFW repository

Note: The following steps need to be performed in the VM where the ranpm setup is installed.

git clone "https://gerrit.o-ran-sc.org/r/aiml-fw/aimlfw-dep"
cd aimlfw-dep/demos/hrelease/scripts
./get_access_tokens.sh

Output of ./get_access_tokens.sh can be used during feature group creation step.

Execute the below script

./prepare_env_aimlfw_access.sh

Add feature group from AIMLFW dashboard, example on how to create a feature group is shown in this demo video: Feature group creation demo

Execute below script to push qoe data into ranpm setup

./push_qoe_data.sh  <source name mentioned when creating feature group> <Number of rows> <Cell Identity>

Example for executing above script

./push_qoe_data.sh  gnb300505 30 c4/B2

Steps to check if data is upload correctly

kubectl exec -it influxdb2-0 -n nonrtric -- bash
influx query 'from(bucket: "pm-logg-bucket") |> range(start: -1000000000000000000d)' |grep pdcpBytesDl

Steps to clear the data in InfluxDB

kubectl exec -it influxdb2-0 -n nonrtric -- bash
influx delete --bucket pm-logg-bucket --start 1801-01-27T05:00:22.305309038Z   --stop 2023-11-14T00:00:00Z

Feature group creation 

From AIMLFW dashboard create feature group (Training Jobs-> Create Feature Group )

NOTE: Below are some example values to be used for the DME based feature group creation for qoe usecase

Parameter	Value
Feature Group Name	featuregroup1
Features	pdcpBytesDl,pdcpBytesUl
Host	<IP of VM where DME is installed>
Port	31812
Db Org	est
Bucket Name	pm-logg-bucket
DB Token	<token obtained using get_access_tokens.sh during DME setup>
DME flag	enable it
Source Name	<any source name. but same needs to be given when running push_qoe_data.sh>
Measured Obj Class	NRCellDU
Dme port	31823

NOTE: Below are some example values to be used for the standalone influx DB creation for qoe usecase

Parameter	Value
Feature Group Name	featuregroup1
Features	pdcpBytesDl,pdcpBytesUl
Host	<IP of VM where Influx DB is installed>
Port	<port of Influx DB>
Db Org	primary
Bucket Name	UEData
DB Token	<token obtained during INflux DB installation>

Register Model (optional)

Register the model using the below steps if using Model management service for training.

curl --location 'http://<VM IP where AIMLFW is installed>:32006/registerModel' \
      --header 'Content-Type: application/json' \
      --data '{
         "model-name":"qoe1",
         "rapp-id": "rapp_1",
         "meta-info" :
         {
             "accuracy":"90",
             "model-type":"timeseries",
             "feature-list":["pdcpBytesDl","pdcpBytesUl"]
         }
      }'

Training job creation with DME as data source 

AIMLFW should be installed by following steps in section Software Installation and Deployment
RANPM setup should be installed and configured as per steps mentioned in section Prepare Non-RT RIC DME as data source for AIMLFW
To create training job, follow the steps in the demo videos stored here: Training Job creation
After training job is created and executed successfully, model can be deployed using steps mentioned in section Deploy trained qoe prediction model on Kserve or Steps to deploy model using Kserve adapter

NOTE: Below are some example values to be used for the QoE usecase training job creation when model management service is not used.

Parameter	Value
Training Job Name	qoetest
Model Management Service	disable
Training Function	qoe_pipeline_h_release
FeatureGroup Name	featuregroup1
Datalake Source	Influx DB
_measurement	test,ManagedElement=nodedntest,GNBDUFunction=1004,NRCellDU=c4_B2
bucket	pm-logg-bucket
Feature Filter
Hyper Parameters	epochs:1
Description	test

NOTE: Below are some example values to be used for the QoE usecase training job creation when model management service is used.

Parameter	Value
Training Job Name	qoetest
Model Management Service	enable
Model name	qoe1
FeatureGroup Name	featuregroup1
Datalake Source	Influx DB
_measurement	test,ManagedElement=nodedntest,GNBDUFunction=1004,NRCellDU=c4_B2
bucket	pm-logg-bucket
Feature Filter
Hyper Parameters	epochs:1
Description	test

Training job creation with standalone Influx DB as data source 

AIMLFW should be installed by following steps in section Software Installation and Deployment
Standalone Influx DB should be setup and configured as mentioned in section Install Influx DB as datalake
To create training job, follow the steps in the demo videos stored here: Training Job creation
After training job is created and executed successfully, model can be deployed using steps mentioned in section Deploy trained qoe prediction model on Kserve or Steps to deploy model using Kserve adapter

NOTE: Below are some example values to be used for the QoE usecase training job creation when model management service is not used

Parameter	Value
Training Job Name	qoetest
Model Management Service	disable
Training Function	qoe_pipeline_g_release
FeatureGroup Name	featuregroup1
Datalake Source	Influx DB
_measurement	liveCell
bucket	UEData
Feature Filter
Hyper Parameters	epochs:1
Description	test

NOTE: Below are some example values to be used for the QoE usecase training job creation when model management service is used

Parameter	Value
Training Job Name	qoetest
Model Management Service	enable
Model Name	qoe_pipeline_g_release
Datalake Source	Influx DB
_measurement	liveCell
bucket	UEData
Feature Filter
Hyper Parameters	epochs:1
Description	test

Obtain Model URL for deploying trained models 

URL for deployment can be obainted from AIMFW dashboard (Training Jobs-> Training Job status -> Select Info for a training job -> Model URL) In case of using AIMLFW Model management service, URL for downloading and deploying model using Model Management Service will be the following:

http://<VM IP where AIMLFW is deployed>:32006/downloadModel/<model name>/model.zip

Install only Kserve for deploying models 

To install Kserve run the below commands

./bin/install_kserve.sh

Uninstall only Kserve 

To uninstall Kserve run the below commands

./bin/uninstall_kserve.sh

Deploy trained qoe prediction model on Kserve 

Create namespace using command below

kubectl create namespace kserve-test

Create qoe.yaml file with below contents

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: qoe-model
spec:
  predictor:
    tensorflow:
      storageUri: "<update Model URL here>"
      runtimeVersion: "2.5.1"
      resources:
        requests:
          cpu: 0.1
          memory: 0.5Gi
        limits:
          cpu: 0.1
          memory: 0.5Gi

To deploy model update the Model URL in the qoe.yaml file and execute below command to deploy model Refer Obtain Model URL for deploying trained models

kubectl apply -f qoe.yaml -n kserve-test

Check running state of pod using below command

kubectl get pods -n kserve-test

Test predictions using model deployed on Kserve 

Use below command to obtain Ingress port for Kserve.

kubectl get svc istio-ingressgateway -n istio-system

Obtain nodeport corresponding to port 80. In the below example, the port is 31206.

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.105.222.242   <pending>     15021:31423/TCP,80:31206/TCP,443:32145/TCP,31400:32338/TCP,15443:31846/TCP   4h15m

Create predict.sh file with following contents

model_name=qoe-model
curl -v -H "Host: $model_name.kserve-test.example.com" http://<IP of where Kserve is deployed>:<ingress port for Kserve>/v1/models/$model_name:predict -d @./input_qoe.json

Update the IP of host where Kserve is deployed and ingress port of Kserve obtained using above method.

Create sample data for predictions in file input_qoe.json. Add the following content in input_qoe.json file.

{"signature_name": "serving_default", "instances": [[[2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56],
       [2.56, 2.56]]]}

Use command below to trigger predictions

source predict.sh

Install both Kserve and Kserve adapter for deploying models 

To install Kserve run the below commands Please note to update the DMS IP in example_recipe_latest_stable.yaml before installation

./bin/install_kserve_inference.sh

Uninstall both Kserve and Kserve adapter for deploying models 

To uninstall Kserve run the below commands

./bin/uninstall_kserve_inference.sh

Steps to deploy model using Kserve adapter 

Prerequisites

Install chart museum
Build ricdms binary

Run ric dms

export RIC_DMS_CONFIG_FILE=$(pwd)/config/config-test.yaml
./ricdms

Create sample_config.json

Create sample_config.json file with the following contents

 {
   "xapp_name": "sample-xapp",
   "xapp_type": "inferenceservice",
   "version": "2.2.0",
   "sa_name": "default",
   "inferenceservice": {
       "engine": "tensorflow",
       "storage_uri": "<Model URL>",
       "runtime_version": "2.5.1",
       "api_version": "serving.kubeflow.org/v1beta1",
       "min_replicas": 1,
       "max_replicas": 1
   }
 }

Refer :ref:`Obtain Model URL for deploying trained models <reference5>`

Copy sample_config.json

Update the below command with kserve adapter pod name

kubectl cp sample_config.json ricips/<kserve adapter pod name>:pkg/helm/data/sample_config.json

Generating and upload helm package

curl --request POST --url 'http://127.0.0.1:31000/v1/ips/preparation?configfile=pkg/helm/data/sample_config.json&schemafile=pkg/helm/data/sample_schema.json'

Check uploaded charts
```
curl http://127.0.0.1:8080/api/charts
```

Deploying the model

curl --request POST --url 'http://127.0.0.1:31000/v1/ips?name=inference-service&version=1.0.0'

Check deployed Inference service
```
kubectl get InferenceService -n ricips
```