Integrating VoltMesh with OCP ** (Experimental Feature) **

Objective

This guide provides instructions on how to integrate Volterra site with OpenShift Cloud Platform (OCP). For more information on Volterra sites, see Volterra Site.

Using the instructions provided in this guide, you can integrate the Volterra site with OCP, discover services, and advertise them.

Note: The following apply:

  • Volterra site in this document refers to the VoltMesh capabilities only.
  • In case of Site on K8s, this is ** Experimental Feature (EF) ** feature only. The feature provides basic functionality of launching Volterra Site on K8s with limited functionality, tools, and observability. The Site Mesh group for Site on K8s is not supported.

Deployment Methods

The instructions provided in this guide cover how to deploy VoltMesh as pods and also how to discover services for webapps deployed in the OCP cluster using the features such as service discovery and advertising.

You can deploy VoltMesh in the following methods:

  • Deploy site as Ingress/Egress gateway for OCP.
  • Deploy site as pods in OCP.

Note: The site as pods supports only VoltMesh functionalities and does not support VoltStack. The supported Kubernetes version is 1.21.

The following image is a high-level view for a sample deployment showing both deployment methods:

voltmesh ocp patterns
Figure: Deployment Methods

When deploying VoltMesh as an Ingress/Egress gateway for OCP cluster, you will need to deploy webapps in OCP, set up service discovery, and advertise the services using HTTP Load Balancer.

When deploying VoltMesh as OCP pods in the OCP cluster, you will need to deploy webapps in OCP and advertise services using HTTP Load Balancer. VoltMesh site deployed as pods will automatically discover services running in the OCP cluster by querying the kube-api for services.


Prerequisites

The following prerequisites apply:

  • A Volterra Account. If you do not have an account, see Create a Volterra Account.
  • A Volterra Site. If you do not have a site, see Site Management.
  • OCP K8s cluster. For more information, see OCP Documentation.

    Note: OCP version 4.7 is supported.

  • Minimum 4 vCPUs and 8 GB of memory per node.
  • Kubernetes StorageClass with enabled Dynamic Persistent Volume Provisioner (PVC) with a minimum 1 GB space.

Note: Use the kubectl get storageclass command to check if dynamic PVC is enabled for your K8s StorageClass. The output with an entry with (default) in the name indicates that K8s storage class is enabled with dynamic PVC.


Deploy as Ingress/Egress Gateway

Deploy VoltMesh node/site as a bare metal or VM or as a public cloud provider instance. For instructions on how install VoltMesh, see Site Management guides.

After successful deployment, perform the following steps:

Step 1: Obtain kubeconfig file of your OCP cluster.

When VoltMesh deployed outside of the OCP cluster, kubeconfig file is required to be loaded onto VoltConsole for service discovery to function. The kubeconfig enables VoltMesh to monitor kube-api, discover OCP services, and discover lifecycle of pods. It is recommenced to create a custom non-expiring kubeconfig with permissions are limited to read-only. Kubeconfig uploaded to VoltConsole is secured with Volterra Blindfold.

Note: For simplicity, in this deployment guide, the system:admin kubeconfig file is used instead of a token-based kubeconfig.

The kubeconfig can be obtained in one of the following ways:

Root Access: Obtain kubeconfig on the system as a root user.

If you have root access to any master node, obtain the kubeconfig files located at /etc/kubernetes/static- pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/.

Note: You may need to use this method when OCP oauth pods not working and you are unable to use token-based authentication to login.

Remote Access: Obtain kubeconfig using remote interaction with the kube-api.

Perform remote login and interact with kube-api. Enter the following commands to extract kubecofig file:

  • Create a directory to store the kubeconfigs and change to that directory.
mkdir kubefile 

cd kubefile
  • Extract kubeconfigs remotely using the extract command.
oc -n openshift-kube-apiserver extract secret/node-kubeconfigs lb-ext.kubeconfig
  • The extract command returns the extracted files. The following is a sample output:
lb-ext.kubeconfig
lb-int.kubeconfig
localhost-recovery.kubeconfig
localhost.kubeconfig
  • Verify the file list with the ls -al command.
Step 2: Use the kubeconfig and log into OCP.

Depend on your environment setup, use one of the kubeconfig files to login. This example uses the internal kubeconfig.

Note: You cannot use the <localhost>.kubeconfig to login. Use the lb-ext.kubeconfig or lb- int.kubeconfig.

  • Set the KUBECONFIG environment variable.
export KUBECONFIG=./lb-int.kubeconfig
  • Connect to the OCP and verify the node information:
oc get node

The following is a sample output of the above commands:

oc get node
Figure: OCP Node List

Step 3: Deploy web apps to OCP.

Deploy your apps to OCP using the app manifest file. This example deploys an app called Hipster Shop whose manifest is hosted here.

Note: Ensure that you update the service type to NodePort for the service you want VoltMesh to advertise. This is because the VoltMesh node is deployed outside of the OCP cluster.

Do the following:

  • Create a namespace in OCP that is same as your application namespace in VoltConsole. This example creates a namespace called foobang-chan.
oc create ns foobang-chan 
  • Deploy the apps.
oc -n foobang-chan create -f hipster.yaml

The following output is returned:

deployment.apps/emailservice created
service/emailservice created
deployment.apps/paymentservice created
service/paymentservice created 
deployment.apps/productcatalogservice created 
service/productcatalogservice created 
deployment.apps/cartservice created 
service/cartservice created 
deployment.apps/currencyservice created 
service/currencyservice created 
deployment.apps/shippingservice created 
service/shippingservice created 
deployment.apps/recommendationservice created 
service/recommendationservice created 
deployment.apps/checkoutservice created 
service/checkoutservice created 
deployment.apps/frontend created 
service/frontend created 
deployment.apps/redis-cart created 
service/redis-cart created
  • Verify that the services are started:
oc -n foobang-chan get svc

The following output is returned:

oc services list
Figure: Deployed Services in OCP

Note: For OCP to be able to send traffic to pod, application service type should be NodePort. In this example, frontend service is planned to be advertised with VoltMesh so its type is NodePort.

Step 4: Verify that VoltMesh can reach the OCP kube-api.

Ensure VoltMesh able to reach the OCP kube-api as using the server URL present in the kubeconfig. The server URL can be obtained by viewing the contents of the kubeconfig file.

  • Use a file reading command such as cat <kubeconfig-file> to view the kubeconfig contents.

oc api url
Figure: API URL in Kubeconfig

  • Log into VoltConsole and go to Sites -> Site List. Click on your VoltMesh site to open its dashboard. Click the Tools tab.\
  • Select Ping from the options of the Select Tool drop-down. Enter the API URL in the Destination field and click Call ping.

oc api ping
Figure: Ping from Site to OCP Kube API

Note: You can also execute ping using the site local UI. See Site Local UI for more information.

Step 5: Create service discovery.
  • In VoltConsole, go to Manage -> App Management -> Service Discoveries in the system namespace. Click Add discovery. Set metadata in the Metadata section and go to Where section.
  • Select Site for the Virtual-Site or Site or Network field and select your site from the the Reference field's drop-down list. Ensure that you select Site Local Network for the Network Type field.
  • Select K8s Discovery Configuration (it is populated by default) for the Discovery Method field and click Configure under the K8s Discovery Configuration.

oc discovery 1
Figure: Service Discovery Where & Discovery Method

  • In the Access Credentials section, select Kubeconfig for the Select Kubernetes Credentials field and click Configure under the Kubeconfig field.
  • In the Secret configuration, ensure that the Text is selected for the Type and enter the contents of the kubeconfig file in the form available beneath the Type field. Click Blindfold. Wait till the secret is encrypted and click Apply.

discovery blindfold
Figure: Encrypt Secret for Service Discovery

Note: Use a file reading utility such as cat to obtain the contents of kubeconfig file. Use the kubeconfig file obtained in previous steps.

  • Click Apply and then click Save and Exit to complete creating the discovery object.

Note: For full set of instructions on Kubernetes-based service discovery, see Service Discovery - K8s.

Step 6: Verify that the services are discovered.
  • Go to Manage -> App Management -> Service Discoveries page and confirm that the service disovery object is created and lists discovered services under the Services column.

discovery created
Figure: Service Discovery Created

  • Click the value displayed in the Services column to view the details of the services discovered.

discovered services
Figure: Discovered Services

  • Search for the frontend service and note down the service name. Ensure that the Service Type value is displayed as NodePort.

frontend nodeport
Figure: Service to be Advertised via VoltMesh


Deploy as OCP Pods

Do the following do deploy VoltMesh node as OCP pods:

Step 1: Ensure that a working OCP enviroment is available.
  • Check the nodes and node states in OCP cluster:
oc get node
oc get node -o wide
  • Confirm that no node has state Not Running or Not Completed.
oc get pod -A | egrep -vi 'Running|Completed'
Step 2: Optionally, approve a pending certificate signing request.

Enter the following command to approve a pending certificate signing request.

oc adm certificate approve `oc get csr |grep Pending |awk '{print $1}' `
Step 3: Configure hugepages.
  • Label node or assigned role for which hugepages is to be configured. Alternatively, you can also configure or label hugepages for all nodes. This example shows labeling for 3 worker nodes.
oc label node worker-1.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=
oc label node worker-2.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=
oc label node worker-3.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=
  • Verify that the labels are created using the oc get node command.

Note: The worker-1, worker-2, and worker-3 are labelled as worker-hp which is an arbitrary tag/label/name of choice.

  • Create two files named hugepages-tuned-boottime.yaml and hugepages-mcp.yaml. The following are sample YAML contents for both:

hugepages-tuned-boottime.yaml

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: hugepages 
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile: 
  - data: |
      [main]
      summary=Boot time configuration for hugepages
      include=openshift-node
      [bootloader]
      cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=1792 
    name: openshift-node-hugepages

  recommend:
  - machineConfigLabels: 
      machineconfiguration.openshift.io/role: "worker-hp"
    priority: 30
    profile: openshift-node-hugepages

hugepages-mcp.yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-hp
  labels:
    worker-hp: ""
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-hp: ""
  • Enable hugepages using the created YAML files.
oc create -f hugepages-tuned-boottime.yaml

The following output is returned:

tuned.tuned.openshift.io/hugepages created
oc create -f hugepages-mcp.yaml

The following output is returned:

machineconfigpool.machineconfiguration.openshift.io/worker-hp created

Note: See Official OCP Documentation for more information on hugepages configuration.

  • Wait for all worker node in Ready status. Use the oc get node command to verify the state of nodes.
Step 4: Verify that hugepages and storage class or dynamic PV is enabled.
  • Verify that hugepages is enabled. This example shows verification command for the worker-1 node. Repeat this command for all the worker nodes.
oc get node worker-1.ocp.f5-udf.com -o jsonpath="{.status.allocatable.hugepages-2Mi}"

Note: Output for the above command must be a non-zero value and should match the hugepages you specified in the hugepages-tuned-boottime.yaml file.

  • Verify that the storage class or dynamic Persistent Volume (PV) is configured. This example shows verification for the storage class:
oc get sc

The following output confirms that storage class is configured:

ocp sc
Figure: Storage Class Validation

Step 5: Log into VoltConsole and create site tokens.
  • Click on the System namespace.
  • Select Manage -> Site Management -> Site Tokens.
  • Click Add site token. In the Name field, enter the token name.
  • Click Add site token. Token gets created.
  • Click > to expand the token details in JSON format and note down the value of the uid field.
Step 6: Create a site deployment manifest file.
  • Download the sample Manifest Template.
  • Edit the configuration in the Vpm section of the manifest file per the following guidelines:

    • In the ClusterName field, type your cluster name.
    • In the Latitude and Longitude fields, type the latitude and longitude values.
    • In the Token field, type the site token.
  • Save your changes.
Step 7: Deploy the Volterra Site
  • Deploy Volterra site using the site manifest file. This example uses the ves-ocp.yaml sample site manifest file.
oc create -f ves-ocp.yaml

The following is sample output of the above command:

namespace/ves-system created
daemonset.apps/volterra-ce-init created
serviceaccount/vpm-sa created
role.rbac.authorization.k8s.io/vpm-role created
rolebinding.rbac.authorization.k8s.io/vpm-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/ver created
configmap/vpm-cfg created
statefulset.apps/vp-manager created
Step 8: Perform registration.
  • Run the following command to watch the progress of the site creation.
watch 'oc -n ves-system get pod -o wide'
  • Check the PV and PVC information.
oc -n ves-system get pv
oc -n ves-system get pvc

Note: Log into the VoltConsole to accept the site registrations. Only vp-manager-0 pod will be displayed as running until registration of site is approved.

  • Log into VoltConsole and then navigate to the System namespace.
  • Click Manage -> Site Management -> Registrations -> Pending Registrations.
  • Find your node name and then click the blue checkmark.
  • In the form that appears, fill in all required fields with the asterisk symbol (*).
  • Click Save and Exit.
  • The watch command you set earlier will start showing the ver-0 pod as well. This indicates that the site is registered and getting provisioned. Upon successful provision, the status of all pods changes to Running.
Step 9: Check site status and health.

It may take a few minutes for the site health and connectivity score information to update.

  • In the System namespace, click Sites -> Site List.
  • Click on your site name. The Dashboard tab appears, along with many other tabs to inspect your site.
  • Click the Site Status tab to verify the following:

    • The Update Status field has a Successful value for the Volterra OS Status section.
    • The Update Status field has a Successful value for the Volterra Software Status section.
    • The Tunnel status and Control Plane fields under the RE Connectivity section have up values.

Note: At this point, site is ready to provide service to apps deployed on OCP. Due to the site pods being inside OCP, you do not need to create service discovery from VoltConsole. The pods are able to query kube-api for services and lifecycle of pods.

Step 10: Deploy apps.

Deploy the Hipster shop app using the app manifest. Download the manifest from here.

  • Create a namespace with the same name as that of your app namespace in VoltConsole.
oc create ns foobang-chan 
  • Deploy the apps.
oc -n foobang-chan create -f hipster.yaml

Note: When VoltMesh deployed as pods inside OCP, the frontend service can be of type ClusterIP.


Step 1: Create origin pool.
  • Log into VoltConsole and click App on the namespace selector. Select your namespace from the drop-downlist to change to it.

Note: Ensure that you have the namespace with same name as the namespace on OCP cluster.

  • Go to Manage -> Load Balancers -> Origin Pools. Click Add Origin Pool.
  • Enter a name in metadata section.
  • Go to Basic Configuration and select K8s Service Name of Origin Server on given Sites for the Select Type of Origin Server field.
  • Enter service name in the <servicename>.<namespace> in the Service Name field. This example sets frontend.foobang-chan.
  • Select Site for the Site or Virtual Site field and select your VoltMesh site from the drop-down list for the Site field.
  • Select Outside Network for the Select Network on the site field and set Port as 80.

orig pool
Figure: Origin Pool Configuration

  • Click Save and Exit

Note: See Origin Pools for more information on origin pools.

Step 2: Create HTTP load balancer.
  • Go to Manage -> Load Balancers -> HTTP Load Balancers in your app namespace on VoltConsole. Click Add HTTP Load Balancer.
  • Enter a name in the metadata section.
  • Go to Basic Configuration and enter a domain name in the Domains field.
  • Select to enable the Automatically Manage DNS Records field.
  • Go to Default Origin Servers section and click Configure under the Origin Pools field.
  • Click Add item in the Origin Pools page. Click on the Select Origin Pool drop-down list and select the origin pool created in the previous step. Click Apply.

http lb
Figure: HTTP Load Balancer Configuration

  • Scroll down and click Save and Exit.
Step 3: Verify that the origin servers are displayed in load balancer monitoring.
  • Go to Virtual Hosts -> HTTP Load Balancers in your app namespace on VoltConsole. A list of load balancers gets displayed.
  • Click on the load balancer you created in previous step. The General Monitoring view with the Dashboard tab loads by default.
  • Click on the Origin Servers tab.

lb orig pool
Figure: Origin Server in HTTP Load Balancer Monitoring

  • Verify that the page shows the pod running your apps as the origin server.

Note: When VoltMesh is deployed as the Ingress/Egress Gateway outside the OCP, it discovers service using the service discovery created. When deployed as pods inside the OCP, it auto-discovers the service using the service name and namespace name as specified in the origin pool.

Step 4: Verify that the application is accessible from internet.

Send a request to your domain using a web browser or a utility such as CURL and confirm that the request is successful and your web page is loaded or HTTP response has data with 200 code.


Concepts


API References