Virtual Kubernetes (vK8s) API Compatibility

Objective

In this document, you’ll get details about Virtual Kubernetes (vK8s) APIs supported in this release. Further details can be found here at Virtual Kubernetes (vk8s) under VES concepts.

Volterra vK8s APIs

Volterra supports a Kubernetes compatible API for centralized orchestration of applications across a fleet of sites (customer sites or Volterra Regional Edge). This API is “Kubernetes compatible” because not all Kubernetes APIs or resources are supported. However, for the API(s) that are supported, it is hundred percent compatible. We have implemented a distributed control plane within our global infrastructure to manage scheduling and scaling of applications across multiple (tens to hundreds of thousands of) sites, where each site in itself is also a Volterra managed physical K8s cluster.

Further details can be found here at Virtual Kubernetes (vk8s) under VES concepts.

Note: The supported Kubernetes version is v1.17.

Workload APIs

Deployment v1 APPs

A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.

Write Operations

Create POST /apis/apps/v1/namespaces/{namespace}/deployments
Patch PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}
Replace PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}
Delete DELETE /apis/apps/v1/namespaces/{namespace}/deployments/{name}
Delete Collection DELETE /apis/apps/v1/namespaces/{namespace}/deployments

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}
List GET /apis/apps/v1/namespaces/{namespace}/deployments
Watch GET /apis/apps/v1/watch/namespaces/{namespace}/deployments/{name}
Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/deployments

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale
Replace Scale PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale
Patch Scale PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale

Job v1 batch

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created. A simple case is to create one Job object in order to reliably run one Pod to completion. The Job object will start a new Pod if the first Pod fails or is deleted (for example due to a node hardware failure or a node reboot

Write Operations

Create POST /apis/batch/v1/namespaces/{namespace}/jobs
Patch PATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name}
Replace PUT /apis/batch/v1/namespaces/{namespace}/jobs/{name}
Delete DELETE /apis/batch/v1/namespaces/{namespace}/jobs/{name}
Delete Collection DELETE /apis/batch/v1/namespaces/{namespace}/jobs

Read Operations

Read GET /apis/batch/v1/namespaces/{namespace}/jobs/{name}
List GET /apis/batch/v1/namespaces/{namespace}/jobs
Watch GET /apis/batch/v1/watch/namespaces/{namespace}/jobs/{name}
Watch List GET /apis/batch/v1/watch/namespaces/{namespace}/jobs

Status Operations

Read Status GET /apis/batch/v1/namespaces/{namespace}/jobs/{name}/status

CronJob v1beta1 batch

A cronjob is a scheduled job and written in cron format and is useful for running periodic tasks.

Write Operations

Create POST /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
Patch PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Replace PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Delete DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Delete Collection DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs

Read Operations

Read GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
List GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
List All Namespaces GET /apis/batch/v1beta1/cronjobs
Watch GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs/{name}
Watch List GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs
Watch List All Namespaces GET /apis/batch/v1beta1/watch/cronjobs

Status Operations

Patch Status PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status
Read Status GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status
Replace Status PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status

Pod v1 Core

A Pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers. A Pod’s contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific “logical host” - it contains one or more application containers which are relatively tightly coupled — in a pre-container world, being executed on the same physical or virtual machine would mean being executed on the same logical host.

Read Operations

Read GET /api/v1/namespaces/{namespace}/pods/{name}
List GET /api/v1/namespaces/{namespace}/pods
Watch GET /api/v1/watch/namespaces/{namespace}/pods/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/pods

Status Operations

Read Status GET /api/v1/namespaces/{namespace}/pods/{name}/status

Misc Operations

Read Log GET /api/v1/namespaces/{namespace}/pods/{name}/log

Proxy Operations

Create Connect Portforward POST /api/v1/namespaces/{namespace}/pods/{name}/portforward
Get Connect Portforward GET /api/v1/namespaces/{namespace}/pods/{name}/portforward
Create Connect Exec POST /api/v1/namespaces/{namespace}/pods/{name}/exec
Get Connect Exec GET /api/v1/namespaces/{namespace}/pods/{name}/exec

ReplicaSet v1 Apps

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}
List GET /apis/apps/v1/namespaces/{namespace}/replicasets
Watch GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets/{name}
Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/scale

StatefulSet v1 Apps

StatefulSet is the workload API object used to manage stateful applications.Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

Write Operations

Create POST /apis/apps/v1/namespaces/{namespace}/statefulsets
Patch PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
Replace PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
Delete DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
Delete Collection DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
List GET /apis/apps/v1/namespaces/{namespace}/statefulsets
Watch GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets/{name}
Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale
Replace Scale PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale
Patch Scale PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale

Service APIs

Service v1 Core

An abstract way to expose an application running on a set of Pods as a network service. With Kubernetes you don’t need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.

Write Operations

Create POST /api/v1/namespaces/{namespace}/services Patch PATCH /api/v1/namespaces/{namespace}/services/{name} Replace PUT /api/v1/namespaces/{namespace}/services/{name}
Delete DELETE /api/v1/namespaces/{namespace}/services/{name}

Read Operations

Read GET /api/v1/namespaces/{namespace}/services/{name}
List GET /api/v1/namespaces/{namespace}/services
Watch GET /api/v1/watch/namespaces/{namespace}/services/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/services

Status Operations

Read Status GET /api/v1/namespaces/{namespace}/services/{name}/status

Endpoints v1 Core

Endpoints track the IP Addresses of the objects the service send traffic to. When a service selector matches a pod label, that IP Address is added to your endpoints.

Read Operations

Read GET /api/v1/namespaces/{namespace}/endpoints/{name}
List GET /api/v1/namespaces/{namespace}/endpoints
Watch GET /api/v1/watch/namespaces/{namespace}/endpoints/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/endpoints

Config and Storage APIs

Secret v1 Core

Kubernetes secret objects let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image. See Secrets design document for more information.

Write Operations

Create POST /api/v1/namespaces/{namespace}/secrets
Patch PATCH /api/v1/namespaces/{namespace}/secrets/{name}
Replace PUT /api/v1/namespaces/{namespace}/secrets/{name}
Delete DELETE /api/v1/namespaces/{namespace}/secrets/{name}
Delete Collection DELETE /api/v1/namespaces/{namespace}/secrets

Read Operations

Read GET /api/v1/namespaces/{namespace}/secrets/{name}
List GET /api/v1/namespaces/{namespace}/secrets
Watch GET /api/v1/watch/namespaces/{namespace}/secrets/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/secrets

ConfigMap v1 Core

ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. This page provides a series of usage examples demonstrating how to create ConfigMaps and configure Pods using data stored in ConfigMaps.

Write Operations

Create POST /api/v1/namespaces/{namespace}/configmaps
Patch PATCH /api/v1/namespaces/{namespace}/configmaps/{name}
Replace PUT /api/v1/namespaces/{namespace}/configmaps/{name}
Delete DELETE /api/v1/namespaces/{namespace}/configmaps/{name}
Delete Collection DELETE /api/v1/namespaces/{namespace}/configmaps

Read Operations

Read GET /api/v1/namespaces/{namespace}/configmaps/{name}
List GET /api/v1/namespaces/{namespace}/configmaps
Watch GET /api/v1/watch/namespaces/{namespace}/configmaps/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/configmaps

PersistentVolumeClaim v1 Core

Managing storage is a distinct problem from managing compute. The PersistentVolume subsystem provides an API for users and administrators that abstracts the details of how storage is provided from how it is consumed. To do this, K8s introduce two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., they can be mounted once read/write or many times read-only).

Write Operations

Create POST /api/v1/namespaces/{namespace}/persistentvolumeclaims
Patch PATCH /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
Replace PUT /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
Delete DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
Delete Collection DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims

Read Operations

Read GET /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
List GET /api/v1/namespaces/{namespace}/persistentvolumeclaims
Watch GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims

Cluster APIs

Node v1 Core

A node is a worker machine in Kubernetes, previously known as a minion. A node may be a VM or physical machine, depending on the cluster. Each node contains the services necessary to run pods and is managed by the master components. The services on a node include the container runtime, kubelet and kube-proxy.

Read Operations

Read GET /api/v1/nodes/{name}
List GET /api/v1/nodes
Watch GET /api/v1/watch/nodes/{name}
Watch List GET /api/v1/watch/nodes

Status Operations

Read Status GET /api/v1/nodes/{name}/status

Namespace v1 Core

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.

Read Operations

Read GET /api/v1/namespaces/{name}
List GET /api/v1/namespaces
Watch GET /api/v1/watch/namespaces/{name}
Watch List GET /api/v1/watch/namespaces

PersistentVolume v1 Core

Managing storage is a distinct problem from managing compute. The PersistentVolume subsystem provides an API for users and administrators that abstracts the details of how storage is provided from how it is consumed. To do this, K8s introduce two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., they can be mounted once read/write or many times read-only).

Read Operations

Read GET /api/v1/persistentvolumes/{name}
List GET /api/v1/persistentvolumes
Watch GET /api/v1/watch/persistentvolumes/{name}
Watch List GET /api/v1/watch/persistentvolumes

Status Operations
Read Status GET /api/v1/persistentvolumes/{name}/status

Metadata APIs

Event v1 core

Read GET /api/v1/namespaces/{namespace}/events/{name}
List GET /api/v1/namespaces/{namespace}/events List All Namespaces GET /api/v1/events
Watch GET /api/v1/watch/namespaces/{namespace}/events/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/events
Watch List All Namespaces GET /api/v1/watch/events

Access Control APIs

ServiceAccount v1 core

Write Operations

Create POST /api/v1/namespaces/{namespace}/serviceaccounts
Patch PATCH /api/v1/namespaces/{namespace}/serviceaccounts/{name}
Replace PUT /api/v1/namespaces/{namespace}/serviceaccounts/{name}
Delete DELETE /api/v1/namespaces/{namespace}/serviceaccounts/{name}
Delete Collection DELETE /api/v1/namespaces/{namespace}/serviceaccounts

Read Operations

Read GET /api/v1/namespaces/{namespace}/serviceaccounts/{name}
List GET /api/v1/namespaces/{namespace}/serviceaccounts
List All Namespaces GET /api/v1/serviceaccounts
Watch GET /api/v1/watch/namespaces/{namespace}/serviceaccounts/{name}
Watch List GET /api/v1/watch/namespaces/{namespace}/serviceaccounts
Watch List All Namespaces GET /api/v1/watch/serviceaccounts


Resource Management for Volterra vK8s

The Volterra vK8s resources can be managed using the ves.io/ annotations in the deployments. These annotations are supported for the deployments on both network cloud and edge cloud.

The following describe the supported annotations to manage Volterra vK8s resources.

  • ves.io/sites: namespace/name: The namespace is the site's namespace and name is the site's name. In case of multiple sites, the ves.io/sites: namespace/name,namespace/name format can be used. In case of CE sites, the system string should be specified as namespace along with CE name. In case of RE sites, the ves-io-system string should be specified as the namespace.

The following is an example annotation for launching vK8s resources to RE sites:

    "annotations": {
      "ves.io/sites": "ves-io-system/pa4-par, ves-io-system/ny8-nyc",
    }

The following is an example annotation for launching vK8s resources to CE sites:

    "annotations": {
      "ves.io/sites": "system/ce01, system/ce02",
    }

Note: When both ves.io/sites and ves.io/virtual-sites annotations are configured, the ves.io/sites annotation gets priority. This can be used to manage all vK8s resources.

  • ves.io/virtual-sites: namespace/name: The namespace is the virtual site's namespace and name is the virtual site's name. In case of multiple virtual sites, the ves.io/virtual-sites: namespace/name,namespace/name format can be used. The following can be managed using this annotation:

    • Deployments
    • Jobs
    • StatefulSets
    • Services
    • PVCs
    • ConfigMaps
    • Secrets

    This annotation selects the sites where the K8s object is instantiated. In case you do not specify this annotation, then the sites are selected based on the virtual sites reference in the vK8s object.

    Note: In case of using the virtual sites in the shared namespace of the ves.io tenant, the ves-io-shared string should be specified as the namespace.

  • ves.io/workload-flavor:workload-flavor-name: The workload-flavor-name can be tiny, medium, or large. The workload flavor maps to CPU and memory requests and limits for a pod. Each container in a Pod consumes compute resources - CPU and memory(RAM). All containers launched using the vK8s APIs have limits on the compute resources. Memory is measured in unit of MiB (mebibyte), power of 2. One MiB is 1,048,576 bytes. CPU is measured in unit of vCPU. Each vCPU is a thread on a CPU core. The following Workload Flavor types are supported:
Flavor Name CPU (in vCPU) Memory (in MiB) Notes
ves-io-tiny 0.1 256 Use the ves-io-tiny annotation for the deployments that are not resource intensive.
ves-io-medium 0.25 512 Use the ves-io-medium annotation for the deployments that consume resources moderately.
ves-io-large 1 2048 Use the ves-io-large annotation for the deployments that are resource intensive.

The following can be managed using this annotation:

  • Deployments
  • Jobs
  • CronJobs
  • DaemonSets
  • StatefulSets

Custom Workload Flavors are also supported. Workload Flavors can be chosen by adding the annotation ves.io/workload-flavor with the value set to the flavor name.

{ves.io/workload-flavor : "ves-io-medium"}

By default if no ves.io/workload-flavor annnotation is present, then vK8s will use the compute resources from the ves-io-tiny flavor.

This annotation determines the CPU and memory requests and limits for pods created by the above controllers and applies to all containers within the Pod. Individual containers within the pod can be override using the ves.io/workload-flavor-container-name: workload-flavor-name annotation where container-name is the name of the container in the pod spec. If no annotation is specified, the default flavor used for all containers in the pod is ves-io-tiny.

  • ves.io/proxy-type:proxy-type-name: The proxy-type-name can be HTTP_PROXY, TCP_PROXY, or TCP_PROXY_WITH_SNI. This applies to Services and determines the proxy type used for the internal virtual host associated with the K8s service. If no annotation is specified, the TCP_PROXY value is used as the default proxy.
  • ves.io/http2-enable: true|false: This applies to Services and determines whether HTTP2 upstream connection proxying is enabled for this service. If no annotation is specified, the default value is false indicating that the HTTP2 connection is not enabled for the service.

In case the pods are exceeding the set resource limits and you specified container-level resource limits in the pod spec, the container-level resource limits are overwritten by that of the workload-flavor annotation.

Note: The annotation ves.io/workload-flavor only takes effect on Volterra global network. The container-level resource limits in the pod spec will be honored in deployments on user's cloud (public or private) or user's edge VoltStack node.

In case of network cloud (Regional Edge(RE)), a per RE per vK8s CPU and memory resource limit is supported. If that is exceeded, no new Pods can be instantiated on the RE.


Concepts


Kubernetes References