Secure Kubernetes Gateway

Objective

This guide provides instructions on how to create a Secure Kubernetes Gateway using VoltConsole and VoltMesh.

The steps to create Secure Kubernetes Gateway are:

SeqSecK8GW
Figure: Steps to Deploy Secure Kubernetes Gateway

The following images shows the topology of the example for the use case provided in this document:

TopSecK8GW
Figure: Secure Kubernetes Gateway Sample Topology

Using the instructions provided in this guide, you can deploy an Secure K8s Gateway in your Amazon Virtual Private Cloud (Amazon VPC), discover the cluster services from that VPC, setup loadbalancer for them, and secure those services with network policies and service policies.

The example shown in this guide deploys the Secure K8s Gateway on a single VPC for an application called as Hipster Webapp deployed in two clusters with one representing the development environment and other representing the staging environment. The application consists of the following services:

  • frontend
  • cartservice
  • productcatalogservice
  • currencyservice
  • paymentservice
  • shippingservice
  • emailservice
  • checkoutservice
  • recommendationservice
  • adservice
  • cache

Prerequisites

  • VoltConsole SaaS account.

    Note: If you do not have an account, see Create a VES Account.

  • Amazon Web Services (AWS) account.

    Note: This is required to deploy a Volterra site.

  • Volterra vesctl utility.

    Note: See vesctl for more information.


Configuration

The use case provided in this guide sets up a Volterra site as Secure K8s Gateway for the ingress and egress traffic for the two K8s clusters deployed in the Amazon VPC. The example web application has a front end service to which all the user requests are sent and it redirects to other services accordingly. The following actions outline the activities in setting up the Secure K8s Gateway:

  1. The frontend service of the application needs to be externally available. Therefore, two HTTPS load balancers are created for each cluster.
  2. The loadbalancer TLS configuration secured by applying Volterra Blindfold encryption to the TLS key.
  3. A WAF configuration is applied to secure the externally available loadbalancer VIPs.
  4. Security policies are configured to allow egress communication to Google DNS for DNS query resolution and github.com for code repository management.
  5. Rest all communication from the k8s clusters is configured to be denied.

Note: Ensure that you keep the Amazon Elastic IP VIPs ready for later use in configuration.

Step 1: Deploy Site

The following video shows the site deployment workflow:

Perform the following steps to deploy a Volterra site as the Secure K8s Gateway in your VPC.

Step 1.1: Download the Volterra quickstart deployment script.
docker run --rm -v $(pwd):/opt/bin:rw docker.io/volterraio/volt-terraform:latest cp /deploy-terraform.sh /opt/bin
Step 1.2: Prepare input variables file for terraform deployment. The following example shows the sample entries of the input variables:
{
    "vpc_cidr": "192.168.32.0/22",
    "deployment": "skgw-demo-1",
    "fleet_label": "skgw-demo-1",
    "access_key": "<aws_access_key>",
    "secret_key": "<aws_secret_key>",
    "api_p12_file": "<path-to-api-credentials>",
    "api_url": "https://<tenant>.console.ves.volterra.io/api",
    "machine_public_key": "<pub-key>"
}

Note: Download the API credentials in the Volterra console from the IAM->API Credentials option. The credentials get downloaded in the file with the .p12 extension.

Step 1.3: Deploy the VPC with the Secure K8s Gateway and the K8s clusters using the deployment script.
./deploy-terraform.sh apply -p aws -i <tfvars> -tn self-serve/secure_kgw 

The deployment performs the following:

  • Creates AWS infrastructure such as VPC, subnets, and route-tables
  • Deploys AMI instance of Volterra node into the created VPC
  • Performs automatic site registration (approval)
  • Creates Volterra fleet and network connector objects with fleet label as transit-vpc
  • Creates the hipster-dev and hipster-stage EKS clusters
  • Creates the hipster-dev and hipster-stage namespaces
  • Deploys the Hipster shop web application in the clusters

Note: The deployment approximately takes 10 to 15 minutes to complete.

After the deployment is complete, download the kubeconfig files of the created K8s clusters using the following commands:


./deploy-terraform.sh output -n skgw-demo-1 dev_cluster_kubeconfig >  hipster-dev-kubeconfig

./deploy-terraform.sh output -n skgw-demo-1 stage_cluster_kubeconfig >  hipster-stage-kubeconfig

Step 2: Discover Services

Discovering services in the VPC requires configuring service discovery objects for the front end service of each cluster.

The following video shows the service discovery workflow:

Perform the following steps for discovering services.

Step 2.1: Create a secret policy.

Log into the Volterra console and navigate to Security -> Secret Policies. Click Add secret policy and enter the following configuration:

  • Enter a name in the Name field. This example uses the name secure-kgw-demo.
  • Select First Rule Match for the Rule Combining Algorithm field.
  • Select the Allow Volterra checkbox. This allows Volterra infrastructure services to decrypt the secret.
  • Click Add secret policy to complete secret policy creation.

SecPol
Figure: Create Secret Policy

Note: Secret policy is created to be used in encrypting the Kubeconfig files and also in loadbalancer configuration.

Step 2.2: Obtain a pubic key and store the output to a file.
vesctl request secrets get-public-key > hipster-co-public-key
Step 2.3: Obtain a policy document for the secret policy created in Step 2.1 and store the output to a file.
vesctl request secrets get-policy-document --namespace system --name secure-kgw-demo > secure-kgw-demo-policy-doc
Step 2.4: Encrypt the kubeconfig files using the Volterra Blindfold.

Use the public key and policy document created in Step 2.2 and Step 2.3. Store the output a file.

For the hipster-dev cluster:

vesctl request secrets encrypt --policy-document secure-kgw-demo-policy-doc --public-key hipster-co-public-key hipster-dev-kubeconfig > hipster-dev-bf-secret

For the hipster-stage cluster:

vesctl request secrets encrypt --policy-document secure-kgw-demo-policy-doc --public-key hipster-co-public-key hipster-stage-kubeconfig > hipster-stage-bf-secret
Step 2.5: Go back to Volterra console and start discovery object creation.

Select Manage -> Site Management. Select Discovery and click Add discovery. Enter the configuration as per the following guidelines:

  • Enter a name in the Name field.
  • Select Site for the Where field.
  • Click Select ref, select hipster-transit-svcs as the site, and click Select ref.
  • Select Site Local Inside Network for the Network Type field.
  • Select Kubernetes for the Type field.
  • Select K8s for the Discovery Service Access Information field and select Kubeconfig for the Oneoff field.

Adddisc
Figure: Discovery Object Configuration

Step 2.6: Perform configuration for the Kubeconfig.

Click Kubeconfig and enter the configuration as per the following guidelines:

  • Select Blindfold for the Secret info field.
  • Enter the encrypted secret in the Location field. Use the secret obtained in Step 2.4.

    Note: Use cat <filename> to copy the secret.

DiscBlf
Figure: Discovery Object Kubeconfig Setting

Step 2.7:Complete creating discovery object.

Select Apply and Add discovery to create discovery object.

Step 2.8: Repeat Step 2.5 to Step 2.7 for the second K8s cluster.
Step 2.9: Change to the namespace of first cluster (hipster-dev) and add endpoint.

Select Manage->Endpoints. Click Add endpoint and enter the configuration as per the following guidelines:

  • Enter a name in the Name field.
  • Enter 8080 for the Port field.
  • Select Service Info for Endpoint Specifier field.
  • Select Kubernetes for the Discovery field and Service Name for the Service field.
  • Enter frontend.hipster-dev as the service name.
  • Enter Site for the Where field and select the hipster-transit-svcs for the Select ref field.
  • Select Site Local Inside Network for the network type and TCP as the protocol.
  • Click Add endpoint to create endpoint.

EP
Figure: Endpoint Creation

Step 2.10: Add healthcheck.

Select Manage->Healthcheck. Click Add healthcheck and enter the configuration as per the following guidelines:

  • Enter a name in the Name field.
  • Select HTTP Healthcheck the Health check field.
  • Enter 5 for Timeout and Interval fields. This sets timeout and interval as 5 seconds for health check.
  • Click Add healthchcek.

HC
Figure: Healthcheck Creation

Step 2.11: Add cluster.

Select Manage->Clusters. Click Add cluster and enter the configuration as per the following guidelines:

  • Enter a name in the Name field.
  • Select the endpoint created in Step 2.11 for the Select endpoint field.
  • Select the healthcheck object created in Step 2.12 for the Select healthcheck field.
  • Select Round Robin for the LoadBalancer Algorithm field.
  • Click Add cluster.

cluster
Figure: Cluster Creation

Step 2.12: Repeat from Step 2.9 to Step 2.11 with the namespace as hipster-stage for the hipster-stage K8s cluster.

Step 3: Create Loadbalancer

Creating a loadbalancer requires you to configure routes pointing to the cluster objects created, advertise policies, and virtual hosts for your K8s clusters.

Note: This chapter provides the details for configuring the required components for loadbalancer. For detailed instructions on creation of virtual host, see Create Virtual Host.

The following video shows the loadbalancer creation workflow:

Perform the following to configure loadbalancer:

Step 3.1: Change to the hipster-dev namespace and add route towards the created cluster.

Select Manage -> Routes. Click Add route to create the route. In the route configuration form, add the cluster object for the Add route field. Select the cluster created for this namespace in the Discover Services chapter. Add Regex for the Path Match field in the Match option.

Step 3.2:Add advertise policy.

Select Manage -> Advertise Policies. Click Add advertise policy and select Site for the Where field and select the hipster-transit-svcs site for the Select ref field.

Step 3.3: Obtain a certificate and key signed from a publicly known CA. Optionally, you can also generate a self signed cert for your application domain.
Step 3.4: Encrypt the private key of the certificate using the Volterra Blindfold.

Use the public key and policy document obtained in the Step 2.4 and Step 2.5 of the Discover Services chapter. This example shows the sample for the hipster-dev cluster. Store the output to a file.

vesctl request secrets encrypt --policy-document secure-kgw-demo-policy-doc --public-key hipster-co-public-key tls.key > tls.key.secret

Note: The tls.key is the private key of the certificate you generated.

Step 3.5: Add a virtual host.

Select Manage -> Virtual Hosts. Click Add virtual host and set the configuration as per the following guidelines:

  • Enter name, application domain, and your proxy type. This sample uses HTTPS_PROXY as the proxy type and hipster-shop-dev.demo.helloclouds.app as the domain.
  • Select previously defined route
  • Apply TLS configuration with the TLS certificate (get Base64 string of your certificate and enter it in the string:/// format)
  • Configure private key - select secret info type as Blindfold secret and enter it under Location, click Apply and Add virtual host.
Step 3.6: Repeat the steps from the hipster-stage namespace for the *hipster-stage* cluster.

Note: Add a DNS record to your domain provider so that for the domain name, it points to the right Elastic IP. Set your DNS to enable the following EIP VIPs:

  • hipster-shop-dev.demo.helloclouds.app
  • Hipster-shop-stage.demo.helloclouds.app

Step 4: Secure the Ingress and Egress

Securing the ingress and egress traffic requires you to set the network policies, service policies, WAF, and network firewall.

The following video shows the workflow of securing the ingress and egress:

This example sets policies that allow egress traffic to github.com, allow the DNS resolution to 8.8.8.8/32, and block any other egress traffic. It also enables the WAF to allow all the ingress traffic.

Step 4.1: Create network policy rule.

Change to the system namespace and select Security -> Network Policy Rules. Click Add network policy rule and enter the following configuration:

  • Set a name for the rule and select Allow for Action field.
  • Click Add port and enter 53.
  • Select udp for Protocol.
  • Select Prefix for Remote Endpoint field and enter 8.8.8.8/32 for Prefix field.
  • Click Add network policy rule to complete creating the network policy rule.

NwPolRule
Figure: Network Policy Rule Configuration

Step 4.2: Create network policy.

Select Security -> Network Policies. Click Add network policy and enter the following configuration:

  • Set a name for the rule and select Prefix for the Local Endpoint field.
  • Click Add prefix and enter prefixes 192.168.33.0/24 and 192.168.34.0/24.
  • Select the network policy rule configured in previous step for the Egress Rules field.
  • Click Add network policy to complete creating the network policy.

NwPol
Figure: Network Policy Configuration

Step 4.3: Create network policy set.

Select Security -> Network Policy Sets. Click Add network policy set and enter the following configuration:

  • Set a name for the policy set.
  • Click Select policy and add the policy created in the previous step.
  • Click Add network policy set to complete creating the network policy.

NwPolSet
Figure: Network Policy Set Configuration

Step 4.4: Create service policy rule.

Select Security -> Service Policy Rules. Click Add service policy rule and enter the following configuration:

  • Set a name for the policy rule.
  • Select Allow for the Action field.
  • Click Add exact value in the Domain Matcher field and add github.com.
  • Click Add service policy rule to complete creating the service policy rule.

SrvcPolRule
Figure: Service Policy Rule Configuration

Step 4.5: Create service policy.

Select Security -> Service Policies. Click Add service policy and enter the following configuration:

  • Set a name for the policy.
  • Select First Rule Match for the Rule Combining Algorithm field.
  • Click Select rule and add the rule created in the previous step.
  • Click Add service policy to complete creating the service policy.

SrvcPol
Figure: Service Policy Configuration

Step 4.6: Create service policy set.

Select Security -> Service Policy Sets. Click Add service policy set and enter the following configuration:

  • Set a name for the policy.
  • Click Select policy and add the policy created in the previous step.
  • Click Add service policy set to complete creating the service policy set.

SrvcPolSet
Figure: Service Policy Configuration

Step 4.7: Create network firewall.

Select Security -> Network Firewall. Click Add network firewall and set the following configuration:

  • Set a name for the firewall.
  • Click Select network policy set and add the network policy set created in Step 4.3.
  • Click Select forward proxy policy set and add the service policy set created in Step 6.
  • Click Add network firewall to complete creating network firewall.
Step 4.8: Apply the network firewall to the fleet.

Select Manage -> Fleet and find the hipster-transit-svcs fleet from the displayed list. Click ... -> Edit to open the fleet edit form. Click Select network firewall and select the firewall created in the previous step. Click Save changes to apply the network firewall to the fleet of sites.

NwfToFleet
Figure: Network Firewall Addition to Fleet


Verification

The created Secure K8s Gateway allows egress traffic to github.com and only allows DNS resolution to the prefix 8.8.8.8/32. Egress traffic for all other destinations is blocked. Also, all ingress requests are allowed to the frontend service. Enter the following commands to verify this:

Step 1: Verify if the traffic to github.com is resolved using the Google DNS server with address 8.8.8.8.
kubectl --kubeconfig hipster-dev-kubeconfig -n hipster-dev exec -t frontend-578b65cdff-r48bf -- nslookup github.com 8.8.8.8
Step 2: Verify if the web request to github.com is allowed and results in successful response:
kubectl --kubeconfig hipster-dev-kubeconfig -n hipster-dev exec -t frontend-578b65cdff-r48bf -- wget --server-response -O /dev/null https://github.com
Step 3: Verify that the request using the Google DNS server with address 8.8.4.4 is blocked:
kubectl --kubeconfig hipster-dev-kubeconfig -n hipster-dev exec -t frontend-578b65cdff-r48bf -- nslookup github.com 8.8.4.4

Concepts