Create Multi Node KVM Site on Equinix

Objective

This document provides instructions on how to install Volterra multi-node KVM site on Equinix baremetal service. To know more about Volterra sites, see Volterra Site.

Using the instructions provided in this document, you can deploy 3 Equinix metal servers with CentOS as host operating system, create virtual machines on them, install Volterra software on the VMs, and register the nodes into a multi-node cluster site.


Prerequisites


Configuration

In case of this deployment, the VoltMesh nodes require two interfaces attached. The first interface is the outside interface through which services running on the node can connect to the internet. The second interface is the inside interface whose IP address is the default gateway IP address for all the application workloads and services present in the private subnets.

The following image presents a high level view of the network topology:

equinix voltmesh topology
Figure: VoltMesh Equinix Deployment Topology

Deploy Equinix Metal Servers

In case of deploying VoltMesh nodes on Equinix, first deploy 3 on-demand Equinix Metal servers.

Note: The Equinix Metal support various types of server instance sizes and different CPU architectures. However, VoltMesh can only be deployed on x86 CPU architecture and not supported for Atom or Arm.

Also, the following Equinix Metal instances sizes do not support the required Layer 2 functionality required for a successful deployment:

  • ​​t1.small.x86
  • c1.small.x86
  • X1.small.x86

Perform the following steps:

Step 1: Start creating Equinix Metal Servers.
  • Log into the Equinix Console with your account credentials.
  • Go to Servers -> Choose Deploy Type -> Deploy On Demand Servers

image22
Figure: Equinix Metal Location and Deploy Type

  • Select a location and server type.
Step 2: Set operating system and the number of servers.
  • Select CentOS and choose CentOS 7 as the operating system version.
  • Use the + button to set 3 servers and enter names for your servers.

image29
Figure: Operating System and Server Names

Step 3: Optionally, add the user data and start deploying the servers.
  • Prepare the user data for configuring packages. The following is a sample:
#cloud-config

package_upgrade: true

packages:
    - qemu-kvm
    - qemu-img
    - virt-manager
    - libvirt
    - libvirt-python
    - libvirt-client
    - virt-install
    - virt-viewer
    - bridge-utils

runcmd:
   - systemctl start libvirtd
   - virsh net-undefine default
   - virsh net-destroy default
  • Enable the Add User Data option and paste the configuration you prepared. Click the Deploy Now button to start deploying the servers.

image14
Figure: Add User Data and Deploy Servers

Step 4: Verify that the servers are deployed.

After successful deployment, the Servers page displays the list of metal servers deployed.

image16
Figure: Deployed Servers

Step 5: Request elastic IP block.

After creating the servers, it is required to configure IP addresses by obtaining elastic IP address block.

  • Go to IPs & Networks -> IPs.

image33
Figure: IPs & Networks Configuration

  • Select Public IPv4, choose a location, choose quantity, and click Submit Request. Repeat for Public IPv6 option as well.

image26
Figure: Request Public IPv4 Block

  • Verify the IP addresses assigned in the PROJECT IP ADDRESSES view.

image10
Figure: Verify the IP Address Blocks

Step 6: Create VLANs.

The dual NIC VoltMesh installation requires 2 VLANs, one for the internal and one for the external interface.

  • Go to IPs & Networks -> Layer 2. Click Add New VLAN. Set the description to indicate it is internal VLAN and click Add.

image9
Figure: Add Internal VLAN

  • Repeat the previous step to create another VLAN and set the description to indicate that it is external VLAN.

image9
Figure: Add External VLAN

  • Verify the created VLANs.

image39
Figure: Created VLANs


Convert Server Networking and Assign IPs/Vlans

When a Equinix Metal server is first deployed, there is a single bonded interface. To support the dual NIC VoltMesh installation, the server networking type needs to be converted to Hybrid Bonded. By converting to hybrid bonded, the VLANs created in the previous section can be assigned to the Metal instance.

Do the following:

Step 1: Convert networking to hybrid bonded mode.
  • Go to Servers page and click on a server for which the networking is required to be converted.

image3
Figure: Equinix Server List

  • The server overview page gets displayed.

image44
Figure: Server Overview

  • Select Network on the left menu to open the network page. Click Convert to Other Network Type.

image13
Figure: Connvert Network Type

  • Select Hybrid in the network type selection window. Select Bonded in the options and select one of the VLANs to assign.

image21
Figure: Convertion to Hybrid Bonded with VLAN

Note: You can select any of the created VLANs. The second VLAN will be assigned in the next step. Therefore, the order of adding does not matter.

Step 2: Assign second VLAN to the metal instance.
  • In the Network page, go to the Layer 2 section and click Add New Vlan.

image42
Figure: Add New VLAN

  • In the Add New Vlan window, select the other VLAN in the Network field and click Add.

image8
Figure: Add Second VLAN

Step 3: Add elastic IP addresses to the metal instances.

Each metal instance requires two elastic IP addresses, one IP address for the network bridge interface, and another for the VoltMesh external interface. To use an elastic IP address, it must be assigned to the metal instance.

From the Elastic IP block that was allocated in previous section, assign two /32 addresses.

In this example, the allocated IP block is 147.28.142.240/29 and the first or last IP addresses are not used as they represent the network and broadcast addresses. The following table shows the sample IP addresses this example sets.

Metal Instance Bridge IP address Volterra external IP address
VoltMesh-01 147.28.142.241 147.28.142.242
VoltMesh-02 147.28.142.243 147.28.142.244
VoltMesh-03 147.28.142.245 147.28.142.246

After planning your address scheme, do the following:

  • In the Network page for your server, click Assign New Elastic in the Elastic IPs section.
  • In the Add Elastic IP window, select Public IPv4 for the Address Type field. Select /32 for the Length field and select an address for the Address/Range field.

image19
Figure: Add Elastic IP Address

  • Click Add. Repeat the above steps to add another IP address.

image43
Figure: Elastic IP Addresses Added

  • Click on the SERVER ACTIONS field and select REBOOT to reboot the metal instance to apply the changes.

image5
Figure: Reboot the Metal Instance

Note: Repeat the steps for the all the metal instances.


Deploy Volterra Site on Equinix Metal Servers

Deploying multi-node Volterra site includes installing VoltMesh node on each of the metal server. This involves configuring KVM on the instances, creating VMs with VoltMesh node images, performing post-installation configuration, and registering the VoltMesh nodes.

Configure KVM on Metal Instances

Configure the KVM hypervisor to support installation of VoltMesh software. Repeat for each metal instance.

Step 1: Connect to your metal instances using SSH.
  • SSH into the first metal instance. Click ... -> SSH Info for your metal instance.

image41
Figure: Obtain SSH Information

  • Click Copy to copy the SSH command displayed on the screen.

image18
Figure: Copy SSH Command

  • Using your preferred SSH client, login to the server as root.

image46
Figure: Connect to Metal Instance Using SSH

Step 2: Edit system configuration to support VoltMesh cluster operation
  • Open the sysctl.conf file for editing.
vi /etc/sysctl.conf
  • Add the following contents to the end of the file and save the changes.
vm.nr_hugepages = 1200
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
  • Load bridge filter module.
modprobe br_netfilter
  • Apply the changes.
sysctl -p
Step 3: Download VoltMesh KVM software.
  • Copy link to the latest software from the KVM Images page and use wget to download the image. The following is a sample command:
wget -O /var/lib/libvirt/images/volterra.iso https://vesio.blob.core.windows.net/dev/images/centos/7.2009.10-202106210938/vsb-ves-ce-certifiedhw-generic-production-centos-7.2009.10-202106210938.1624275431.iso
  • Wait for the download to complete. Download progress is indicated and after completion, the command prompt is returned.

image32
Figure: Download of VoltMesh KVM Software

Step 4: Create a disk image for VoltMesh node.

Enter the following command:

qemu-img create /var/lib/libvirt/images/disk1.qcow2 30G
Step 5: Define KVM internal and external network interfaces.

Define the internal and external KVM network interfaces and apply configuration parameters. The format of the device name is bond0.<VLAN ID>. In this example, the VLAN IDs assigned to the Volterra internal and external VLANs are 1000 and 1001 respectively.

  • Create the internal VLAN interface file for editing.
vi /etc/sysconfig/network-scripts/ifcfg-bond0.1000
  • Add the following values and save the file:
DEVICE=bond0.1000
ONBOOT=yes
BOOTPROTO=none
BRIDGE=br0
VLAN=yes
  • Create the external VLAN interface:
vi /etc/sysconfig/network-scripts/ifcfg-bond0.1001
  • Add the following values and save the file:
DEVICE=bond0.1001
ONBOOT=yes
BOOTPROTO=none
BRIDGE=br1
VLAN=yes
Step 6: Configure internal VLAN interface network bridge IP addresses.

Define the Volterra internal VLAN interface network bridge IP addresses. The external Volterra interface uses public IP addresses. For internal interface, the RFC1918 private addresses are used. This step configures only the bridge IP Address for each instance.

Note: The Volterra internal IP Address configuration is done within the VoltConsole during a later step.

You can choose an IP address scheme as per your choice for internal addressing except the IP address space used by the bond0:0 interface.

Note: You can find the IP being used by the bond0:0 interface by running the ip addr command.

In this example, the 192.168.100.0/24 address block is used. The following is the allocation scheme for this example:

Metal Instance Bridge IP Address Volterra Internal IP Address
VoltMesh-01 192.168.100.10 192.168.100.11
VoltMesh-02 192.168.100.20 192.168.100.21
VoltMesh-03 192.168.100.30 192.168.100.31

Note: Bridge IP Address is the SiteLocalInside gateway and Volterra internal IP address is the SiteLocalInside IP prefix within the VoltConsole.

This example shows configuration steps for a metal instance.

  • Create and open the internal bridge interface file for editing
vi /etc/sysconfig/network-scripts/ifcfg-br0
  • Add DNS1, DNS2, IPADDR, and PREFIX settings and save the file. The following is a sample for first metal instance.
DEVICE=br0
STP=no
TYPE=Bridge
BOOTPROTO=none
DEFROUTE=yes
NAME=br0
ONBOOT=yes
DNS1=8.8.8.8
DNS2=8.8.4.4
IPADDR=192.168.100.10 
PREFIX=24
  • Repeat the steps for each metal instance with the appropriate IP address based on your address scheme.
Step 7: Configure external VLAN interface network bridge IP addresses

Define the Volterra external VLAN network bridge IP addresses. It is recommended to use the IP address pattern as shown in this example. Ensure not to use the first and last IP addresses in the /29 address range.

This example uses the elastic IP block 147.28.142.240/29.

Note: This step configures only the bridge IP address for each instance. Volterra external IP Address configuration is done within the VoltConsole during a later step.

The following is the scheme used for this example:

Metal Instance Bridge IP address Volterra external IP address
VoltMesh-01 147.28.142.241 147.28.142.242
VoltMesh-02 147.28.142.243 147.28.142.244
VoltMesh-03 147.28.142.245 147.28.142.246

Note: Use the Elastic IP addresses from the block obtained in the Deploy Equinix Metal Servers chapter. The Bridge IP Address is the SiteLocal GW and Volterra external IP address is the SiteLocal IP prefix within the VoltConsole.

This example shows configuration steps for a metal instance.

  • Create and open the internal bridge interface file for editing
vi /etc/sysconfig/network-scripts/ifcfg-br1
  • Add DNS1, DNS2, IPADDR, and PREFIX settings and save the file. The following is a sample for first metal instance.
DEVICE=br1
STP=no
TYPE=Bridge
BOOTPROTO=none
DEFROUTE=yes
NAME=br1
ONBOOT=yes
DNS1=8.8.8.8
DNS2=8.8.4.4
IPADDR=147.28.142.241
PREFIX=29
  • Repeat the steps for each metal instance with the appropriate IP address based on your address scheme.

Reboot the instances and ping the elastic IP addresses to verify connectivity.


Create and Configure VoltMesh Virtual Machines

Create a VoltMesh virtual machine, assign IP addresses to internal and external interfaces and configure VoltMesh.

Step 1: Create a VoltMesh VM using the KVM CLI command.
  • Enter the following command:
virt-install --name volterra \
--ram 16384 --vcpus=8 \
--os-variant=centos7.0 \
--network bridge=br1,model=virtio \
--network bridge=br0,model=virtio \
--accelerate \
--disk path=/var/lib/libvirt/images/disk1.qcow2,bus=virtio,cache=none,size=64 \
--cdrom /var/lib/libvirt/images/volterra.iso \
--noautoconsole --noreboot
  • Enter the following commands to start the VM and for it to automatically start when the underlying host operating system is started:
virsh start volterra
virsh autostart volterra
  • After the VM is started, use the KVM console to connect to it. The ID of the VM assigned by KVM is required for this and enter the following command to obtain the ID assigned to the VM.
virsh list

image37
Figure: Obtain the Virtual Machine ID

Step 2: Connect to the local Volterra console.
  • Connect to the local console of the started VM using the VM ID obtained in the previous step. The following is a sample command:
virsh console 1
  • At the login prompt, use admin as the username and Volterra123 as the password.

image20
Figure: Volterra Local Console

  • After logging in for the first time, choose a new password. Passwords must be at least 8 characters and include at least one upper case letter.
Step 3: Configure VoltMesh external interface.
  • Use the tab key to open the configuration menu and navigate to configure-network and press enter key to start network configuration.

image4
Figure: Select Outside Network Interface

  • Use the arrow keys to navigate to the OUTSIDE option and press the spacebar to select.
  • On the screen that follows, populate the fields as shown in the following image:

image34
Figure: Configure Outside Interface

Step 4: Configure VoltMesh internal interface.
  • Use the tab key to open the configuration menu and navigate to configure-network and press enter key to start network configuration.

image11
Figure: Select Inside Network Interface

  • Use the arrow keys to navigate to the INSIDE option and press the spacebar to select.
  • On the screen that follows, populate the fields as shown in the following image:

image15
Figure: Configure Inside Interface

Note: Ensure that you set the primary interface as OUTSIDE.

Step 5: Create or obtain a site token.

To configure VoltMesh and register your site, a site token is required. You can create a new token or use an existing token. This example shows steps to create a new token.

  • Log into VoltConsole and go to Manage -> Site Management -> Site Tokens in the System namespace.
  • Click Add site token. Enter a name for the token and click Add site token.
  • A site token gets generated. Note down the generated token code under the UID field for your token in the list of displayed tokens.
Step 6: Configure other required VoltMesh settings.

At this point, you can either connect using the KVM console or use SSH to connect. In case of SSH you can use the elastic IP address you configured in previous section. This example shows connecting using SSH.

  • Connect to the VoltMesh node using SSH and the elastic IP address for your instance.

image1
Figure: Connect Using SSH

  • Use the tab key to open the configuration menu and navigate to configure and hit enter key to begin configuration.
  • On the screen that follows, populate the fields as shown in the following image:

image35
Figure: VoltMesh Node Configuration

Note: For What is your token and What is your site name?, enter the same name and token for all nodes. Use the token created in the previous step.

  • Type y and enter to confirm configuration. Your node will start to connect to VoltConsole for registration.
Step 7: Register your site.
  • Go to Manage -> Site Management -> Registrations in the system namespace.
  • Choose your site from the list of sites displayed under Pending Registrations tab. Click the approval button in blue checkmark.

image40
Figure: Pending Registrations

  • On the page that opens, set the cluster size to 3 and then click the Save and Exit button to complete the registration process. Make sure that you set each node to the same cluster size.

image36
Figure: Registration Fields

  • Repeat above steps on remaining nodes.

image12
Figure: Registration for All Nodes

Note: It may take few minutes for the registrations to complete and site to become ONLINE.


Volterra Concepts