Monitor HTTP Load Balancer

Objective

This document provides instructions on how to monitor your HTTP load balancer. Volterra provides support to monitor your application for security. To know more about how Volterra secures your applications, see Security.

Using the instructions provided in this document, you can check various views that present HTTP load balancer monitoring information such as statistics, events, etc.


Prerequisites


Monitor Load Balancer

Volterra offers 2 types of monitoring for load balancer—general monitoring and security monitoring. General monitoring offers operational information such as metrics, events, alerts, etc. Security monitoring offers security related information such as suspicious users, security events, API discovery, etc.

Go through the steps in the following chapters to learn detailed information on both monitoring.

Find a Load Balancer

Step 1: Select the namespace where the load balancer is configured.

Click App on the namespace selector and select your namespace from the drop-down list of namespaces.

image2
Figure: Navigate namespace

Step 2: Navigate to the load balancer monitoring.
  • Select Virtual Hosts -> HTTP Load Balancers on the configuration menu to display a list of load balancers.

image1
Figure: Load Balancer Monitoring

  • Hover mouse pointer on the tile of your load balancer. The General Monitoring and Security Monitoring options get enabled.

Explore General Monitoring

Click on General Monitoring for your load balancer in the load balancer monitoring page. The Dashboard tab is displayed by default.

In the various monitoring tabs, one or more common options are available to use. The following list describes commonly available options:

  • Refresh option refreshes the information displayed on the page.
  • Time interval selector to apply from a list of intervals. You can also set a custom 24 hour interval.
  • Filter option to apply filters to the displayed information.
  • Search option to search for specific information.

The following entries describe the various tab views available for general monitoring:

Dashboard

The dashboard tab displayed by default and offers a snapshot view for entire general monitoring information. Dashboards shows sections such as healthscore, alerts, metrics, clients, devices, policy, security, etc.

lb dash
Figure: Load Balancer Generic Monitoring Dashboard

The following list provides overview on the dashboard and the various sections it offers:

  • Metrics include requests, throughputs, and latency. However, you can filter the Top Clients view to display error rate also.
  • Client information includes details such as top clients, TLS fingerprints, client location, etc.
  • Device information includes device type and browser type.
  • Security information includes details such as top ASN, TSL/SSL statistics, URLs visited, service policy, etc. Also, HTTP error code trend is presented.
  • In case a section title is enabled with hyperlink, clicking the link switches to the tab for that section. For example, click on Requests or Active Alerts to switch to Requests or Alerts tabs.
Metrics

Click the Metrics tab to load the load balancer application metrics view:

The metrics present the trend of the following metrics in graph view over a default or configured time interval:

  • Health score in terms of percentage.
  • Request Rate, Error Rate, and Drop Rate.
  • Latency.
  • App Latency.
  • Client and Server Round-Trip Time (RTT).
  • Connection Duration.
  • Upstream and Downstream Throughput.

lb metrics
Figure: Load Balancer Metrics

Note: The metrics are grouped into fields such as Rate, Throughput, etc. A field may have one or more metrics.

Select a metric from the available fields on the right-hand side to display its trend. Hover your mouse pointer over a graph bar to view information specific to the time interval of that bar. You can also click on the bar to switch to Requests tab.

You can select any two metrics under a field such as Rate to display the combined graph for them. To do this, do the following:

  • Each metric has 2 small graph bar buttons to its left arranged in a vertical stack. Select a metric under one field.
  • Click on the lower graph button for other metric of the same field to display combined graph.

lb metrics combined
Figure: Combined Trend for 2 Metrics

Note: Click Last 1 hour dropdown on the upper right end of the dashboard and select a time interval to inspect your site dashboard for that interval. The default for this is 1 hour and maximum allowed interval is 24 hours. You can customize the interval by selecting the Custom option and choosing date range. This can also be set graphically by adjusting the controls beneath the main graph.

Traffic

Click the Traffic tab to view the monitoring page for traffic from requestor to origin server. The following information is displayed:

  • The view shows a graphical representation where the traffic trend is presented between requestor and origin server. The representation shows sections for the trend of traffic from requesting site to load balancer and then from load balancer to origin server.
  • Hover mouse pointer over any border bar to view details of the entity represented by that bar. For example, click on the bar representing origin servers to view detailed information on the applications at those origin servers.

lb traffic
Figure: Load Balancer Traffic View

  • Hover mouse pointer over any section to view details such as source, target, and request rate.
  • Click the Request Rate filter above the graphical representation and select Response Throughput to change the details to show response throughput instead of requests.
  • Click the Group by Service filter and select an option to change the origin server details. For example, if you select Group by Site, the bar representing origin server shows the site of origin server upon hovering mouse pointer over it.
Origin Servers

Click the Origin Servers tab to view the monitoring information for origin servers. In this view, you can see the list of origin servers for your load balancer and metrics associated with the origin server.

lb originserver
Figure: Load Balancer Origin Servers

Click > for an origin server entry to view its data in JSON format.

Alerts

Click the Alerts tab to load the alerts view. The active alerts are displayed by default.

You can filter the display for alerts of a specific severity using the severity selection options. All severity types are selected by default. Click on a severity selection option to hide the alerts for that severity. You can again click on it to display alerts for that severity.

Note: Severity selection options are color-coded and located beneath the Add filter option.

Use the toggle selection and select All Alerts to view alerts. The all alerts view shows graph for alerts over a specific period. The list of alerts are displayed beneath the graph.

Hover mouse pointer over a graph bar to view the alerts information specific to the time interval in which the bar is generated. Clicking the bar updates the graph and the list beneath the graph for the interval in which the bar is generated.

lb alerts
Figure: Load Balancer Alerts

Note: You can also set a time interval in the all alerts view to display alerts over a specific period of time.

Click > for any alert entry to display its details in JSON format.

Requests

Click the Requests tab to load the view for the trend of sampled HTTP requests.

The requests are displayed in a graphical trend as well as in a list for the default or specific time interval. Click > for any listed request to display detailed information in JSON format.

Note: Use the Hide Chart option on the top right side of the page to hide the graph and display only list entries.

lb reqs
Figure: Load Balancer Requests View

You can apply filters to display the trend for specific HTTP codes. For example, de-select all and select only 2xx to display the requests for HTTP code 2XX.

You can apply filters to the display using the Forensics option at the right of the graph to show the Forensics side panel. Select a filter and click Apply to filter the display accordingly. You can also include more filter options by editing the default options and adding more from the displayed list.

Note: You can apply filters using the Add Filter option located above the requests graph.

Errors

Click the Errors tab to load the view for the trend of client or origin server errors.

The errors are displayed in a graphical trend for the default or specific time interval. You can adjust the time interval either using the drop-down selector located on the top right side of the page or using the controls beneath the graph.


Explore Security Monitoring

Switch to security monitoring view. This can be done in any of the following 2 ways:

  • Click on the General Monitoring drop-down option in the general monitoring view and select Security Monitoring option.
  • Go to Virtual Hosts -> HTTP Load Balancers page. Hover mouse pointer on your load balancer and select the enabled Security Monitoring option.

The following entries describe the various tab views available for security monitoring:

Dashboard

The dashboard tab displayed by default and offers a snapshot view for entire security monitoring information. Dashboards shows various security details such as security events, WAF events, service policy events, attack events, DDoS, Bots, etc.

The following list provides overview on the dashboard and the various sections it offers:

  • Security Events section shows the snapshot of security events. This displays a list of security events in the last 12 hours by default. Click on any event to switch to security events tab and view full information for that event. For example, click on l7_policy_sec_event to load the related information in Security Events page.
  • Top WAF Rules Hit section shows the WAF rule hit statistics.

image5
Figure: Security Monitoring View

  • Security Events by Location section shows the security events arranged in a map view. Use the Security Events drop-down filter to change the section to show DDoS events. Click on the location with hits to switch to the Security Events or DDoS tab accordingly. You can also click the Security Events or DDoS links to switch to their respective tabs.
  • Recent WAF and Policy Events section shows the list of recent WAF events. Click on the URL or Type or Method for an event entry to switch to the security events page and view detailed information about that event.
  • Policy Rules Hit shows the rule hits for the service policy applied to the load balancer.
  • Top Attack Types, Bots Request, and Bots by Category shows information on attack types and bots.
  • DDoS Security Events shows the events flagged as DDoS events and suspicious clients.
  • Malicious Users shows the list of users flagged as malicious users and information such as their user id, suspicion score, etc.
Malicious Users

Click on the Malicious Users tab to view trend and list of events flagged as malicious user activity.

The malicious users view shows graph representing trend of malicious user activity over a default or selected time period. The view shows graphical trend as well as a tiled list of events (to the left side of the graph) flagged as malicious user events. Upon selection of an event entry from the left-side list, the graph on the right-side reflects that user's trend.

lb mal users
Malicious User Monitoring and Mitigation

The view also displays summary of timeline beneath the graph where suspicion scores for a user over the selected time period is displayed. The scores are categorized in terms of the severity of the events.

Malicious user mitigation is supported using the Block User option located on the top of the page. You can also use the Add to Allow List option to remove the user from malicious user list.

Security Events

Click on the Security Events tab to load security events view. This shows various types of security events over default time period of 12 hours in a graph view. This page also displays filters various types of events that are represented in different colored dots. Beneath the graph, the security event page displays the events in a list arranged into different tabs namely Security Events, Malicious User Events, and DDoS Events. The Security Events tab is displayed by default.

Perform the following to inspect various security events.

  • Click on a dot to select or deselect those events from being displayed.
  • Click on the Add Filter option and select a key-value pair to apply specific filters. You can select available key-value pairs. You can also choose a custom entry. Type a key, click Select Custom Key, type a value, and click Select Custom Value to apply a custom filter.
  • Click on the time interval drop-down list on the top right side of the page to select another time interval or specify a custom interval.
Security Events

Click Security Events tab beneath the graph chart to view the list of security events. The following list provides information on each field of the list.

  • Time: Time the event was created.
  • Country,City: Location of the event.
  • Src IP: Source of the suspicious request
  • Method: Method type of the HTTP request (GET, POST, DELETE, PUT, etc.)
  • Rsp Code: The HTTP response code (200, 403,404, etc.)
  • Rules Hit: Number of rules hit.
  • Authority: Load balancer domain.
  • Request Path: String of characters that unambiguously identifies a particular resource (for example /testcase-6/test.com)

image3
Figure: Security Events Page

Note: You can click > on any entry to display information of that event in fully expanded view. Select JSON tab to obtain the information in JSON format.

Click ... for an entry on the list of security events and select one option as per the following guidelines.

  • Select Create Exception Rule to create exception for that event so that it is not flagged as a security event. This will open the load balancer edit form with a WAF rule to exclude this event. Enter name, select values for Exclude WAF Rules field, click Apply, and click Save and Exit in the load balancer configuration page to apply the exception rule.
  • Select Add to Blocked Clients to add this client to blocked clients. This will open the load balancer edit form with a rule to block the specific client. Click Apply and click Save and Exit in the load balancer configuration page to apply the blocking rule.
  • Select Add to Trusted Clients to add this client to trusted clients. This will open the load balancer edit form with a rule to whitelist the specific client. Click Apply and click Save and Exit in the load balancer configuration page to apply the trusted clients rule.
Malicious User Events

Click on the Malicious User Events tab beneath the graph to inspect list of events flagged as malicious user events.

Click > to view detailed information of an event in JSON format.

DDoS Events

Click on the DDoS Events tab beneath the graph to inspect list of events flagged as DDoS events.

Click > to view detailed information of an event in JSON format.

Note: Click Refresh on the top right side of the page to refresh the information displayed on the page. Click Hide Chart to hide the graph and show only events list.

DDoS

Click on the DDoS tab to monitor the DDoS information for this load balancer. The DDoS view shows the information on DDoS events occurring over default or select time interval. The view shows a geographical map showing the event location. Hover the mouse pointer over the location to view attack score and location information.

lb ddos map
DDoS Events Map

Click on the Timeline option at the bottom of the map to display trend for request rate, error rate, and throughput. This indicates which metric is associated with the DDoS event.

Click on the DDoS Events drop down located at the top of the page to display the trend of DDoS events with list of events beneath the graph. Hover the mouse pointer over a graph bar to view the start time, end time, and number of events represented by that bar.

lb ddos events
DDoS Events Graph

Click > for an entry to view detailed information in JSON format. The information includes IP addresses of users flagged as suspicious users.

Click Analytics on the top of the page to view DDoS statistics for top IP addresses, regions, ASNs, and TLS fingerprints. Click on the downward arrow for any field such as IP address to view the member list of that field. You can select members of any field and click Apply to filter the display for the selected members.

lb ddos analytics
DDoS Analytics View

After selecting members, click Add Rule to create and apply a DDoS mitigation rule to the load balancer. This opens the load balancer configuration rule with the selected members.

lb ddos rules
DDoS Mitigation Rules

For example, you can select an IP address and click Apply to filter the display for that IP address. Then clicking Add Rule opens the load balancer edit view with the DDoS rule populated with IP address as the source and blocking that IP address as the mitigation action. Click Apply and Save and Exit to apply the rule to load balancer.

Note: Click View Rules to open the load balancer DDoS rules configuration page and view the existing rules.

Requests

Click the Requests tab to load the view for the trend of sampled HTTP requests.

The requests are displayed in a graphical trend as well as in a list for the default or specific time interval. Click > for any listed request to display detailed information in JSON format.

Note: Use the Hide Chart option on the top right side of the page to hide the graph and display only list entries.

You can apply filters to display the trend for specific HTTP codes. For example, de-select all and select only 2xx to display the requests for HTTP code 2XX.

You can apply filters to the display using the Forensics option at the right of the graph to show the Forensics side panel. Select a filter and click Apply to filter the display accordingly. You can also include more filter options by editing the default options and adding more from the displayed list.

Note: You can apply filters using the Add Filter option located above the requests graph.

API Endpoints

Click on the API Endpoints to view the discovered API endpoints and information on the various metrics associated with each API endpoint.

Real Time

Click on the Real Time to view the real time information for the following metrics:

  • Requests
  • Errors
  • WAF security events
  • Drop rate

lb realtime
Figure: Real Time Events View

The information is shown as a graph for a default period of 12 hours and refreshed for every 2 minutes. The graph shows combined information for all the metrics and you can hide or include any of the metrics by selecting the metric options beneath the Overview heading.


Concepts


API References