Bot Defense

Objective

This guide provides instructions on how to enable Bot Defense and apply it to your web applications using VoltConsole. For more information on Bot Defense, see About Bot Defense.


Prerequisites

Note: If you do not have an account, see Create a Volterra Account.

  • An Organization plan.

Note: If you do not have an Organization plan, upgrade your plan.


Enabling Bot Defense

You can enable the Bot Defense add-on service in one of two ways:

Option 1

Step 1: On the VoltConsole Home page, click Billing.

VoltConsoleHomeBilling
Figure: Click Billing

Step 2: Scroll to the right until you see the Organization plan.

OrgPlan
Figure: Scroll to the right until you see Organization Plan.

Step 3: Under the Organization plan, click the Bot Defense link.

billingBotDefense
Figure: Click Bot Defense

Option 2

Step 1: In VoltConsole, navigate to the Load Balancers page.

VoltConsoleHomeNav
Figure: Click Load Balancers

Step 2: Click the … in the Actions column next to your load balancer and select Manage Configuration.

ManageConfigurationNav
Figure: Click '...' > Manage Configuration

Step 3: Click Edit Configuration.

EditConfigurationNav
Figure: Click Edit Configuration

Step 4: In the left-side navigation menu, click Security Configuration.

EditConfigurationSecurity
Figure: Click Security Configuration

Step 5: In the Bot Defense Config section, click the drop-down menu and select Specify Bot Defense Configuration.

SpecifyBotDefenseConfigNav
Figure: Select Specify Bot Defense Configuration

Note: For load balancers with types “HTTPS with Custom Certificate” and “HTTPS with Automatic Certificate” set the Path normalize field to Enable path normalization. This is the default setting for new load balancer objects.

pathNormalize
Figure: Path normalize

Note: Make sure the "Disable" checkbox is unchecked under Service Policy.

servicePolicy
Figure: Uncheck the Disable checkbox under Service Policy.


Adding Endpoints for Bot Defense Protection

In the Bot Defense Config section, follow these steps:

Step 1: Under Bot Defense Regional Endpoint, select the region where the endpoint resides.

RegionalEndpoint
Figure: Select Regional Endpoint

Step 2: In the Bot Defense Policy section, click Configure.

BotDefensePolicyConfigure
Figure: Click Configure

Step 3: In the Protected App Endpoints section, click Configure.

ProtectedAppEndpointsConfigure
Figure: Click Configure

Step 4: Click the Add Item button to add an application endpoint.

Note: You can create multiple endpoints.

AddEndpoint1
Figure: Click Add Item

Step 5: Enter the following for each endpoint:
  • Name: The name of the message. Must follow DNS-1035 format.
  • Description: A human-readable description of the endpoint.
  • HTTP Methods: Which HTTP methods are monitored on this endpoint.

    Note: Commonly used methods include POST/PUT/GET(XHR). GET requests are protected only if they are sent by XHTTPRequest from the page that has Bot Defense JavaScript injected, not from direct navigation via address bar or link. The ANY method should be used carefully and only when intended.

  • Protocol: Which protocols are protected.
  • Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
  • Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
  • Bot Traffic Mitigation: Specify what action to take when a bot is detected.

    • Block: The endpoint returns a status code and message. You can select the code and edit the message here.
    • Redirect: The endpoint forwards the browser to a URI, specified here.
    • Flag: Creates a log record only.

Injecting the Bot Defense JS into Your Web Pages

After you have added the domains in which to apply Bot Defense protection, you need to inject the Bot Defense JavaScript (JS) into the web pages.

To inject the Bot Defense JS, follow these steps:

In the JavaScript Insertion section:

Step 1: Under JavaScript Download Path, enter the path where the HTTP load balancer can find the JavaScript to serve to the client browser.

ProtectedAppEndpointsJavascriptPath
Figure: JavaScript Path

Step 2: Under JavaScript Insertion Settings, specify if the HTTP load balancer should insert JavaScript into all pages, or if some pages should be excluded.
If you select Insert JavaScript in All Pages:
  • Choose where the JavaScript will be inserted:

    • After <head> tag
    • After </title> tag
    • Before <script> tag

InsertJsInAllPagesJsLocation
Figure: JavaScript location

If you select Insert JavaScript in All Pages with the Exceptions:
  1. Choose where the JavaScript will be inserted:

    • After <head> tag
    • After </title> tab
    • Before <script> tag
  2. Click the Add Item button to add an excluded page. The JavaScript Insertion Exclusion Rule page appears.
    InsertJsInAllPageswithExceptionsExcludePages
    Figure: Exclude pages
  3. In the JavaScript Insertion Exclusion Rule page, enter the following for each excluded page:

    • Name: The name of the message. Must follow DNS-1035 format.
    • Description: A human-readable description of the endpoint.
    • Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
    • Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
  4. Click the Add Item button.

If you select Custom JavaScript Insertion Rules:
  1. Under JavaScript Insertions, click Configure. The JavaScript Insertions page appears.
    customJIRC
    Figure: Configure
    javascriptInsertionsAddItem
    Figure: Add Item
  2. Click the Add Item button and enter the following for each endpoint:

    • Name: The name of the message. Must follow DNS-1035 format.
    • Description: A human-readable description of the endpoint.
    • Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
    • Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
    • JavaScript location:

      • After <head> tag
      • After </title> tab
      • Before <script> tag
  3. When you are finished adding endpoints, click the Back button. The JavaScript Insertions screen appears.
  4. Click the Back button.
    customJIREP
    Figure: Add Item
  5. Under Exclude Paths, click the Add Item button and enter the following for any paths you want to exclude from inserting JavaScript:
  6. Name: The name of the message. Must follow DNS-1035 format.
  7. Description: A human-readable description of the endpoint.
  8. Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
  9. Path: Specify protected paths here. Enter a prefix, exact path, or regex value.

If you select Disable JavaScript Insertion, no further action is necessary. JavaScript will not be inserted to any page.

Viewing Traffic Data with the Bot Defense Dashboard and Bot Traffic Overview Dashboard

When Bot Defense is enabled and configured, the HTTP load balancer's security screen has two additional tabs: Bot Defense and Bot Traffic Overview. You can use these tabs to access their respective dashboards to view data about your traffic.

To access the Bot Defense and Bot Traffic Overview dashboards:

Step 1: In VoltConsole, navigate to Web App & API Protection > Apps & APIs > Security.

Note: You can also access the Bot Defense Dashboards via the HTTP Load Balancer's “Security Monitoring” page.

VoltConsoleHome2
Figure: Click Web App & API Protection

HttpLoadBalancers1
Figure: Click Security

Step 2: Click a load balancer in the Load Balancers list. The Security dashboard appears.

clickLoadBalancer
Figure: Select a load balancer

securityDashboard
Figure: Security dashboard

One tile in the Security dashboard shows the Bot Defense top three automation types.

Step 3: In the Security dashboard, click the Bot Defense or Bot Traffic Overview tab.

BotTrafficOverview
Figure: Bot traffic overview

The Bot Defense Dashboard

The Bot Defense dashboard provides a snapshot of human and malicious bot activity in your web traffic for a specified time period.

The dashboard presents key information like which bots are making the most malicious requests, which endpoints are attacked most, and which automation types are being used most. You can customize the time period, filter results, and make other adjustments using the dashboard features described below.

BotDefenseDashboard
Figure: Bot Defense dashboard

Time Window selector: Choose a time range to analyze from this drop-down menu.

Hide Filter/Show Filter button: Click this to toggle the Add filter link and any active filters. You can filter results by IP address, AS Organization, and User Agent.

Traffic Types: Shows the total number of transactions in the selected time window, how many were malicious bots, and how many were humans.

Top Automation Types: Shows the most common automation type for the selected time window.

Traffic Overview: This graph shows the transactions per minute of human and malicious bot traffic for the selected time period. You can hover over the graph to see specific values.

Top Malicious Bots: This table shows the five bots that have made the most malicious requests in the selected time period. The table includes the source IP, ASN, user agent, country where the bot is based, and number of malicious requests in the selected time period. You can view each bot's Source IP, AS Organization, and User Agent.

Top Endpoints Attacked: This table shows the five endpoints that are being attacked most frequently by malicious bots. The table includes the host name, endpoint path, and number of malicious requests in the selected time period.

The Bot Traffic Overview Dashboard

The Bot Traffic Overview dashboard provides detailed insight into traffic on the HTTP load balancer.

BotTrafficOverview
Figure: Bot traffic overview

In addition to the chart showing transactions per minute for a specified time window, you can also view details about every HTTP request sent through Bot Defense. Each HTTP request includes the following information:

  • Time
  • Country
  • IP Address
  • ASN
  • AS Organization
  • User Agent
  • Host
  • Path
  • Method
  • Inference

You can rearrange and sort the columns by clicking them. You can show/hide columns by clicking the gear icon in the upper right corner of the list.


References