Alerting with Webhooks

Objective

This document provides instructions on how to configure alert notifications in Volterra Console to route the alert messages to the third-party applications. Volterra supports the following third-party applications for alert notifications:

  • Slack
  • PagerDuty
  • OpsGenie

Using the instructions provided in this document, you can create an alert notification to your third-party application.


Alert Notification Objects

Volterra supports alert notifications using the following objects:

  1. Alert Receiver - The alert receiver specifies configuration for a receiver. The receiver is the third-party application and configuration includes alert settings such as the URL, routing key, etc. An alert receiver may be associated with one or more alert policy objects.
  2. Alert Policy - The alert policy is used to specify a set of routes called as match conditions to match the incoming alerts. An alert policy may be associated with one or more alert receiver objects. Each route has an action associated with to send the alert or not to send. The route evaluation is as per the order of routes and the first match condition. If the match is successful and the action is to send the alert, the notification is sent to all receivers associated with the policy object. In case the action is to not send, then the notification is not sent to the associated receivers.
  3. Alert Policy Set - The alert policy set consists of a list of references to one or more alert policy objects. The alert policies are effective only when an alert policy set object is created in a namespace and are referenced from it. All the alert policies in the policy set are evaluated irrespective of the order and the alert notifications are sent based on the action defined in the first matching route to all the alert receivers attached to the corresponding alert policy.

    Note: The routes in the alert policy are evaluated as per their order but evaluation of policies in the set is irrespective of the order.

The alert policy set is a singleton object per namespace. However, alert policy and alert receivers can be created in individual namespaces and in the shared namespace. This makes it possible to effectively associate multiple alert policies and receivers across namespaces to the alert policy sets.

The following image shows the mapping between the 3 objects across the namespaces:

AlertsReln
Figure: Alert Object Mapping Across Namespaces


Prerequisites


Configuration

The following image shows the work-flow of enabling sending of alert notifications to third-party applications:

AlertsSeq
Figure: Alert Object Configuration Workflow

Configuration Sequence

Creating alert notifications to your third-party application includes the following sequence of activities:

Activity Description
Create Webhook Generate the webhook configuration from your 3rd party application.
Create Alert Receiver Create alert receiver objects and apply the webhook configuration obtained from your third-party application.
Create Alert Policy Create alert policy objects with the routes and associating alert receivers.
Create Alert Policy Set Create alert policy set and associate the alert policies created.

Create Webhook

Posting alerts to a 3rd party application requires you to first generate an incoming webhook for that application. Obtain webhook configuration for alerts from your 3rd party application.

Note: For examples, see Integrations.


Create Alert Receiver

This example shows the alert notification creation. It is required that you to obtain the webhook configuration for your 3rd party application.

Step 1: Log into the Volterra console and change to your namespace. Select Manage from the configuration menu and Alert Config from the options pane. Select Alert Receiver from the options under the Alert Config and click Add alert receiver.

AlertRecNav
Figure: Alert Receiver Creation Form

Step 2: Enter name and select one of the supported options for the third-party applications in the Receiver field.

Note: The webhook configuration is dependent on the receiver type selected and the following guidelines describe the associated options

  • For the receiver Slack, the webhook configuration is the webhook URL and the channel
  • For the receiver PagerDuty, the webhook configuration is the routing key and the URL to send the API request to.
  • For the receiver OpsGenie, the webhook configuration is the API key and the URL to send the API request to.

Step 3: Configure the settings for the Secret field as per the following guidelines:

  • Select Blindfold Secret for the Secret info field.
  • Enter the string obtained from applying Blindfold to the webhook configuration. See Blindfold your Application Secrets for information on how to encrypt your application secrets.
  • Select Base64 for the Secret Encoding field.

ar basic
Figure: Alert Receiver Configuration

Note: You can also select other values for the Secret info field and set the subsequent values as per the following guidelines.

Secret Guidelines
Vault Secret Enter the values for the provider and location.
Clear Secret Enter the value for the URL in the string:///<string> format.
Bootstrap Secret Enter the name of the secret.

It is recommended to encode your secret using the Base64 encryption and set the secret encoding as Base64. However, in case of clear secret, select the encoding as EncodingNone. You can apply Base64 encoding to the webhook URL using the echo -n <URL> | base64 command.

Step 4: Enter the name of the channel or URL as per your application configuration. Click Continue to complete creating the alert receiver.

Note: Depending on your Receiver field selection, appropriate option gets enabled. The option Channel is enabled for Slack selection and URL is enabled for other receiver selection.


Create Alert Policy

Step 1: Log into the Volterra console and change to your namespace. Select Manage from the configuration menu and Alert Config from the options pane. Select Alert Policy from the options under the Alert Config and click Add alert policy.

Step 2: Enter a name for the policy and click Select receiver to open the receiver selection form. Select the receiver you created in the Create Alert Receiver chapter and click Select receiver to add the receiver to the policy.

ap basic
Figure: Alert Policy Receiver Configuration

Step 3: Enter the configuration for the Notification Parameters as per the following guidelines:

  • Enter a value in the [0-9][smhd] format for the Notify Interval for a Alert field. The [smhd] specifies seconds, minutes, hours, and days. An example value is 1h and default is 4h.
  • Enter a value in the [0-9][smhd] format for the Wait to Notify field. The [smhd] specifies seconds, minutes, hours, and days. An example value is 2m and default is 30s.
  • Enter a value in the [0-9][smhd] format for the Notify Interval for a Group field. The [smhd] specifies seconds, minutes, hours, and days. An example value is 5m and default is 1m.

Step 4: Select options for the Select Alerts Match Condition and the Action fields. This example configures the match for any condition using the Any value and send action using the Send value.

ap routes
Figure: Alert Policy Route Configuration

Note: You can also set the match conditions based on the severity, group name, and alert name.

Step 5: Optionally, click Add route to add more policy rules.

Step 6: Click Continue to complete creating the alert policy.


Create Alert Policy Set

Step 1: Log into the Volterra console and change to your namespace. Select Manage from the configuration menu and Alert Config from the options pane. Select Alert Policy Set from the options under the Alert Config and click Add alert policy set.

Step 2: Add a name and click Select policy to open the policy selection form. Select the alert policies you want to add and click Select policy again to add the policies to the policy set.

aps basic
Figure: Alert Policy Set Configuration

Step 3: Click Continue to complete creating the alert policy set.

You can now verify the alert notifications by visiting the 3rd party application.


Concepts


How-to