Custom Webhooks

This guide walks through how to create a custom webook for integrating the ThousandEyes platform with third-party services. For more in-depth documentation for particular services, see Custom Webhook Examples.

Creating a New Webhook

To create a new webhook, do the following:

1. In the ThousandEyes platform, go to Integrations.

2. Click + New Integration.

3. In the Add New Custom Integration panel, configure the fields for your new custom webhook:

Field	Description
Name	A name for your custom webhook. We recommend you use a human-readable, meaningful string. If you create multiple webhooks, it’s a good practice to use a consistent naming convention.
URL	The target URL of the server associated with your webhook. Note: HTTP redirects are not accepted. Make sure you provide the final URL in this field.
Preset Configurations	These optional preset configurations provide a starting place from which you can create your custom webhook. If you use a preset configuration, it populates the fields below with sample content that you can adapt. Note: These preset configurations can serve as a guide, but you should verify that the resulting webhook’s structure and variables meet your requirements. You can use any of the following preset configurations for your webhook: Generic AppDynamics Cisco Webex Microsoft Teams Slack Splunk [additional preset configurations to come] Caution: When you select a preset configuration, any settings you have already made in the fields below are deleted.
Auth Type	Set the authentication method for your webhook: None Basic Token OAuth For basic and token authentication, see Webhook Authentication. For OAuth, see Using OAuth Authentication for Your Custom Webhook.
Header	The Header field contains key/value pairs that make up the webhook’s HTTP header to a server and that function as metadata for the webhook. The key/value pairs you enter here can be static (a set key and its set value) or dynamic (a set key and a variable as its value) To add additional key/value pairs to the header of your webhook, click + Key-Value Pair and enter the key and value. To remove a key/value pair, click the - symbol.
URL Query Parameter	These parameters are appended to a URL to define content or actions, especially when integrating into a third-party system. For guidance on defining query parameters, see Introduction to Handlebars. You can type directly into the text box to add URL query parameters to your webhook. To see an expanded view of the contents of the field, click the arrows button.
Body	Define the body of your webhook to suit your needs. Note that the body Must be valid JSON. Can include static or dynamic key/value pairs. Variables must match Handlebars syntax. You can type directly into the text box to customize the body of your webhook. To see an expanded view of the contents of the field, click the arrows button. For guidance on the editor for the Body field, see Working in the Webhook Editor. For the most up-to-date list of the variables you can use in your custom webhook, use the ThousandEyes API. For convenience, the variables and some guidance on using them are available in Webhook Variables.

5. [Optional] Click Test to test your new custom webhook.

   For detailed information, see Testing a Webhook.

6. Click Save.

   Your new webhook is listed on the Integrations screen.

7. [Optional] Attach alert rules to your new webhook.

   For detailed instructions, see Attaching Alert Rules to Your Webhook.

Working in the Webhook Editor

Handlebars Templating Language

The ThousandEyes webhook templates use the Handlebars templating language. In fact, in a custom webhook, the Body field is a Handlebars template.

When you write your own webhook, the syntax in the Body field must conform to the Handlebars syntax rules. For detailed information on the Handlebars language, see the Handlebars guide.

Hinting and Intellisense

When you edit the Headers, URL Query Parameters, and Body fields, the webhook editor UI automatically suggests variables that match the text you have written or that was present in the template.

For example, if you type “a” into the editor, it suggests “alerts” as a possible completion. Once a given variable is completed, the editor suggests sub-variables available for it. For example, if the variable completed is “alert.”, the editor suggests all possible sub-variables, such as “id” “rule," and “firstSeen." You can prompt the editor's suggestions manually by typing Ctrl+space.

Testing a Webhook

It’s a good practice to test your webhooks before deploying them. You can test a webhook while you are creating it, or after you have saved it:

1. In the ThousandEyes platform, go to Integrations.

2. In the list of your integrations, find the webhook you want to test.

   The Test Status column shows whether the webhook has been tested.

3. Click the row for the webhook you want to test (or click its context menu and select Edit).

4. In the Edit Custom Integration panel, click Test.

   If your test fails, use the error message as displayed in this panel to troubleshoot your configuration.

How Webhooks Are Tested

When you click Test, the following elements of your webhook’s configuration are tested:

• The URL you have provided

• The authentication you have provided

• The format of the payload

  The ThousandEyes platform sends a dummy HTTP alert trigger to test the format of the payload as defined in your webhook. It evaluates the server’s response to determine whether the server can accept the payload.

The Test function in this panel is not a foolproof verification of your webhook. The best way to test a webhook is with a real alert.

Test Status

On the Integrations screen, your webhooks are listed, showing their test status. The Test Status field can have the following values:

Untested	This webhook has not been tested through the ThousandEyes platform.
Failure	This webhook has been tested through the ThousandEyes platform, but has failed this test. If your webhook has a Failure status, it can still be attached to alert rules and might still be dispatched. The Failure status typically means either that the server did not respond, or that the payload itself is malformed. However, in some cases these failures do not prevent the successful dispatch of a custom webhook.
Success	This webhook has been tested through the ThousandEyes platform, and has successfully dispatched a test payload. However, note that in some cases a successful test does not guarantee the successful dispatch of a custom webhook attached to an alert rule.

Attaching Alert Rules to Your Webhook

Once you have created your custom webhook and tested it, you’ll attach alert rules to it so that ThousandEyes alert notifications can flow into the system you’ve integrated with.

1. In the ThousandEyes platform, go to Integrations.

2. In the list of your integrations, find the webhook you want to attach alert rules to.

3. From its context menu, select Manage Alert Rules.

4. In the Manage Alert Rules panel, select the alert rule or rules that should be associated with this webhook.

   If you have a large number of alert rules, use the Search box to filter on a relevant string. You can also sort alert rules by name and by alert type.

5. Click Save.

You can attach any custom webhook to any Cloud and Enterprise Agent alert rule. However, some variables used in a webhook may not be applicable to some alert rules. In these cases, the inapplicable variable is passed as a null value to the system you have integrated with.

Failed Alert Notifications

If alert notifications fail to be sent via custom webhooks, we continuously retry sending the notification for up to two hours after the first failure. We retry up to nine times at the following intervals (in minutes) after the first failure:

5 + 10 + 15 + 15 + 15 + 15 + 15 + 15 + 15

These intervals map to the following minutes after the first failure:

5 - 15 - 30 - 45 - 60 - 75 - 90 - 105 - 120

If after this time the notification still isn't sent, the notification is dropped.

Editing a Webhook

1. In the ThousandEyes platform, go to Integrations.

2. In the list of integrations, find the webhook you want to edit.

3. Click the row for the webhook you want to edit (or click its context menu and select Edit).

4. In the Edit Custom Integration panel, make your changes.

   For a detailed description of the fields in this panel, see Creating a New Webhook.

5. [Optional] Click Test to test your updated custom webhook.

6. Click Save.

Duplicating an Existing Webhook

1. In the ThousandEyes platform, go to Integrations.

2. In the list of integrations, find the webhook you want to duplicate.

3. From its context menu, select Duplicate.

   The system creates a new webhook that has

   • The name of the previous webhook, with the string copy appended to it.

   • The same configuration as the previous webhook.

   • No alert rules attached to it.

4. [Optional] Click Test to test your updated custom webhook.

5. Click Save.

6. [Optional] Attach alert rules to your new webhook.

   For detailed instructions, see Attaching Alert Rules to Your Webhook.

Deleting a Webhook

1. In the ThousandEyes platform, go to Integrations.

2. In the list of integrations, select the webhook you want to delete.

3. From the context menu, select Delete.

Note: Deleting a webhook removes it from any alert rules that were attached to it. Deleted webhooks can be manually re-created, but cannot be restored.

Firewall Exception Requirements

If your web server only allows requests from allowlisted senders, you will need to allow requests from ThousandEyes web servers, which come from the following IP addresses, which are region-specific:

US1 Region

13.56.245.241
52.9.183.148
18.232.232.61
35.168.54.3
107.22.84.44
3.220.243.232
3.218.27.195
3.221.227.188

US2 Region

3.141.159.49
3.17.98.26
3.134.227.22
3.18.18.42
3.13.54.169
3.138.52.162
52.27.149.70
52.32.30.54
52.89.210.182
44.227.213.61
35.155.240.202
35.81.172.197

EU1 Region

18.157.124.37
3.70.3.30
18.156.17.238
18.158.163.183
35.158.19.241
3.127.8.252
46.51.169.205
54.75.173.76
54.217.22.60
34.243.129.225
54.216.15.243
108.128.60.238

For more information about regions, see Multi-Region Cloud Support.

Troubleshooting Your Custom Webhook

If you’re encountering issues with your custom webhook, consider the following points:

• A common way a webhook might fail despite passing testing is when conditional Handlebars helpers are used. For example, the template might contain a clause like the following:

  "timeTriggered": {{#eq type 2 }} {{ alert.firstSeen.epochSecond }} {{/eq}}

  If the underlying alert’s state is triggered, as it is in the sample we use when we test a webhook, this syntax renders correctly. But if the alert’s state is cleared, this syntax fails due to being invalid JSON: the “timeTriggered” key in the resulting object will have no value.

• Watch for illegal characters in the underlying alert data, particularly for custom headers. We apply URL escaping to query parameters, but not to the webhook's header or payload.

• Do not use # for custom helpers like formatDate or formatExpression.