Custom Webhooks

Custom webhook operations let you send ThousandEyes alert data to any external system using an HTTP POST request. You choose the destination URL, define request headers, and format the payload to meet your needs. Use custom webhook operations when you need to integrate ThousandEyes with tools or workflows that aren't supported by built-in integrations, like Slack or ServiceNow.

Custom webhook operations give you more control over how alerts are handled. For example, you can:

• Send alerts to an internal monitoring or ticketing system

• Trigger automated responses, such as updating routing rules or scaling resources

• Post custom messages to a chat system with enriched data

• Format alerts to match your existing logging or incident workflows

Use custom webhook operations to:

• Deliver alerts to a system ThousandEyes doesn’t support by default

• Automate a response when certain alerts fire

• Customize how alert data is structured or transmitted

Custom webhook operations are a flexible way to fit ThousandEyes into your existing observability and incident response workflows.

For more in-depth documentation for particular services, see Custom Webhook Examples.

Creating a New Custom Webhook Operation

Creating a New Custom Webook Operation Using the Integration Template

Use the Integration Templates tab to create a new custom webhook when you don't have an existing connector already configured. The integration template creates the connector and operator in one flow.

If you already have a connector configured, you can create the custom webhook operation manually to associate it with your existing connector. See the section Creating a New Custom Webook Operation Manually.

To create a custom webhook connector and operation using integration templates:

1. In the ThousandEyes app, navigate to Manage > Integrations. Click the Integrations 2.0 tab. Click on the Integrations Templates tab.

2. Select the Custom Webhook template.

3. Follow the instructions to set up your connector. For more information about setting up a connector, see.

4. Once you have finished filling out the fields to configure your connector, click the Save & Assign Operation button at the bottom right.

5. Fill out the fields to configure your custom webhook operation. See the section Configuring Fields for a Custom Webhook Operation.

6. Click the Save Integration button at the bottom right to complete the setup.

Custom Webhooks created with the integrations template appear in the list under v1, but they are only manageable under v2.

Creating a New Custom Webook Operation Manually

To create a new custom webhook operation when you already have an existing connector:

1. In the ThousandEyes app, navigate to Manage > Integrations. Click the Integrations 2.0 tab. Click on the Operations tab.

2. Select the + New Operation button at the top right of the screen.

3. From the side modal that appears, select the Custom Webhook button.

To configure the fields for your new custom webhook, see the section Configuring Fields for a Custom Webhook Operation.

Configuring Fields for a Custom Webhook Operation

1. In the Add New Operation panel (or Setup your Operations for the integration template), fill out the fields for your new custom webhook:

Field	Description
Operation Name	A name for your custom webhook operation. We recommend you use a human-readable, meaningful string. If you create multiple operations, it’s a good practice to use a consistent naming convention.
URL	The target URL of the server associated with your webhook. Note: HTTP redirects are not accepted. Make sure you provide the final URL in this field.
Preset Configurations	These optional preset configurations provide a starting place from which you can create your custom webhook. If you use a preset configuration, it populates the fields below with sample content that you can adapt. Note: These preset configurations can serve as a guide, but you should verify that the resulting webhook’s structure and variables meet your requirements. You can use any of the following preset configurations for your webhook: Generic AppDynamics Cisco Webex Microsoft Teams Slack Splunk [additional preset configurations to come] Caution: When you select a preset configuration, any settings you have already made in the fields below are deleted.
Auth Type	Set the authentication method for your webhook: None Basic Token OAuth For basic and token authentication, see Webhook Authentication. For OAuth, see Using OAuth Authentication for Your Custom Webhook.
Headers	The Headers field contains key/value pairs that make up the webhook’s HTTP header to a server and that function as metadata for the webhook. The key/value pairs you enter here can be static (a set key and its set value) or dynamic (a set key and a variable as its value) To add additional key/value pairs to the header of your webhook, click + Key-Value Pair and enter the key and value. To remove a key/value pair, click the - symbol.
URL Query Parameters	These parameters are appended to a URL to define content or actions, especially when integrating into a third-party system. For guidance on defining query parameters, see Introduction to Handlebars. You can type directly into the text box to add URL query parameters to your webhook. To see an expanded view of the contents of the field, click the arrows button.
Body	Define the body of your webhook to suit your needs. Note that the body Must be valid JSON. Can include static or dynamic key/value pairs. Variables must match Handlebars syntax. You can type directly into the text box to customize the body of your webhook. To see an expanded view of the contents of the field, click the arrows button. For guidance on the editor for the Body field, see Working in the Webhook Editor. For convenience, the variables and some guidance on using them are available in Webhook Variables.

1. Click Save.

   Your new webhook is listed under the Operations tab.

To attach alert rules to your new webhook, see the section Attaching Alert Rules to Your Webhook.

Working in the Custom Webhook Operation Editor

Handlebars Templating Language

The ThousandEyes webhook templates use the Handlebars templating language. In fact, in a custom webhook, the Body field is a Handlebars template.

When you write your own webhook, the syntax in the Body field must conform to the Handlebars syntax rules. For detailed information on the Handlebars language, see the Handlebars guide.

Hinting and Intellisense

When you edit the Headers, URL Query Parameters, and Body fields, the webhook editor UI automatically suggests variables that match the text you have written or that was present in the template.

For example, if you type “a” into the editor, it suggests “alerts” as a possible completion. Once a given variable is completed, the editor suggests sub-variables available for it. For example, if the variable completed is “alert.”, the editor suggests all possible sub-variables, such as “id” “rule," and “firstSeen." You can prompt the editor's suggestions manually by typing Ctrl+space.

Custom Webhook Operations Connection Status

To see the connection status for your custom webhook operation, navigate to Manage > Integrations. Click the Integrations 2.0 tab. Click on the Operations tab. The Status column can show the following statuses:

Status	Definition
Pending	The integration request has been received and is awaiting further action.
Connected	The integration has been successfully established and is currently active.
Unverified	The integration exists but its connector status has not been confirmed or validated.

• When an integration is first set up, its status is Pending.

• Once communication is established, the status updates to Connected.

• If the integration's state cannot be confirmed, it is marked as Unverified.

Attaching Alert Rules to Your Webhook

Once you have created your custom webhook operation, you can attach alert rules to it so that ThousandEyes alert notifications can flow into the system you’ve integrated with. To learn more about alert rules, see Creating and Editing Alert Rules.

1. In the ThousandEyes platform, go to Manage > Integrations and select the Operations tab.

2. In the list of operations, find the custom webhook operation you want to attach alert rules to.

3. Click the vertical dots at the end of the row, and select Manage Alert Rules from the dropdown.

4. In the Manage Alert Rules panel, select the alert rule or rules you want to associate with this custom webhook operation.

   If you have a large number of alert rules, use the Search box to filter on a relevant string. You can also sort alert rules by name and by alert type.

5. Click Save.

You can attach any custom webhook operation to any Cloud and Enterprise Agent alert rule. However, some variables used in a webhook may not be applicable to some alert rules. In these cases, the inapplicable variable is passed as a null value to the system you have integrated with.

Failed Alert Notifications

If alert notifications fail to be sent via custom webhook operations, we continuously retry sending the notification for up to two hours after the first failure. We retry up to nine times at the following intervals (in minutes) after the first failure:

5 + 10 + 15 + 15 + 15 + 15 + 15 + 15 + 15

These intervals map to the following minutes after the first failure:

5 - 15 - 30 - 45 - 60 - 75 - 90 - 105 - 120

If after this time the notification still isn't sent, the notification is dropped.

Editing a Custom Webhook Operation

To edit a custom webhook operation, see Integrations.

Deleting a Custom Webhook Operation

To delete a custom webhook operation, see Integrations.

Firewall Exception Requirements

If your web server only allows requests from allowlisted senders, you will need to allow requests from ThousandEyes web servers, which come from the following IP addresses, which are region-specific:

US1 Region

13.56.245.241
52.9.183.148
18.232.232.61
35.168.54.3
107.22.84.44
3.220.243.232
3.218.27.195
3.221.227.188

US2 Region

3.141.159.49
3.17.98.26
3.134.227.22
3.18.18.42
3.13.54.169
3.138.52.162
52.27.149.70
52.32.30.54
52.89.210.182
44.227.213.61
35.155.240.202
35.81.172.197

EU1 Region

18.157.124.37
3.70.3.30
18.156.17.238
18.158.163.183
35.158.19.241
3.127.8.252
46.51.169.205
54.75.173.76
54.217.22.60
34.243.129.225
54.216.15.243
108.128.60.238

For more information about regions, see Multi-Region Cloud Support.

Troubleshooting Your Custom Webhook

If you’re encountering issues with your custom webhook, consider the following points:

• A common way a webhook might fail despite passing testing is when conditional Handlebars helpers are used. For example, the template might contain a clause like the following:

  "timeTriggered": {{#eq type 2 }} {{ alert.firstSeen.epochSecond }} {{/eq}}

  If the underlying alert’s state is triggered, as it is in the sample we use when we test a webhook, this syntax renders correctly. But if the alert’s state is cleared, this syntax fails due to being invalid JSON: the “timeTriggered” key in the resulting object will have no value.

• Watch for illegal characters in the underlying alert data, particularly for custom headers. We apply URL escaping to query parameters, but not to the webhook's header or payload.

• Do not use # for custom helpers like formatDate or formatExpression.