# Classic Webhooks for Alert Notifications In this article, we'll describe the configuration of alerts to use classic webhooks (henceforth, simply "webhooks"). Additionally, for those unfamiliar with webhooks or who will benefit from constructing a quick example, we provide information to set up a test web service using [Heroku](https://www.heroku.com/). For reference, we've also included example POST body data at the end of this article. {% hint style="warning" %} Webhooks (including custom webhooks) are not available in the ThousandEyes for Government instance for alert notifications. {% endhint %} ## Configuring Webhook Notifications To configure notifications to a webhook server, you'll need to define a webhook in [**Manage > Alert Rules**](https://app.thousandeyes.com/manage/alerts/rules/?tab=CloudAndEnterpriseAgent). Expand an alert rule and select the **Notifications** tab. Under the **Webhooks** section, click the **Configure Webhooks** link. If you already have configured webhooks, click the **Edit webhooks** link: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-0a6eae7db9a89aa4ceba14c42747926bae23e1bf%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-1.png?alt=media\&token=b64c0731-7155-4b34-8b6f-1787b2a5cce7) Click **Add New Webhook**, then complete the **New Webhook** dialog: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-63b3cdcb05991379c6e24b67392561dfe56611f9%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-2.png?alt=media\&token=0432463c-a61f-4571-b9c1-71f965b837a4) Enter the **Name** of your Webhook, and the target **URL** (note that HTTP redirects are not accepted, so be sure to provide the final URL). Configure required authentication if required, and click **Test** to check your configuration: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-5f642219efdc2a9f3f8c31825452def4c154797b%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-3.png?alt=media\&token=864ec8de-4e98-4e8a-95a6-4a51ba7d038e) **NOTE: Encoded HTML URLs are currently not supported for webhook configuration. Enter the URL in decoded form.** If the test succeeds, a **Webhook test completed successfully** message displays: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-a3657214491753fcde78c48988a9eed128f1322e%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-4.png?alt=media\&token=a52f1066-e8d1-4601-b029-ba10cafb73a0) Once you have tested your webhook server successfully, click **Add New Webhook**, and select your webhook from the list of configured webhooks. You can add any number of custom, static key-value pairs to the webhook payload. These custom payloads can be used on any Cloud or Enterprise Agent alert type and are specific to the selected alert rule. To add these elements, navigate to an individual alert rule, select a webhook, and click **Add Custom Payload** (1). A key-value pair (2) field appears that allows you to enter any static key-value pair. You can add or remove these payloads using the **-** and **+** buttons (3). ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-88763cfc3e3a82042a25d36aa5850f92d91f89b7%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-12_version-2.png?alt=media\&token=22251fa9-a714-4af8-9daa-a711ab177094) Additionally, this change adds two new fields to the webhook payload: testLabels, which includes test label name and ID, as well as testTargetsDescription, which includes the description of the test target as defined in the test configuration. ## Webhook Authentication The webhook interface supports the use of a token, OAuth, and basic HTTP authentication. This is an optional configuration step and can be configured for a webhook service if the target server requires authentication. ### Basic A **Username** and corresponding **Password** are necessary to configure Basic HTTP Authentication: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-70872f00e0b6f3a060fc80f3a81b6d79fa9a7704%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-5.png?alt=media\&token=6418f2f8-60eb-41ae-b6b9-546c3f9568fe) ### Token The Token Auth Type uses a **Bearer Token** to authenticate with the webhook server: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-4b0b85aecdd7a192b0bc16917b94aab45f15f6db%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-6.png?alt=media\&token=d84c7e81-81e2-4173-87ac-e6a1c747788e) ### OAuth #### Webhook Server-Side Configuration Create a new OAuth endpoint for ThousandEyes Alerts with as **redirect/call back** URL. Grab the newly created **Client ID** and Authorization Endpoint URL (**Auth URL**) for the webhook server to configure ThousandEyes-side authentication. **NOTICE:** Ensure that the token has a sufficient lifespan to avoid manual regeneration on timeout. #### ThousandEyes-Side Configuration Configure the webhook **Name** and **URL** as explained in [Configuring Webhook Notifications](#configuring_webhook_notifications). Click **OAuth** in **Auth Type.** Fill out the **Auth URL** and **Client ID** according to the *Webhook Server-Side Configuration* section above. Click **Get Token**: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-f978f302d092f0e2573fe87436b101a0582b2182%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-8.png?alt=media\&token=b4e94840-524a-4379-b73e-9e8485d360dd) A new tab opens, requesting consent to authorize ThousandEyes. Below is a snippet from ServiceNow application that we are using in this guide as an example: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-3081c6bb2f4c1d2f791f0691d94a29a33d5d2ac9%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-9.png?alt=media\&token=f9117b85-c777-4d4b-a4f3-5ac1a1ca1453) Once the authorization is complete, a **Token retrieved successfully** message is displayed, as shown in the following figure: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-98123107a6fbf059454fb7127a094dfcba3fa556%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-10.png?alt=media\&token=d17b29d1-20a4-46d1-ade3-324fe9a8b8de) Now click **Test** to verify the operability of your new webhook: ![](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-b3b8a6de220f0fd1104d3b0d11ad0befc85c5f0e%2Fproduct-documentation_alerts_using-webhooks-server-sample-code-included-11.png?alt=media\&token=fcdbbe32-4e22-4437-ad91-3fb6e7d149a9) ## Setting Up a Webhook Server If you visit [Heroku](https://www.heroku.com/), you can deploy a free web application. If you're new to Heroku, it's worth running through their instructional tutorials to understand how the deployment process works. Follow these instructions to deploy a simple version of your web service. 1. Install the Heroku command line interface (CLI). Go to , and follow the instructions to download the Heroku CLI for your operating system. 2. Clone your git repository. ```` $ git clone https://github.com/my-organization/my-webhook-server.git Then, move to the directory created. ```text $ cd my-webhook-server ```` 3. Create a Heroku remote in your git repository. ``` $ heroku create ``` 4. Push your content to the Heroku remote. This will build the application. ``` $ git push heroku master ``` 5. Open the site to confirm the target location. This opens a web browser in the context of your Heroku application. ``` $ heroku open ``` 6. Open a console to review the logs of the target server. In this window, you'll see POST requests hitting your server. ``` $ heroku logs --tail 2014-11-11T20:33:42.767724+00:00 heroku[api]: Deploy e5d1a07 by dave@thousandeyes.com 2014-11-11T20:33:42.767753+00:00 heroku[api]: Release v15 created by dave@thousandeyes.com 2014-11-11T20:33:43.724332+00:00 heroku[web.1]: State changed from up to starting 2014-11-11T20:33:46.814245+00:00 heroku[web.1]: Starting process with command `node webhook-server.js` 2014-11-11T20:33:50.439787+00:00 heroku[web.1]: State changed from starting to up 2014-11-11T20:33:50.347118+00:00 app[web.1]: Webhook Server started... port: 30490 2014-11-11T20:33:59.586785+00:00 heroku[router]: at=info method=GET path="/" host=sleepy-mesa-2108.herokuapp.com request_id=228650f8-d810-477e-858b-b61235766260 fwd="54.90.223.146" dyno=web.1 connect=2ms service=10ms status=200 bytes=266 ``` ## Webhook POST Content Webhooks use events to signify activity. Our system is configured today to support the following types of events: * **WEBHOOK\_TEST** - sent when a user clicks the Test Webhook link in ThousandEyes webhook configuration. * **ALERT\_NOTIFICATION\_TRIGGER** - sent when an alert is triggered * **ALERT\_NOTIFICATION\_CLEAR** - sent when an alert is cleared, the alert rule is deleted, or the alert rule is removed from a test while in an active alerting status. Note: All `eventIds` are unique. To correlate two events (for example, an alert trigger and clear) reference the `alertId`, which is unique per generated alert. ### Agent Webhook Notifications Notifications sent by agents use the same `eventType` as regular notifications. See the payload examples for agent notifications in the Sample Webhook Event Payloads section below. * **ALERT\_NOTIFICATION\_TRIGGER** - sent when an agent notification triggers - may be related to clock offset, version number, or agent online status * **ALERT\_NOTIFICATION\_CLEAR** - sent when the agent notification clears Note: All `eventIds` are unique. To correlate two events (for example, an alert trigger and clear) reference the `alertId`, which is unique per generated alert. ## Sample Webhook Event Payloads Each event type shown below contains a sample JSON payload, which is posted to the target server. The Alert notification data corresponds to alerting data from the ThousandEyes API, wrapped in an event wrapper. See the [API Reference](https://developer.cisco.com/docs/thousandeyes/v7/integrations-api-overview/) for more information on data returned via the API. #### WEBHOOK\_TEST ``` { "eventType":"WEBHOOK_TEST", "eventId":"1415733887743" } ``` #### ALERT\_NOTIFICATION\_TRIGGER ``` { "eventId": "446192921-119034123", "alert": { "severity": "Major", "dateStartZoned": "2022-05-02 17:20:00 UTC", "apiLinks": [ { "rel": "related", "href": "https://api.thousandeyes.com/v4/tests/2854771" }, { "rel": "data", "href": "https://api.thousandeyes.com/v4/web/http-server/2854771" } ], "testLabels": [], "active": 1, "ruleExpression": "Response Time ≥ 0 ms", "type": "HTTP Server", "ruleAid": 11626, "agents": [ { "dateStart": "2022-05-02 17:20:00", "active": 1, "metricsAtStart": "Response Time: 510 ms", "metricsAtEnd": "", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=3", "agentId": 3, "agentName": "Singapore" }, { "dateStart": "2022-05-02 17:20:00", "active": 1, "metricsAtStart": "Response Time: 81 ms", "metricsAtEnd": "", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=4", "agentId": 4, "agentName": "Tokyo, Japan" }, { "dateStart": "2022-05-02 17:20:00", "active": 1, "metricsAtStart": "Response Time: 286 ms", "metricsAtEnd": "", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=27674", "agentId": 27674, "agentName": "Austin, TX" } ], "testTargetsDescription": [ "http://google.com" ], "violationCount": 3, "dateStart": "2022-05-02 17:20:00", "ruleName": "AI - AG - Http - Webhook Sample", "testId": 2854771, "alertId": 119034123, "ruleId": 4735623, "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123", "testName": "AI - AG - Http - Webhook Sample" }, "eventType": "ALERT_NOTIFICATION_TRIGGER" } ``` #### ALERT\_NOTIFICATION\_CLEAR ``` { "eventId": "446194841-119034123", "alert": { "severity": "Major", "dateStartZoned": "2022-05-02 17:20:00 UTC", "apiLinks": [ { "rel": "related", "href": "https://api.thousandeyes.com/v4/tests/2854771" }, { "rel": "data", "href": "https://api.thousandeyes.com/v4/web/http-server/2854771" } ], "testLabels": [], "active": 0, "ruleExpression": "Response Time ≥ 0 ms", "dateEnd": "2022-05-02 17:23:42", "type": "HTTP Server", "ruleAid": 11626, "agents": [ { "dateStart": "2022-05-02 17:20:00", "dateEnd": "2022-05-02 17:23:42", "active": 0, "metricsAtStart": "Response Time: 510 ms", "metricsAtEnd": "N/A (alerts disabled)", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=3", "agentId": 3, "agentName": "Singapore" }, { "dateStart": "2022-05-02 17:20:00", "dateEnd": "2022-05-02 17:23:42", "active": 0, "metricsAtStart": "Response Time: 81 ms", "metricsAtEnd": "N/A (alerts disabled)", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=4", "agentId": 4, "agentName": "Tokyo, Japan" }, { "dateStart": "2022-05-02 17:21:00", "dateEnd": "2022-05-02 17:23:42", "active": 0, "metricsAtStart": "Response Time: 94 ms", "metricsAtEnd": "N/A (alerts disabled)", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=14410", "agentId": 14410, "agentName": "San Francisco, CA" }, { "dateStart": "2022-05-02 17:20:00", "dateEnd": "2022-05-02 17:23:42", "active": 0, "metricsAtStart": "Response Time: 286 ms", "metricsAtEnd": "N/A (alerts disabled)", "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123&agentId=27674", "agentId": 27674, "agentName": "Austin, TX" } ], "testTargetsDescription": [ "http://google.com" ], "violationCount": 4, "dateStart": "2022-05-02 17:20:00", "dateEndZoned": "2022-05-02 17:23:42 UTC", "ruleName": "AI - AG - Http - Webhook Sample", "testId": 2854771, "alertId": 119034123, "ruleId": 4735623, "permalink": "https://app.thousandeyes.com/alerts/list/?__a=11626&alertId=119034123", "testName": "AI - AG - Http - Webhook Sample" }, "eventType": "ALERT_NOTIFICATION_CLEAR" } ``` ### Agent Notifications #### ALERT\_NOTIFICATION\_TRIGGER ``` { "eventId": "5009976-107130750", "alert": { "dateStartZoned": "2022-02-09 14:00:46 CST", "agentId": 388926, "hostname": "agent01", "dateStart": "2022-02-09 14:00:46", "ipAddress": "192.168.1.143", "ruleName": "Default Agent Offline Notification", "agentName": "Agent-01", "alertId": 107130750, "ruleExpression": "Last Contact ≥ 5 minutes ago", "ruleId": 292726, "type": "Agent", "ruleAid": 239471 }, "eventType": "ALERT_NOTIFICATION_TRIGGER", "agentAlert": { "dateStartZoned": "2022-02-09 14:00:46 CST", "agentId": 388926, "hostname": "agent01", "dateStart": "2022-02-09 14:00:46", "ipAddress": "192.168.1.143", "ruleName": "Default Agent Offline Notification", "agentName": "Agent-01", "alertId": 107130750, "ruleExpression": "Last Contact ≥ 5 minutes ago", "ruleId": 292726, "type": "Agent", "ruleAid": 239471 } } ``` #### ALERT\_NOTIFICATION\_CLEAR ``` { "eventId": "5009986-107130750", "alert": { "dateStartZoned": "2022-02-09 14:00:46 CST", "agentId": 388926, "ipAddress": "192.168.1.143", "agentName": "Agent-01", "ruleExpression": "Last Contact ≥ 5 minutes ago", "dateEnd": "2022-02-09 14:05:46", "type": "Agent", "ruleAid": 239471, "hostname": "agent01", "dateStart": "2022-02-09 14:00:46", "dateEndZoned": "2022-02-09 14:05:46 CST", "ruleName": "Default Agent Offline Notification", "alertId": 107130750, "ruleId": 292726 }, "eventType": "ALERT_NOTIFICATION_CLEAR", "agentAlert": { "dateStartZoned": "2022-02-09 14:00:46 CST", "agentId": 388926, "ipAddress": "192.168.1.143", "agentName": "Agent-01", "ruleExpression": "Last Contact ≥ 5 minutes ago", "dateEnd": "2022-02-09 14:05:46", "type": "Agent", "ruleAid": 239471, "hostname": "agent01", "dateStart": "2022-02-09 14:00:46", "dateEndZoned": "2022-02-09 14:05:46 CST", "ruleName": "Default Agent Offline Notification", "alertId": 107130750, "ruleId": 292726 } } ``` ## Allowlist Requirements If your web server only allows requests from allowlisted senders, you will need to allow requests from ThousandEyes web servers. You can find the list of allowed IP addresses at [Firewall Exception Requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhooks#firewall-exception-requirements). ## Related Articles * For a guide on creating your own alert rules, see [Creating and Editing Alert Rules](https://docs.thousandeyes.com/product-documentation/alerts/creating-and-editing-alert-rules).