# Splunk Alert Notification

This section explains how to receive ThousandEyes alert notifications in Splunk using custom webhooks.

## Set Up Splunk

1. Log in to **Splunk**.
2. Create an **HTTP Event Collector (HEC) token** in your Splunk instance:
   * [Configure HTTP Event Collector on Splunk Cloud Platform](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Cloud_Platform)
   * [Configure HTTP Event Collector on Splunk Enterprise](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Enterprise)
3. Identify the target endpoint based on your Splunk deployment:
   * **Splunk Cloud Platform:**

     ```
     https://http-inputs-<host>.splunkcloud.com:443/services/collector/event
     ```
   * **Splunk Enterprise:**

     ```
     https://<host>:8088/services/collector/event
     ```

## Set Up ThousandEyes

1. In the ThousandEyes platform, go to **Manage > Integrations** and click **+ New Integration** in the top right.

   ![New Integration button on Integrations screen](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-caac75b48dc559bc8160f723eb4c7c49a41e20da%2Fms-teams-for-alerts_17.png?alt=media)
2. In the **Add New Integration** side panel that opens, select **Custom Webhook**.

   ![Custom Webhook button](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-567d3499c2f3accc8b8dc8a1bd1603ccf0c4bfa5%2Fms-teams-for-alerts_18.png?alt=media)
3. Fill in the fields as follows:

   * **Name:** Enter a descriptive name for this integration.
   * **URL:** Use the target endpoint identified in the [previous step](#set-up-splunk).
   * **Preset Configuration:** Select **Splunk**.
   * **Headers:** Add the following key-value pairs:
     * `Content-Type`: `application/json`
     * `Authorization`: `Splunk <HEC Token>`

   ![Completed fields in the integration](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-9fd3ea3474610087a8eb6263bdc8c344ce8f7347%2Fspluk-alerts-fields.png?alt=media\&token=4c99dc1d-2091-4aaa-8fdf-410753e304d0)
4. Click **Test** to verify the webhook:
   * If successful, you will see **"Testing completed successfully!"**.
   * If the test fails:
     * Click **Save** to save the integration.
     * Reopen the integration and test it again.
     * Verify that the HEC target and token are correct.
     * If the test still fails, contact ThousandEyes support.
5. Click **Save**.
6. Apply the webhook to existing alert rules using the **Manage Alert Rules** panel.

## Receive Alerts in Splunk

1. Log in to **Splunk**.
2. When an alert is triggered, search for the event using the following query: `index="*" eventType="THOUSANDEYES_ALERT_NOTIFICATION"`

**Sample Output:**

```
{
  "eventId": "0-0",
  "eventType": "THOUSANDEYES_ALERT_NOTIFICATION",
  "id": "0",
  "type": "2",
  "accountId": "0",
  "orgId": "0",
  "testId": "0",
  "thousandeyes_test_id": "0",
  "test_description": "Sample Description",
  "test_type": "HTTP",
  "itsiDrilldownURI": "https://app.thousandeyes.com/network-app-synthetics/views/?testId=0",
  "severity_id": "1",
  "vendor_severity": "INFO",
  "app": "THOUSANDEYES",
  "src": "Sample Target",
  "signature": "Sample Rule",
  "alert_type": "Http",
  "alert": {
    "id": "0",
    "type": "Http",
    "severity": "INFO",
    "test": {
      "name": "Sample Test"
    },
    "targets": [
      "Sample Target"
    ],
    "rule": {
      "id": "0",
      "name": "Sample Rule",
      "expression": "Response Time ≥ 111 ms",
      "notes": "Sample Notes"
    },
    "triggered": 1738322223247,
    "cleared": 1738322223247,
    "details": [
      {
        "metricsAtStart": "Response Time: 888 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 1"
        }
      },
      {
        "metricsAtStart": "null",
        "metricsAtEnd": "Response Time: 999 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 2"
        }
      }
    ]
  }
}
```

### Integration with Splunk IT Service Intelligence

Some fields, such as `itsiDrilldownURI` and `app`, are required for [Splunk IT Service Intelligence (ITSI) application](https://splunkbase.splunk.com/app/1841/). Splunk ITSI will receive ThousandEyes alerts, analyze them, and aggregate them with other events.

For more information on configuring webhooks, see [Custom Webhooks](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhooks).