> For the complete documentation index, see [llms.txt](https://docs.thousandeyes.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhook-examples/splunk-alert-notifs.md).

# Splunk Alert Notification

This section explains how to receive ThousandEyes alerts in Splunk using a custom webhook connector and operation.

## Set Up Splunk

1. Log in to **Splunk**.
2. Create an **HTTP Event Collector (HEC) token** in your Splunk instance:
   * [Configure HTTP Event Collector on Splunk Cloud Platform](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Cloud_Platform)
   * [Configure HTTP Event Collector on Splunk Enterprise](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Enterprise)
3. Identify the target endpoint based on your Splunk deployment:
   * **Splunk Cloud Platform:**

     ```
     https://http-inputs-<host>.splunkcloud.com:443/services/collector/event
     ```
   * **Splunk Enterprise:**

     ```
     https://<host>:8088/services/collector/event
     ```

## Set Up ThousandEyes

### Create a Connector and Operation

1. In the ThousandEyes platform, go to **Manage > Integrations**.
2. Go to **Integrations 2.0 > Integration Templates**.

   ![Custom Webhook template in the Integration Templates tab](/files/dNQc7jQmj63nV3vv3NKx)
3. Select **Custom Webhook**.
4. Configure the connector:
   * **Name**: Enter a descriptive name for the Splunk connector.
   * **Target**: Use the target endpoint identified in [Set Up Splunk](#set-up-splunk).
   * **Auth Type**: Select the **Custom** authentication type.
   * **Custom Headers**: Add an `Authorization` header with the value `Splunk <HEC Token>`.
5. Click **Save & Assign Operation**.
6. Configure the custom webhook operation:
   * **Operation Name**: Enter a descriptive name for the operation.
   * **Preset Configurations**: Select **Splunk**.
   * **Custom Headers**: Add a `Content-Type` header with the value `application/json`.
   * **Body**: Review the Splunk preset payload.
7. Click **Test** to verify the operation:
   * If the test succeeds, ThousandEyes displays a confirmation message.
   * If the test fails, verify that the HEC target, HEC token, headers, and payload are correct.
8. Click **Save Integration**.

![Splunk custom webhook connector and operation configuration](/files/6Nm9vy6CEhUvDD48eHda)

### Attach Alert Rules to the Operation

1. In the ThousandEyes platform, go to **Manage > Integrations**.
2. Go to **Integrations 2.0 > Operations**.
3. Find the Splunk custom webhook operation.
4. Click the actions menu (⋮) at the end of the row, then select **Manage Alert Rules**.
5. In the **Manage Alert Rules** panel, select the alert rules you want to send to Splunk.
6. Click **Save**.

![Manage Alert Rules panel with the Splunk custom webhook operation selected](/files/2wGfPzPXUDpGbQwtIYy3)

## Receive Alerts in Splunk

1. Log in to **Splunk**.
2. When an alert is triggered, search for the event using the following query: `index="*" eventType="THOUSANDEYES_ALERT_NOTIFICATION"`

**Sample Output:**

```json
{
  "eventId": "0-0",
  "eventType": "THOUSANDEYES_ALERT_NOTIFICATION",
  "id": "0",
  "type": "2",
  "accountId": "0",
  "orgId": "0",
  "testId": "0",
  "thousandeyes_test_id": "0",
  "test_description": "Sample Description",
  "test_type": "HTTP",
  "itsiDrilldownURI": "https://app.thousandeyes.com/network-app-synthetics/views/?testId=0",
  "severity_id": "1",
  "vendor_severity": "INFO",
  "app": "THOUSANDEYES",
  "src": "Sample Target",
  "signature": "Sample Rule",
  "alert_type": "Http",
  "alert": {
    "id": "0",
    "type": "Http",
    "severity": "INFO",
    "test": {
      "name": "Sample Test"
    },
    "targets": [
      "Sample Target"
    ],
    "rule": {
      "id": "0",
      "name": "Sample Rule",
      "expression": "Response Time ≥ 111 ms",
      "notes": "Sample Notes"
    },
    "triggered": 1738322223247,
    "cleared": 1738322223247,
    "details": [
      {
        "metricsAtStart": "Response Time: 888 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 1"
        }
      },
      {
        "metricsAtStart": "null",
        "metricsAtEnd": "Response Time: 999 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 2"
        }
      }
    ]
  }
}
```

![Splunk search results showing a ThousandEyes alert event](/files/OkqpS4ekWkcLfEVfEInk)

### Integration with Splunk IT Service Intelligence

Some fields, such as `itsiDrilldownURI` and `app`, are required for [Splunk IT Service Intelligence (ITSI) application](https://splunkbase.splunk.com/app/1841/). Splunk ITSI will receive ThousandEyes alerts, analyze them, and aggregate them with other events.

For more information on configuring webhooks, see [Custom Webhooks](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhooks).