# Splunk Alert Notification

This section explains how to receive ThousandEyes alert notifications in Splunk using custom webhooks.

## Set Up Splunk

1. Log in to **Splunk**.
2. Create an **HTTP Event Collector (HEC) token** in your Splunk instance:
   * [Configure HTTP Event Collector on Splunk Cloud Platform](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Cloud_Platform)
   * [Configure HTTP Event Collector on Splunk Enterprise](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Enterprise)
3. Identify the target endpoint based on your Splunk deployment:
   * **Splunk Cloud Platform:**

     ```
     https://http-inputs-<host>.splunkcloud.com:443/services/collector/event
     ```
   * **Splunk Enterprise:**

     ```
     https://<host>:8088/services/collector/event
     ```

## Set Up ThousandEyes

1. In the ThousandEyes platform, go to **Manage > Integrations** and click **+ New Integration** in the top right.

   ![New Integration button on Integrations screen](/files/vo8mRotAH91wTn8pVTFx)
2. In the **Add New Integration** side panel that opens, select **Custom Webhook**.

   ![Custom Webhook button](/files/jU4zJ6D7dmGnzlaE6AgT)
3. Fill in the fields as follows:

   * **Name:** Enter a descriptive name for this integration.
   * **URL:** Use the target endpoint identified in the [previous step](#set-up-splunk).
   * **Preset Configuration:** Select **Splunk**.
   * **Headers:** Add the following key-value pairs:
     * `Content-Type`: `application/json`
     * `Authorization`: `Splunk <HEC Token>`

   ![Completed fields in the integration](/files/XA9YVpKqiMS3DLeTKDFd)
4. Click **Test** to verify the webhook:
   * If successful, you will see **"Testing completed successfully!"**.
   * If the test fails:
     * Click **Save** to save the integration.
     * Reopen the integration and test it again.
     * Verify that the HEC target and token are correct.
     * If the test still fails, contact ThousandEyes support.
5. Click **Save**.
6. Apply the webhook to existing alert rules using the **Manage Alert Rules** panel.

## Receive Alerts in Splunk

1. Log in to **Splunk**.
2. When an alert is triggered, search for the event using the following query: `index="*" eventType="THOUSANDEYES_ALERT_NOTIFICATION"`

**Sample Output:**

```
{
  "eventId": "0-0",
  "eventType": "THOUSANDEYES_ALERT_NOTIFICATION",
  "id": "0",
  "type": "2",
  "accountId": "0",
  "orgId": "0",
  "testId": "0",
  "thousandeyes_test_id": "0",
  "test_description": "Sample Description",
  "test_type": "HTTP",
  "itsiDrilldownURI": "https://app.thousandeyes.com/network-app-synthetics/views/?testId=0",
  "severity_id": "1",
  "vendor_severity": "INFO",
  "app": "THOUSANDEYES",
  "src": "Sample Target",
  "signature": "Sample Rule",
  "alert_type": "Http",
  "alert": {
    "id": "0",
    "type": "Http",
    "severity": "INFO",
    "test": {
      "name": "Sample Test"
    },
    "targets": [
      "Sample Target"
    ],
    "rule": {
      "id": "0",
      "name": "Sample Rule",
      "expression": "Response Time ≥ 111 ms",
      "notes": "Sample Notes"
    },
    "triggered": 1738322223247,
    "cleared": 1738322223247,
    "details": [
      {
        "metricsAtStart": "Response Time: 888 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 1"
        }
      },
      {
        "metricsAtStart": "null",
        "metricsAtEnd": "Response Time: 999 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 2"
        }
      }
    ]
  }
}
```

### Integration with Splunk IT Service Intelligence

Some fields, such as `itsiDrilldownURI` and `app`, are required for [Splunk IT Service Intelligence (ITSI) application](https://splunkbase.splunk.com/app/1841/). Splunk ITSI will receive ThousandEyes alerts, analyze them, and aggregate them with other events.

For more information on configuring webhooks, see [Custom Webhooks](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhooks).


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhook-examples/splunk-alert-notifs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.