Splunk Alert Notification

This section explains how to receive ThousandEyes alerts in Splunk using a custom webhook connector and operation.

Set Up Splunk

1. Log in to Splunk.

2. Create an HTTP Event Collector (HEC) token in your Splunk instance:

   • Configure HTTP Event Collector on Splunk Cloud Platform

   • Configure HTTP Event Collector on Splunk Enterprise

3. Identify the target endpoint based on your Splunk deployment:

   • Splunk Cloud Platform:

     https://http-inputs-<host>.splunkcloud.com:443/services/collector/event

   • Splunk Enterprise:

     https://<host>:8088/services/collector/event

Set Up ThousandEyes

Create a Connector and Operation

1. In the ThousandEyes platform, go to Manage > Integrations.

2. Go to Integrations 2.0 > Integration Templates.

   

   Custom Webhook template in the Integration Templates tab
3. Select Custom Webhook.

4. Configure the connector:

   • Name: Enter a descriptive name for the Splunk connector.

   • Target: Use the target endpoint identified in Set Up Splunk.

   • Auth Type: Select the Custom authentication type.

   • Custom Headers: Add an Authorization header with the value Splunk <HEC Token>.

5. Click Save & Assign Operation.

6. Configure the custom webhook operation:

   • Operation Name: Enter a descriptive name for the operation.

   • Preset Configurations: Select Splunk.

   • Custom Headers: Add a Content-Type header with the value application/json.

   • Body: Review the Splunk preset payload.

7. Click Test to verify the operation:

   • If the test succeeds, ThousandEyes displays a confirmation message.

   • If the test fails, verify that the HEC target, HEC token, headers, and payload are correct.

8. Click Save Integration.

Splunk custom webhook connector and operation configuration

Attach Alert Rules to the Operation

1. In the ThousandEyes platform, go to Manage > Integrations.

2. Go to Integrations 2.0 > Operations.

3. Find the Splunk custom webhook operation.

4. Click the actions menu (⋮) at the end of the row, then select Manage Alert Rules.

5. In the Manage Alert Rules panel, select the alert rules you want to send to Splunk.

6. Click Save.

Manage Alert Rules panel with the Splunk custom webhook operation selected

Receive Alerts in Splunk

1. Log in to Splunk.

2. When an alert is triggered, search for the event using the following query: index="*" eventType="THOUSANDEYES_ALERT_NOTIFICATION"

Sample Output:

{
  "eventId": "0-0",
  "eventType": "THOUSANDEYES_ALERT_NOTIFICATION",
  "id": "0",
  "type": "2",
  "accountId": "0",
  "orgId": "0",
  "testId": "0",
  "thousandeyes_test_id": "0",
  "test_description": "Sample Description",
  "test_type": "HTTP",
  "itsiDrilldownURI": "https://app.thousandeyes.com/network-app-synthetics/views/?testId=0",
  "severity_id": "1",
  "vendor_severity": "INFO",
  "app": "THOUSANDEYES",
  "src": "Sample Target",
  "signature": "Sample Rule",
  "alert_type": "Http",
  "alert": {
    "id": "0",
    "type": "Http",
    "severity": "INFO",
    "test": {
      "name": "Sample Test"
    },
    "targets": [
      "Sample Target"
    ],
    "rule": {
      "id": "0",
      "name": "Sample Rule",
      "expression": "Response Time ≥ 111 ms",
      "notes": "Sample Notes"
    },
    "triggered": 1738322223247,
    "cleared": 1738322223247,
    "details": [
      {
        "metricsAtStart": "Response Time: 888 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 1"
        }
      },
      {
        "metricsAtStart": "null",
        "metricsAtEnd": "Response Time: 999 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 2"
        }
      }
    ]
  }
}

Splunk search results showing a ThousandEyes alert event

Integration with Splunk IT Service Intelligence

Some fields, such as itsiDrilldownURI and app, are required for Splunk IT Service Intelligence (ITSI) application. Splunk ITSI will receive ThousandEyes alerts, analyze them, and aggregate them with other events.

For more information on configuring webhooks, see Custom Webhooks.