Splunk Alert Notification

This section explains how to receive ThousandEyes alert notifications in Splunk using custom webhooks.

Set Up Splunk

1. Log in to Splunk.

2. Create an HTTP Event Collector (HEC) token in your Splunk instance:

   • Configure HTTP Event Collector on Splunk Cloud Platform

   • Configure HTTP Event Collector on Splunk Enterprise

3. Identify the target endpoint based on your Splunk deployment:

   • Splunk Cloud Platform:

     https://http-inputs-<host>.splunkcloud.com:443/services/collector/event

   • Splunk Enterprise:

     https://<host>:8088/services/collector/event

Set Up ThousandEyes

1. In the ThousandEyes platform, go to Manage > Integrations and click + New Integration in the top right.

   

   New Integration button on Integrations screen
2. In the Add New Integration side panel that opens, select Custom Webhook.

   

   Custom Webhook button
3. Fill in the fields as follows:

   • Name: Enter a descriptive name for this integration.

   • URL: Use the target endpoint identified in the previous step.

   • Preset Configuration: Select Splunk.

   • Headers: Add the following key-value pairs:

     • Content-Type: application/json

     • Authorization: Splunk <HEC Token>

   

   Completed fields in the integration
4. Click Test to verify the webhook:

   • If successful, you will see "Testing completed successfully!".

   • If the test fails:

     • Click Save to save the integration.

     • Reopen the integration and test it again.

     • Verify that the HEC target and token are correct.

     • If the test still fails, contact ThousandEyes support.

5. Click Save.

6. Apply the webhook to existing alert rules using the Manage Alert Rules panel.

Receive Alerts in Splunk

1. Log in to Splunk.

2. When an alert is triggered, search for the event using the following query: index="*" eventType="THOUSANDEYES_ALERT_NOTIFICATION"

Splunk Alert Event

Sample Output:

{
  "eventId": "0-0",
  "eventType": "THOUSANDEYES_ALERT_NOTIFICATION",
  "id": "0",
  "type": "2",
  "accountId": "0",
  "orgId": "0",
  "testId": "0",
  "thousandeyes_test_id": "0",
  "test_description": "Sample Description",
  "test_type": "HTTP",
  "itsiDrilldownURI": "https://app.thousandeyes.com/view/cloud-and-enterprise-agents/?testId=0",
  "severity_id": "1",
  "vendor_severity": "INFO",
  "app": "THOUSANDEYES",
  "src": "Sample Target",
  "signature": "Sample Rule",
  "alert_type": "Http",
  "alert": {
    "id": "0",
    "type": "Http",
    "severity": "INFO",
    "test": {
      "name": "Sample Test"
    },
    "targets": [
      "Sample Target"
    ],
    "rule": {
      "id": "0",
      "name": "Sample Rule",
      "expression": "Response Time ≥ 111 ms",
      "notes": "Sample Notes"
    },
    "triggered": 1738322223247,
    "cleared": 1738322223247,
    "details": [
      {
        "metricsAtStart": "Response Time: 888 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 1"
        }
      },
      {
        "metricsAtStart": "null",
        "metricsAtEnd": "Response Time: 999 ms",
        "source": {
          "id": "0",
          "name": "Sample Agent 2"
        }
      }
    ]
  }
}

Integration with Splunk IT Service Intelligence

Some fields, such as itsiDrilldownURI and app, are required for Splunk IT Service Intelligence (ITSI) application. Splunk ITSI will receive ThousandEyes alerts, analyze them, and aggregate them with other events.

For more information on configuring webhooks, see Custom Webhooks.