Splunk Cloud Platform and Splunk Enterprise

Splunk Cloud Platform and Splunk Enterprise are data analytics platforms that collect, index, and analyze machine data from any source to deliver operational intelligence. They support ingesting OpenTelemetry (OTel) data as events or metrics for real-time monitoring, search, and visualization. This guide shows how to stream ThousandEyes metrics, traces, and logs to Splunk Cloud Platform or Splunk Enterprise using OpenTelemetry.

For more information about the HTTP Event Collector (HEC), see Use the HTTP Event Collector to get data in in the Splunk documentation. For the signals ThousandEyes supports, see Supported Signals.

Prerequisites

• You have created a ThousandEyes test.

• You have configured your Splunk instance for ThousandEyes:

  1. Enable the HTTP Event Collector (HEC). For more information, see Configure HTTP Event Collector on Splunk Cloud Platform or Configure HTTP Event Collector on Splunk Enterprise.

  2. (Optional) Create an index to control whether data is stored as an event or a metric. For more information, see Splunk: Set up multiple indexes.

  

  Splunk index settings used to route ThousandEyes OpenTelemetry data

  1. Create an HEC token and, optionally, associate it with your index for routing.

  2. Identify the target endpoints for your Splunk deployment:

  • Splunk Cloud Platform:

    • events: https://http-inputs-<host>[.<subdomain>].splunkcloud.com:443/services/collector/event

    • metrics: https://http-inputs-<host>[.<subdomain>].splunkcloud.com:443/services/collector

    • Splunk Cloud Platform URLs can include an environment-specific subdomain, such as .stg. For example: https://http-inputs-<host>.stg.splunkcloud.com:443/services/collector/event

  • Splunk Enterprise:

    • events: https://<host>:8088/services/collector/event

    • metrics: https://<host>:8088/services/collector

• You have a Splunk HEC connector in Integrations 2.0 before you add operations for metrics, traces, or logs. If you still need to create it, see Creating a Splunk HEC Connector in ThousandEyes.

Creating a Splunk HEC Connector in ThousandEyes

ThousandEyes Integrations 2.0 uses a Splunk HEC connector for every OpenTelemetry stream to Splunk (metrics, traces, and logs). Create this connector once per Splunk target, then add a separate operation for each signal you want to send.

1. In the ThousandEyes platform, go to Manage > Integrations > Integrations 2.0.

2. Click + New Connector.

3. Click Splunk Cloud Platform HEC or Splunk Enterprise HEC depending on your target.

   

   Select Splunk HEC Connector
4. Enter a Name for your connector.

5. Enter the Target HEC URL, including port when required.

   For example, https://http-inputs-<host>.splunkcloud.com:443/services/collector/event for Splunk Cloud Platform or https://<splunk_hostname>/services/collector/event for Splunk Enterprise. When you create a stream, the target URL must satisfy the Stream endpoint URL requirements.

6. Enter your Splunk HEC Token.

   

   Creating a ThousandEyes for OpenTelemetry to Splunk Enterprise connector
7. Click Save & Assign Operation to save the connector. This connector is now visible in the list view (Connectors tab).

ThousandEyes for OpenTelemetry currently does not support Splunk trial accounts due to an issue with TLS self-signed certificates.

Stream OpenTelemetry Metrics to Splunk

Step 1: Create a Metrics Integration

Using the ThousandEyes UI

Use the Splunk HEC connector from Creating a Splunk HEC Connector in ThousandEyes. Then create an operation for metrics:

Creating an Operation for Metrics

1. In ThousandEyes, go to Manage > Integrations > Integrations 2.0.

2. Click + New Operation.

3. Select Splunk Enterprise, Splunk Cloud Platform to open the configuration form and assign it to your Splunk HEC connector.

4. Enter an Operation Name.

5. (Optional) Set Index, Source, and Source Type.

   For more information, see the Splunk documentation for indexes, source, and source type.

6. Set OpenTelemetry Signal to Metric.

7. Set Integration Status to enable or disable the integration as needed.

8. Under Test Data Configuration, select Network & App Synthetics Tests and Endpoint Experience Tests for the test data to send.

9. (Optional) Under Test Data Configuration, select Tags from your account group to include tests with matching tags.

   For more information on creating a test, see General Setup Instructions.

10. (Optional) At the bottom of the configuration form, click Test.

    A test message is sent to the target to verify the connection. A success or failure message is shown.

    

    Test Success Message

    

    Test Failure Message
11. Click Save. The operation appears on the Operations tab.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "testMatch": [
    {
      "id": "{TEST_ID}",
      "domain": "{DOMAIN}"
    }
  ],
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}"
    }
  }
}'

When you create a stream from Splunk Enterprise or Splunk Cloud Platform, ensure the following:

• The type is "splunk-hec".

• The endpointType is "http".

• exporterConfig.splunkHec.token is set to your Splunk HEC token.

• If you associated an index with the HEC token, set exporterConfig.splunkHec.index to that index name.

• The streamEndpointUrl satisfies the Stream endpoint URL requirements.

The signal field can be metric, trace, or log. If omitted, it defaults to metric.

Replace {HOST}, {TOKEN}, {TEST_ID}, and {DOMAIN} with your Splunk Cloud host, HEC token, and test identifiers.

Searching Metrics in Splunk

Search metrics using Splunk’s metrics search language. For example:

• | mcatalog values(metric_name) WHERE index=*

• | mstats avg(_value) WHERE index=* AND metric_name=network.latency span=30s

For more information, see Splunk: Search and monitor metrics.

Search the metrics

ThousandEyes metric values in Splunk

Stream OpenTelemetry Traces to Splunk

Step 1: Create a Traces Integration

Using the ThousandEyes UI

After you complete Creating a Splunk HEC Connector in ThousandEyes, add an operation for traces:

1. In ThousandEyes, go to Manage > Integrations > Integrations 2.0.

2. Click + New Operation.

3. Select Splunk Enterprise, Splunk Cloud Platform.

4. Enter an Operation Name.

5. (Optional) Set Index, Source, and Source Type.

   For more information, see the Splunk documentation for indexes, source, and source type.

6. Set OpenTelemetry Signal to Trace.

7. (Optional) At the bottom of the configuration form, click Test.

   A test message is sent to the target to verify the connection. A success or failure message is shown.

8. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "signal": "trace",
  "testMatch": [
    {
      "id": "{TEST_ID}",
      "domain": "{DOMAIN}"
    }
  ],
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}"
    }
  }
}'

When you create a stream from Splunk Enterprise or Splunk Cloud Platform, ensure the following:

• The type is "splunk-hec".

• The endpointType is "http".

• Set signal to "trace".

• exporterConfig.splunkHec.token is set to your Splunk HEC token.

• If you associated an index with the HEC token, set exporterConfig.splunkHec.index to that index name.

• The streamEndpointUrl satisfies the Stream endpoint URL requirements.

Replace {HOST}, {TOKEN}, {TEST_ID}, and {DOMAIN} with your Splunk Cloud host, HEC token, and test identifiers.

Step 2: Visualize Traces in Splunk

Trace data ingested through HEC appears as events. In Splunk Search & Reporting, search using queries such as source=ThousandEyesOTel or your configured index (for example, index="thousandeyes_otel_events_index").

Stream OpenTelemetry Logs to Splunk

Step 1: Create a Logs Integration

Using the ThousandEyes UI

After you complete Creating a Splunk HEC Connector in ThousandEyes, add an operation for logs:

1. In ThousandEyes, go to Manage > Integrations > Integrations 2.0.

2. Click + New Operation.

3. Select Splunk Enterprise, Splunk Cloud Platform.

4. Enter an Operation Name.

5. (Optional) Set Index, Source, and Source Type.

   For more information, see the Splunk documentation for indexes, source, and source type.

6. Set OpenTelemetry Signal to Log.

7. (Optional) At the bottom of the configuration form, click Test.

   A test message is sent to the target to verify the connection. A success or failure message is shown.

8. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "signal": "log",
  "testMatch": [
    {
      "id": "{TEST_ID}",
      "domain": "{DOMAIN}"
    }
  ],
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}"
    }
  }
}'

When you create a stream from Splunk Enterprise or Splunk Cloud Platform, ensure the following:

• The type is "splunk-hec".

• The endpointType is "http".

• Set signal to "log".

• exporterConfig.splunkHec.token is set to your Splunk HEC token.

• If you associated an index with the HEC token, set exporterConfig.splunkHec.index to that index name.

• The streamEndpointUrl satisfies the Stream endpoint URL requirements.

Replace {HOST}, {TOKEN}, {TEST_ID}, and {DOMAIN} with your Splunk Cloud host, HEC token, and test identifiers.

Step 2: Search logs in Splunk

Search event data using queries such as source=ThousandEyesOTel or your configured index (for example, index="thousandeyes_otel_events_index").

Search in Splunk

ThousandEyes log attributes in Splunk