> For the complete documentation index, see [llms.txt](https://docs.thousandeyes.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms/splunk-cloud-or-enterprise.md).

# Splunk Cloud Platform and Splunk Enterprise

Splunk Cloud Platform and Splunk Enterprise are data analytics platforms that collect, index, and analyze machine data from any source to deliver operational intelligence. They support ingesting OpenTelemetry (OTel) data as events or metrics for real-time monitoring, search, and visualization. This guide shows how to stream ThousandEyes metrics, traces, and logs to Splunk Cloud Platform or Splunk Enterprise using OpenTelemetry.

For more information about the HTTP Event Collector (HEC), see [Use the HTTP Event Collector to get data in](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) in the Splunk documentation. For the signals ThousandEyes supports, see [Supported Signals](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry#supported-signals).

## Prerequisites

* You have [created a ThousandEyes test](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#create-a-test).
* You have configured your Splunk instance for ThousandEyes:

  1. Enable the **HTTP Event Collector (HEC)**. For more information, see [Configure HTTP Event Collector on Splunk Cloud Platform](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Cloud_Platform) or [Configure HTTP Event Collector on Splunk Enterprise](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Enterprise).
  2. (Optional) Create an index to control whether data is stored as an event or a metric. For more information, see [Splunk: Set up multiple indexes](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes).

  ![Splunk index settings used to route ThousandEyes OpenTelemetry data](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-12f3f9cb1f48a5f59e48dce3f4e368ca103e4b13%2Fsplunk-index.png?alt=media)

  3. Create an HEC token and, optionally, associate it with your index for routing.

  4. Identify the target endpoints for your Splunk deployment:

  * Splunk Cloud Platform:
    * `events`: `https://http-inputs-<host>[.<subdomain>].splunkcloud.com:443/services/collector/event`
    * `metrics`: `https://http-inputs-<host>[.<subdomain>].splunkcloud.com:443/services/collector`
    * Splunk Cloud Platform URLs can include an environment-specific subdomain, such as `.stg`. For example: `https://http-inputs-<host>.stg.splunkcloud.com:443/services/collector/event`
  * Splunk Enterprise:
    * `events`: `https://<host>:8088/services/collector/event`
    * `metrics`: `https://<host>:8088/services/collector`
* You have a **Splunk HEC connector** in **Integrations 2.0** before you add operations for metrics, traces, or logs. If you still need to create it, see [Creating a Splunk HEC Connector in ThousandEyes](#creating-a-splunk-hec-connector-in-thousandeyes).

## Creating a Splunk HEC Connector in ThousandEyes

ThousandEyes **Integrations 2.0** uses a **Splunk HEC connector** for every OpenTelemetry stream to Splunk (metrics, traces, and logs). Create this connector once per Splunk target, then add a separate **operation** for each signal you want to send.

1. In the ThousandEyes platform, go to **Manage > Integrations > Integrations 2.0**.
2. Click **+ New Connector**.
3. Click **Splunk Cloud Platform HEC** or **Splunk Enterprise HEC** depending on your target.

   ![Select Splunk HEC Connector](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-8cb5755374a4c57c22f4afd04897b7bf7a2042b3%2Fproduct-documentation_integration-guides_custom-built-integrations_splunk-connector_select.png?alt=media)
4. Enter a **Name** for your connector.
5. Enter the **Target** HEC URL, including port when required.

   For example, `https://http-inputs-<host>.splunkcloud.com:443/services/collector/event` for Splunk Cloud Platform or `https://<splunk_hostname>:8088/services/collector/event` for Splunk Enterprise. When you create a stream, the target URL must satisfy the [Stream endpoint URL requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/url-target-requirements).
6. Enter your Splunk HEC **Token**.

   ![Creating a ThousandEyes for OpenTelemetry to Splunk Enterprise connector](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-ac8e189b6ec203cddb6101bb725d65e8f7765a3c%2Fproduct-documentation_integration-guides_custom-built-integrations_splunk-enterprise-add-connector-view.png?alt=media)
7. Click **Save & Assign Operation** to save the connector. This connector is now visible in the list view (**Connectors** tab).

{% hint style="info" %}
ThousandEyes for OpenTelemetry does not support the Splunk Cloud Platform free trial because ThousandEyes cannot validate the self-signed TLS certificate presented by the trial's HEC endpoint. For more information, see [Splunk Ideas PLECID-I-816](https://ideas.splunk.com/ideas/PLECID-I-816).
{% endhint %}

## Stream OpenTelemetry Metrics to Splunk

### Step 1: Create a Metrics Integration

#### Using the ThousandEyes Platform

Use the Splunk HEC connector from [Creating a Splunk HEC Connector in ThousandEyes](#creating-a-splunk-hec-connector-in-thousandeyes). Then create an operation for metrics:

**Creating an Operation for Metrics**

1. In ThousandEyes, go to **Manage > Integrations > Integrations 2.0**.
2. Click **+ New Operation**.
3. Select **Splunk Enterprise, Splunk Cloud Platform** to open the configuration form and assign it to your Splunk HEC connector.
4. Enter an **Operation Name**.
5. (Optional) Set **Index**, **Source**, and **Source Type**.

   For more information, see the Splunk documentation for [indexes](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Aboutindexesandindexers), [source](https://docs.splunk.com/Splexicon:Source), and [source type](https://docs.splunk.com/Splexicon:Sourcetype).
6. Set **OpenTelemetry Signal** to **Metric**.
7. Set **Integration Status** to enable or disable the integration as needed.
8. Under **Test Data Configuration**, select **Network & App Synthetics Tests** and **Endpoint Experience Tests** for the test data to send.
9. (Optional) Under **Test Data Configuration**, select **Tags** from your account group to include tests with matching tags.

   For more information on creating a test, see [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions).
10. (Optional) At the bottom of the configuration form, click **Test**.

    A test message is sent to the target to verify the connection. A success or failure message is shown.

    ![Test Success Message](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-806bbecfd60cea48f6a2d557063e231d2e48c50d%2Fproduct-documentation_integration-guides_custom-built-integrations_splunk-enterprise-test-success.png?alt=media)

    ![Test Failure Message](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-fc1f9ceeab73a16ddac69253c84922d242fcbb77%2Fproduct-documentation_integration-guides_custom-built-integrations_splunk-enterprise-test-fail.png?alt=media)
11. Click **Save**. The operation appears on the **Operations** tab.

#### Using the ThousandEyes API

Create the HTTP stream programmatically:

```curl
curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "testMatch": [
    {
      "id": "{TEST_ID}",
      "domain": "{DOMAIN}"
    }
  ],
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}"
    }
  }
}'
```

When you create a stream from Splunk Enterprise or Splunk Cloud Platform, ensure the following:

* The `type` is `"splunk-hec"`.
* The `endpointType` is `"http"`.
* `exporterConfig.splunkHec.token` is set to your Splunk HEC token.
* If you associated an index with the HEC token, set `exporterConfig.splunkHec.index` to that index name.
* The `streamEndpointUrl` satisfies the [Stream endpoint URL requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/url-target-requirements).

The `signal` field can be `metric`, `trace`, or `log`. If omitted, it defaults to `metric`.

Replace `{HOST}`, `{TOKEN}`, `{TEST_ID}`, and `{DOMAIN}` with your Splunk Cloud host, HEC token, and test identifiers.

### Searching Metrics in Splunk

Search metrics using Splunk’s metrics search language. For example:

* `| mcatalog values(metric_name) WHERE index=*`
* `| mstats avg(_value) WHERE index=* AND metric_name=network.latency span=30s`

For more information, see [Splunk: Search and monitor metrics](https://docs.splunk.com/Documentation/Splunk/9.3.0/Metrics/Search).

![Search the metrics](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-1020c97e3f74e00f87d845e3111186b1741cff50%2Fproduct-documentation_integration-guides_custom-built-integrations_opentelemetry-integrations-splunk-metrics-search-names.png?alt=media\&token=15eec055-8a1e-487a-8b42-8f30abf87bf6)

![ThousandEyes metric values in Splunk](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-3ef3a68a2fccd6f9ee327cf3b1c4838288affcc8%2Fproduct-documentation_integration-guides_custom-built-integrations_opentelemetry-integrations-splunk-metrics-search-values.png?alt=media\&token=fdf05618-ac3d-4880-89bf-b227c5354b1a)

## Stream OpenTelemetry Traces to Splunk

ThousandEyes exports two trace sources with data model v2:

| Trace source           | Test data                                                                                                                    | What to inspect                                                                       |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| HAR application traces | HTTP Archive (HAR) data from page load, web transaction, and API tests.                                                      | HTTP requests, response codes, and request durations.                                 |
| Network Path traces    | Network topology from agent-to-server and agent-to-agent tests, and other supported tests with network measurements enabled. | Connections between hops and round-trip time (RTT) from the source agent to each hop. |

For supported tests and the Network Path span hierarchy, see [ThousandEyes for OpenTelemetry Data Model v2 - Traces](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/traces).

### Step 1: Create a Traces Integration

#### Using the ThousandEyes Platform

After you complete [Creating a Splunk HEC Connector in ThousandEyes](#creating-a-splunk-hec-connector-in-thousandeyes), add an operation for traces:

1. In ThousandEyes, go to **Manage > Integrations > Integrations 2.0**.
2. Click **+ New Operation**.
3. Select **Splunk Enterprise, Splunk Cloud Platform**.
4. Enter an **Operation Name**.
5. (Optional) Set **Index**, **Source**, and **Source Type**.

   For more information, see the Splunk documentation for [indexes](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Aboutindexesandindexers), [source](https://docs.splunk.com/Splexicon:Source), and [source type](https://docs.splunk.com/Splexicon:Sourcetype).
6. Set **OpenTelemetry Signal** to **Trace**.
7. Under **Data Sources**, select the source to export:
   * For HAR application traces, select **HAR** and clear **Network Path**.
   * For Network Path traces, select **Network Path** and clear **HAR**.
   * To export both sources, select both checkboxes.
8. Select tests that produce the selected trace data.
9. (Optional) At the bottom of the configuration form, click **Test**.

   A test message is sent to the target to verify the connection. A success or failure message is shown.
10. Click **Save**.

#### Using the ThousandEyes API

The following examples create separate integrations for HAR application traces and Network Path traces. Both use an events index and the source `ThousandEyesOTel` so that you can search for the exported spans.

Replace `{HOST}`, `{TOKEN}`, and `{EVENT_INDEX}` with your Splunk HEC host, token, and events index. For Splunk Enterprise, replace `streamEndpointUrl` with `https://<host>:8088/services/collector/event`. The URL must satisfy the [Stream endpoint URL requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/url-target-requirements).

**HAR Application Trace Example**

Replace `{HAR_TEST_ID}` with the ID of a page load, web transaction, or API test. This example enables HAR input and disables Network Path input:

```curl
curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "signal": "trace",
  "dataModelVersion": "v2",
  "testMatch": [
    {
      "id": "{HAR_TEST_ID}",
      "domain": "cea"
    }
  ],
  "inputConfig": {
    "trace": {
      "har": { "enabled": true },
      "netpath": { "enabled": false }
    }
  },
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}",
      "index": "{EVENT_INDEX}",
      "source": "ThousandEyesOTel"
    }
  }
}'
```

**Network Path Trace Example**

Replace `{NETWORK_PATH_TEST_ID}` with the ID of an agent-to-server or agent-to-agent test. This example enables Network Path input and disables HAR input:

```curl
curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "signal": "trace",
  "dataModelVersion": "v2",
  "testMatch": [
    {
      "id": "{NETWORK_PATH_TEST_ID}",
      "domain": "cea"
    }
  ],
  "inputConfig": {
    "trace": {
      "har": { "enabled": false },
      "netpath": { "enabled": true }
    }
  },
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}",
      "index": "{EVENT_INDEX}",
      "source": "ThousandEyesOTel"
    }
  }
}'
```

To export both trace sources from one integration, set both `inputConfig.trace.har.enabled` and `inputConfig.trace.netpath.enabled` to `true`. Select tests that produce the corresponding trace data.

### Step 2: Visualize Traces in Splunk

HEC stores each exported span as an event. The span event contains `trace_id`, `span_id`, `parent_span_id`, `name`, `start_time`, `end_time`, and `attributes`. The start and end times are Unix timestamps in nanoseconds. Shared resource attributes, such as `thousandeyes.test.name`, are available as event fields.

1. Open **Search & Reporting** in Splunk Cloud Platform or Splunk Enterprise.
2. Set the time range to include completed test rounds.
3. Search for `index="{EVENT_INDEX}" source="ThousandEyesOTel"`, replacing `{EVENT_INDEX}` with the events index configured for the integration.
4. On the **Events** tab, expand a span event and copy its `trace_id`. HAR request spans contain HTTP attributes. Network Path spans include names such as `ThousandEyes test task` and `network path edge to <receiving node>`.

If you configured a different source, replace `ThousandEyesOTel` in the searches below. The searches use automatically extracted JSON fields. If your source type does not extract those fields, replace the first line with `index="{EVENT_INDEX}" source="ThousandEyesOTel" | spath | search trace_id="{TRACE_ID}"`, using the trace ID you want to inspect, then keep the remaining commands starting with `dedup`. Extract the fields before filtering on `trace_id`. These examples use Splunk search and visualization features and do not require an add-on.

#### Inspecting HAR Application Traces

Replace `{HAR_TRACE_ID}` with a HAR trace ID. This search shows the HTTP request spans in that trace:

```spl
index="{EVENT_INDEX}" source="ThousandEyesOTel" trace_id="{HAR_TRACE_ID}"
| dedup trace_id span_id
| where isnotnull('attributes.http.request.method')
| eval method='attributes.http.request.method', response_code='attributes.http.response.code', url='attributes.url.full'
| eval duration_ms=round((end_time-start_time)/1000000,3)
| sort 0 start_time
| table name method response_code duration_ms url span_id parent_span_id
```

Open the **Statistics** tab to inspect request names, HTTP response codes, durations, and parent span IDs.

![Splunk HAR search and Statistics table showing HTTP request names, methods, response codes, durations, and URLs](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-fc6911ee7f07fd524f408bc2d056e63a351e4193%2Fsplunk-har-request-table.jpg?alt=media)

To compare the 10 longest HTTP request spans in a chart, run:

```spl
index="{EVENT_INDEX}" source="ThousandEyesOTel" trace_id="{HAR_TRACE_ID}"
| dedup trace_id span_id
| where isnotnull('attributes.http.request.method')
| eval duration_ms=round((end_time-start_time)/1000000,3)
| eval request=name . " [" . substr(span_id,1,8) . "]"
| chart max(duration_ms) AS duration_ms OVER request
| sort 0 - duration_ms
| head 10
```

Open the **Visualization** tab and select a bar chart. The chart shows the 10 longest HTTP request spans, with each bar showing a request duration in milliseconds. Use the request URL and response code in the table to investigate slow or failed requests. To retain the chart, select **Save As > Dashboard Panel**.

![Splunk bar chart of the ten longest HAR request durations in a trace](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-de438120eed992c161ef72035f1d1a29432a6c35%2Fsplunk-har-request-durations.jpg?alt=media)

#### Inspecting Network Path Traces

Replace `{NETWORK_PATH_TRACE_ID}` with a Network Path trace ID. Each topology edge has a client and server span pair. This search selects client spans so that each edge appears once per path attempt:

```spl
index="{EVENT_INDEX}" source="ThousandEyesOTel" trace_id="{NETWORK_PATH_TRACE_ID}"
| dedup trace_id span_id
| where kind="SPAN_KIND_CLIENT" AND isnotnull('attributes.network.path.hop.index')
| eval hop=tonumber('attributes.network.path.hop.index'), sender='attributes.network.local.address', receiver='attributes.network.peer.address'
| eval receiver_name='attributes.network.peer.hostname', rtt_ms=tonumber('attributes.network.path.hop.rtt')
| sort 0 hop
| table hop sender receiver receiver_name rtt_ms span_id parent_span_id
```

![Splunk Statistics table showing Network Path hop addresses and source-to-hop RTT](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-dd18fea7827ddb85e92c25175ac170de88b04f4a%2Fsplunk-network-path-hop-table.jpg?alt=media)

Open the **Statistics** tab to follow the connections from each sending node to its receiving node. Multiple path attempts can produce more than one row for a hop. To compare source-to-hop RTT, run:

```spl
index="{EVENT_INDEX}" source="ThousandEyesOTel" trace_id="{NETWORK_PATH_TRACE_ID}"
| dedup trace_id span_id
| where kind="SPAN_KIND_CLIENT" AND isnotnull('attributes.network.path.hop.index')
| eval hop=tonumber('attributes.network.path.hop.index'), rtt_ms=tonumber('attributes.network.path.hop.rtt')
| chart max(rtt_ms) AS source_to_hop_rtt_ms OVER hop
```

Open the **Visualization** tab and select a column chart. The chart shows the maximum reported source-to-hop RTT for each hop index within the selected trace. If a trace contains multiple path attempts, the maximum is taken across those attempts. To retain the chart, select **Save As > Dashboard Panel**.

Network Path attempt and edge spans are point observations with equal start and end times. Use `network.path.hop.rtt` for source-to-hop RTT in milliseconds. Span duration does not represent link latency. A hop without a reported RTT has no RTT value in the chart. For the complete topology and timing semantics, see [Network Path Traces](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/traces#network-path-traces).

**Creating a Network Path Graph in Dashboard Studio**

If your Splunk deployment provides the native Dashboard Studio [Network graph](https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/dashboard-studio/10.6/visualizations/network-graph) visualization, you can display the selected trace as a directed topology. Use the following search as the chart's primary data source, replacing `{EVENT_INDEX}` and `{NETWORK_PATH_TRACE_ID}` with your events index and one Network Path trace ID:

```spl
index="{EVENT_INDEX}" source="ThousandEyesOTel" trace_id="{NETWORK_PATH_TRACE_ID}"
| dedup trace_id span_id
| where kind="SPAN_KIND_CLIENT" AND isnotnull('attributes.network.path.hop.index')
| eval hop=tonumber('attributes.network.path.hop.index'), rtt_ms=tonumber('attributes.network.path.hop.rtt')
| eval sender=coalesce(nullif('service.name',""),nullif('attributes.network.local.address',""),"unknown hop " . tostring(hop-1))
| eval receiver=replace(name,"^network path edge to ","")
| eval sender=if(match(sender,"^unknown hop [0-9]+$"),"Unresponsive",sender), receiver=if(match(receiver,"^unknown hop [0-9]+$"),"Unresponsive",receiver)
| eval source=printf("H%02d | %s",hop-1,sender), target=printf("H%02d | %s",hop,receiver)
| stats count AS observations max(rtt_ms) AS source_to_hop_rtt_ms BY source target
| appendpipe [ stats values(target) AS source | mvexpand source | eval target=null() ]
| table source target observations source_to_hop_rtt_ms
```

The query uses client spans to create directed connections. Node labels combine the hop position with the source agent name, an intermediate IP address, or the destination hostname. Hop positions keep repeated IP addresses at different positions separate. The appended rows contain a source node with an empty target so that the graph also includes terminal nodes.

1. Create or edit a **Dashboard Studio** dashboard.
2. Select **Add chart**, then **Network graph**.
3. Under **Data sources**, select **+ Create search** and paste the search above. Set the search time range to include the selected trace.
4. Under **Data configurations**, set **Source** to `source`, **Target** to `target`, and **Node labels** to `source`.
5. Under **Data display**, select **Hierarchical** layout, set the hierarchy direction to left to right, and select one-way links.
6. Save the dashboard.

![Dashboard Studio Network graph showing directed connections from a ThousandEyes Agent through four hop positions](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-bbb8502f366e728ab6a3f9e2fdfdcf86f0806253%2Fsplunk-network-path-graph.jpg?alt=media)

Follow the arrows from `H00`, the source agent, toward the destination. Each arrow represents an observed connection between consecutive hop positions. Branches show alternatives observed across attempts. Use the graph's zoom controls to inspect long paths.

The graph combines observed connections across all path attempts in the selected trace. `observations` counts the client spans for each connection, and `source_to_hop_rtt_ms` contains the maximum reported source-to-receiving-hop RTT across those observations. This RTT is not the delay between adjacent nodes. Keep the table search above when you need the individual edge spans.

`Unresponsive` labels represent hops with no response. The graph combines these placeholders at the same hop position across attempts; they do not identify a particular router. The combined graph does not reconstruct each attempt separately.

## Stream OpenTelemetry Logs to Splunk

### Step 1: Create a Logs Integration

#### Using the ThousandEyes Platform

After you complete [Creating a Splunk HEC Connector in ThousandEyes](#creating-a-splunk-hec-connector-in-thousandeyes), add an operation for logs:

1. In ThousandEyes, go to **Manage > Integrations > Integrations 2.0**.
2. Click **+ New Operation**.
3. Select **Splunk Enterprise, Splunk Cloud Platform**.
4. Enter an **Operation Name**.
5. (Optional) Set **Index**, **Source**, and **Source Type**.

   For more information, see the Splunk documentation for [indexes](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Aboutindexesandindexers), [source](https://docs.splunk.com/Splexicon:Source), and [source type](https://docs.splunk.com/Splexicon:Sourcetype).
6. Set **OpenTelemetry Signal** to **Log**.
7. (Optional) At the bottom of the configuration form, click **Test**.

   A test message is sent to the target to verify the connection. A success or failure message is shown.
8. Click **Save**.

#### Using the ThousandEyes API

Create the HTTP stream programmatically:

```curl
curl -i -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
  "type": "splunk-hec",
  "signal": "log",
  "dataModelVersion": "v2",
  "testMatch": [
    {
      "id": "{TEST_ID}",
      "domain": "{DOMAIN}"
    }
  ],
  "endpointType": "http",
  "streamEndpointUrl": "https://http-inputs-{HOST}.splunkcloud.com:443/services/collector/event",
  "exporterConfig": {
    "splunkHec": {
      "token": "{TOKEN}"
    }
  }
}'
```

When you create a stream from Splunk Enterprise or Splunk Cloud Platform, ensure the following:

* The `type` is `"splunk-hec"`.
* The `endpointType` is `"http"`.
* Set `signal` to `"log"`.
* `exporterConfig.splunkHec.token` is set to your Splunk HEC token.
* If you associated an index with the HEC token, set `exporterConfig.splunkHec.index` to that index name.
* The `streamEndpointUrl` satisfies the [Stream endpoint URL requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/url-target-requirements).

Replace `{HOST}`, `{TOKEN}`, `{TEST_ID}`, and `{DOMAIN}` with your Splunk Cloud host, HEC token, and test identifiers.

### Step 2: Search logs in Splunk

Search event data using queries such as `source=ThousandEyesOTel` or your configured index (for example, `index="thousandeyes_otel_events_index"`).

![Search in Splunk](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-5f3a525cf8a2bfb651b5cc418583f63b2c863908%2Fproduct-documentation_integration-guides_custom-built-integrations_opentelemetry-integrations-splunk-events-search.png?alt=media\&token=45619b5d-d00f-44e0-88f3-487f81bd57fd)

![ThousandEyes log attributes in Splunk](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-95ee63c53b06e1bf6f91fd0dba349b9b6b8ea266%2Fsplunk-log-event-attributes.png?alt=media)
