# Distributed Tracing with Splunk Observability APM

This guide explains how to integrate Splunk Observability Cloud APM (Application Performance Monitoring) with ThousandEyes. This integration helps you trace requests across services and identify whether issues are caused by the network or the application layer.

## Requirements

Before you start, ensure the following requirements are met:

* The monitored endpoint is configured correctly. See [Monitored Endpoint Requirements](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/distributed-tracing#monitored-endpoint-requirements).
* You have access to [Splunk Observability Cloud](https://www.splunk.com/en_us/observability.html).
  * Your user account must have the *Power* or *Admin* role to create an access token.
* You have [created a ThousandEyes test with distributed tracing](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/distributed-tracing#create-a-thousandeyes-test-with-distributed-tracing).

## Configuration steps

### Step 1: Create the Splunk APM Integration in ThousandEyes

1. In ThousandEyes, go to **Manage > Integrations > Integrations 2.0**.
2. Create a **Generic Connector** with the following details:
   * **Target URL**:\
     `https://api.<REALM>.signalfx.com`\
     Replace `<REALM>` with your Splunk realm (for example, `us0`, `eu1`).
   * **Custom Headers**:
     * **Key**: `X-SF-Token`
     * **Value**: Your Splunk access token with the **API** scope.\
       For more information, see [Create and manage organization access tokens using Splunk Observability Cloud](https://help.splunk.com/en/splunk-observability-cloud/administer/authentication-and-security/authentication-tokens/org-access-tokens).

       ![Splunk APM Generic Connector](/files/icax9XIIaF1aIshDiJCg)
3. Create an operation:

   1. Click **+ New Operation**.
   2. Choose **Splunk Observability APM**.
   3. Enter an operation name.
   4. Enable the operation.

   ![Splunk APM Operation](/files/rJl6S1G0nz9IijWWAy1L)

### Step 2: View the Service Map

![Service Map](/files/Mvp8K6mnKYma2ji8jSym)

1. Open the **Service Map** tab in ThousandEyes.
2. Use the service map to analyze the trace path. You can identify:
   * The services involved in the request.
   * Any latency issues, highlighted in red if thresholds are exceeded.
   * Any errors between services, shown as red lines if a request fails.
   * Trace metadata, such as the trace ID and request flow details.

### Step 3: Debug the Trace in Splunk Observability Cloud

From the **Service Map** tab in ThousandEyes, follow the link to the trace in Splunk. There, you can:

* Drill into service-level trace data.
* Use Splunk’s trace search, filters, and dashboards for deeper analysis.

![Trace in Splunk](/files/No2Fb1FIoK33rxfhyDgq)

Splunk enriches the trace with the following attributes:

* `thousandeyes.account.id`
* `thousandeyes.test.id`
* `thousandeyes.permalink`
* `thousandeyes.source.agent.id`

These attributes provide context and allow you to navigate back to the related test in ThousandEyes.

## Troubleshooting

### Missing ThousandEyes Link in Splunk Observability Cloud

This issue occurs when the `b3` propagator overrides the `trace_state` span attribute and clears its value. Set the `OTEL_PROPAGATORS` environment variable to the following value:

```
OTEL_PROPAGATORS=baggage,b3,tracecontext
```

After updating the environment variable, restart the instrumented service for the change to take effect.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/distributed-tracing/distributed-tracing-splunk-apm.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.