Distributed Tracing with Splunk Observability APM

This guide explains how to integrate Splunk Observability Cloud APM (Application Performance Monitoring) with ThousandEyes. This integration helps you trace requests across services and identify whether issues are caused by the network or the application layer.

Requirements

• Access to Splunk Observability Cloud.

• The monitored endpoint must:

  • Be instrumented for distributed tracing.

  • Send telemetry data to Splunk Observability Cloud.

Step 1: Create the Splunk APM Integration in ThousandEyes

1. In the ThousandEyes platform, go to Manage > Integrations > Integrations 2.0.

2. Create a Generic Connector with the following details:

   • Target URL: https://api.<REALM>.signalfx.com Replace <REALM> with your Splunk realm (for example, us0, eu1).

   • Custom headers:

     • Key: X-SF-Token

     • Value: Your Splunk access token with API scope. For more information, see Create and manage organization access tokens using Splunk Observability Cloud.

       

       Splunk APM Generic Connector
3. Follow these steps to create an operation:

   1. Click + New Operation to open the menu to select the operation type.

   2. Choose Splunk Observability APM to proceed to the configration form.

   3. Enter the Operation Name.

   4. Enabled the operation.

   

   Splunk APM Operation

Step 2: Create a ThousandEyes Test with Distributed Tracing

1. Create a HTTP Server Test or API Test in ThousandEyes. For more information on creating a test, see Getting Started with API Test and HTTP Server Tests.
2. Under Advanced settings, enable Distributed Tracing.
3. ThousandEyes automatically injects the following trace headers into requests:

   • b3

   • traceparent

   • tracestate

   

   Request Headers

   This enables ThousandEyes to know the TraceId of the request such as 0a0c3e94ca224a898f43038e8e99519a.

Step 3: View the Service Map

Service Map

1. Open the Service Map tab in ThousandEyes.

2. Use the service map to analyze the trace path. You can identify:

   • The services involved in the request.

   • Any latency issues, highlighted in red if thresholds are exceeded.

   • Any errors between services, shown as red lines if a request fails.

   • Trace metadata, such as the trace ID and request flow details.

Step 4: Debug the Trace in Splunk Observability Cloud

From the Service Map tab in ThousandEyes, follow the link to the trace in Splunk. There, you can:

• Drill into service-level trace data.

• Use Splunk’s trace search, filters, and dashboards for deeper analysis.

Trace in Splunk

Splunk enriches the trace with the following attributes:

• thousandeyes.account.id

• thousandeyes.test.id

• thousandeyes.permalink

• thousandeyes.source.agent.id

These attributes provide context and allow you to navigate back to the related test in ThousandEyes.