Splunk Observability Cloud

Splunk Observability Cloud is a real-time observability platform purpose-built for monitoring metrics and traces at scale. It ingests OpenTelemetry data and provides advanced dashboards and analytics to help teams detect and resolve performance issues quickly. This guide explains how to integrate ThousandEyes telemetry with Splunk Observability Cloud using OpenTelemetry.

For more information about sending metrics to Splunk Observability Cloud using OpenTelemetry, see Splunk Observability Cloud: Manage Data.

Prerequisites

• You have created a ThousandEyes test.

• You know your Splunk Observability Cloud realm (for example us0, us1, eu0). To find your realm, see About Splunk Observability Cloud Realms.

• You use an ingest hostname supported by Splunk. You can use the current hostname ingest.{REALM}.observability.splunkcloud.com or the legacy hostname ingest.{REALM}.signalfx.com. Splunk documents both hostnames for OTLP/HTTP in OTLP/HTTP exporter.

• You have created a Splunk Observability Cloud access token with permission to send data:

  1. In the Splunk Observability Cloud platform, go to Settings > Access Token.

  2. Click Create Token.

  3. Enter a Name.

  4. Select the Ingest scope.

  5. Click Create to generate your access token.

  6. Copy the access token and store it securely.

     You need the access token to send telemetry data to Splunk Observability Cloud.

For more information, see Create and manage organization access tokens using Splunk Observability Cloud.

Stream OpenTelemetry Metrics to Splunk Observability Cloud

Step 1: Create a Metrics Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select ThousandEyes for OpenTelemetry.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. In Target, enter the OTLP/HTTP metrics ingest URL:

   https://ingest.{REALM}.observability.splunkcloud.com/v2/datapoint/otlp

   Replace {REALM} with your realm (for example us1). If your organization still uses the legacy hostname, use https://ingest.{REALM}.signalfx.com/v2/datapoint/otlp instead.

6. For Preset Configurations, select Splunk Observability Cloud.

7. For Auth Type, select Custom.

8. Add the following Custom Headers:

   • X-SF-Token: your Splunk Observability Cloud access token from Prerequisites.

   • Content-Type: application/x-protobuf.

9. For the OpenTelemetry Signal, select Metric.

10. For the Data Model Version, select v2.

11. Select a test.

    For more information on creating a test, see General Setup Instructions.

12. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "http",
    "signal": "metric",
    "dataModelVersion": "v2",
    "streamEndpointUrl": "https://ingest.{REALM}.observability.splunkcloud.com/v2/datapoint/otlp",
    "customHeaders": {
      "X-SF-Token": "{TOKEN}",
      "Content-Type": "application/x-protobuf"
    }
  }'

Using gRPC (optional)

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "grpc",
    "signal": "metric",
    "dataModelVersion": "v2",
    "streamEndpointUrl": "https://ingest.{REALM}.observability.splunkcloud.com:443",
    "customHeaders": {
      "X-SF-Token": "{TOKEN}",
      "Content-Type": "application/x-protobuf"
    }
  }'

Replace {REALM}, {TOKEN}, and the testMatch values with your Splunk realm, access token, and test identifiers.

ThousandEyes Dashboard in Splunk Observability Cloud

When metrics from your integration appear in Splunk Observability Cloud, you can use the ThousandEyes Network Monitoring Dashboard. The dashboard includes:

• HTTP Server Availability (%): Availability of monitored HTTP servers.

• HTTP Throughput (bytes/s): Data transfer rate over time.

• Client Request Duration (seconds): Latency of client requests.

• Web Page Load Completion (%): Percentage of successful page loads.

• Page Load Duration (seconds): Time taken to load pages.

ThousandEyes Dashboard in Splunk Observability Cloud

You can download the dashboard template from ThousandEyes Splunk Observability Cloud dashboard template on GitHub.

Stream OpenTelemetry Traces to Splunk Observability Cloud

Step 1: Create a Traces Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select ThousandEyes for OpenTelemetry.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. In Target, enter the OTLP/HTTP trace ingest URL:

   https://ingest.{REALM}.observability.splunkcloud.com/v2/trace/otlp

   Replace {REALM} with your realm. If your organization uses the legacy hostname, use https://ingest.{REALM}.signalfx.com/v2/trace/otlp instead.

6. For Preset Configurations, select Splunk Observability Cloud.

7. For Auth Type, select Custom.

8. Add the following Custom Headers:

   • X-SF-Token: your Splunk Observability Cloud access token from Prerequisites.

   • Content-Type: application/x-protobuf.

9. For the OpenTelemetry Signal, select Trace.

10. Select a test.

    For more information on creating a test, see General Setup Instructions.

11. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "http",
    "signal": "trace",
    "streamEndpointUrl": "https://ingest.{REALM}.observability.splunkcloud.com/v2/trace/otlp",
    "customHeaders": {
      "X-SF-Token": "{TOKEN}",
      "Content-Type": "application/x-protobuf"
    }
  }'

Using gRPC (optional)

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "grpc",
    "signal": "trace",
    "streamEndpointUrl": "https://ingest.{REALM}.observability.splunkcloud.com:443",
    "customHeaders": {
      "X-SF-Token": "{TOKEN}",
      "Content-Type": "application/x-protobuf"
    }
  }'

Replace {REALM}, {TOKEN}, and the testMatch values with your Splunk realm, access token, and test identifiers.

Step 2: Visualize Traces in Splunk Observability Cloud

1. In the Splunk Observability Cloud, open APM (Application Performance Monitoring).

2. Go to Traces and open the trace explorer or service view for your environment.

3. Filter or search using ThousandEyes-related attributes (for example service name or resource attributes your team maps from OpenTelemetry) to locate ThousandEyes spans.

For more information about APM and traces, see Introduction to Splunk APM. If you also link ThousandEyes tests to Splunk APM, see Distributed Tracing with Splunk Observability APM.