Grafana

Grafana is an open-source visualization and monitoring platform that lets you create customizable dashboards and analyze observability data from multiple sources. This guide explains how to integrate ThousandEyes metrics, traces, and logs with Grafana using OpenTelemetry.

For more information about sending telemetry to Grafana Cloud using OpenTelemetry, see Grafana Cloud: Send data using OpenTelemetry Protocol (OTLP).

ThousandEyes Data Model Version 2 uses OpenTelemetry resource attributes. Grafana Cloud does not promote resource attributes to queryable metric labels by default.

To use data Data Model Version 2 for metric streams to Grafana, configure resource attribute promotion in Grafana Cloud. For the list of ThousandEyes resource attributes, see Resource attributes. For more information about promotion, see OpenTelemetry with Prometheus: Better integration through resource attribute promotion.

If you cannot configure resource attribute promotion, use Data Model Version 1 for metric streams to Grafana. For more information about this limitation, see OpenTelemetry metric resource attributes are dropped when exported to Grafana Cloud.

Prerequisites

• You have created a ThousandEyes test.

• You have a Grafana Cloud account.

• You have created a Grafana Cloud OpenTelemetry connection:

  1. In the Grafana Cloud platform, go to Connections.

  2. Select Add new connection.

  3. Search for OpenTelemetry Collector.

  4. Scroll to the Generate OpenTelemetry Collector Configuration step.

  5. Enter an Access Policy token name.

  6. Select Create token.

  7. Copy the generated OpenTelemetry Collector Configuration for future use.

Grafana OpenTelemetry Collector Configuration

Stream OpenTelemetry Metrics to Grafana

Step 1: Create a Metrics Integration

Using the ThousandEyes UI for Metrics

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Select +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/metrics to the end of your endpoint URL.

   For example: https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/metrics.

   

   Grafana Endpoint URL
6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Metric.

10. For the Data Model Version, select v2.

11. Select a test.

    For more information on creating a test, see General Setup Instructions.

12. Select Save.

Using the ThousandEyes API for Metrics

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/streams -H "Content-Type: application/json" -H \
"Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/metrics",
  "customHeaders": {
    "Authorization": "Basic <BASE64_ENCODED_GRAFANA_CREDENTIALS>"
  },
  "dataModelVersion": "v2"
}'

Replace {STREAM_ENDPOINT_URL} and <BASE64_ENCODED_GRAFANA_CREDENTIALS> with the correct values for your Grafana instance.

Troubleshooting Grafana Integration Issues

Self-Managed Prometheus Out-of-Order Sample Errors

If you use Grafana with a self-managed Prometheus backend, Prometheus might reject OpenTelemetry metrics with an "out of order sample" error when samples arrive out of order. This issue might occur when metrics are batched or when multiple OpenTelemetry Collector replicas send samples without coordinated ordering.

This Prometheus setting applies to the Prometheus server configuration. You cannot configure it in the Grafana Cloud OpenTelemetry connection or by adding it to the OpenTelemetry Collector configuration.

To allow Prometheus to accept samples that arrive within a configured delay, add an out-of-order time window to the Prometheus configuration file loaded by the --config.file flag. Choose a window that matches your environment. For example:

storage:
  tsdb:
    out_of_order_time_window: 30m

For more information, see Enable out-of-order ingestion in the Prometheus documentation.

ThousandEyes Dashboard in Grafana

To access the ThousandEyes Dashboard in Grafana, visit the official Grafana dashboard page ThousandEyes Dashboard on Grafana.

ThousandEyes Dashboard in Grafana

ThousandEyes Dashboard in Grafana

Stream OpenTelemetry Traces to Grafana

Step 1: Create a Traces Integration

Using the ThousandEyes UI for Traces

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Select +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/traces to the end of your endpoint URL.

   For example, https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces.

6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Trace.

10. Select a test.

    For more information on creating a test, see General Setup Instructions.

11. Select Save.

Using the ThousandEyes API for Traces

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "http",
    "signal": "trace",
    "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/traces",
    "customHeaders": {
      "Authorization": "Basic <BASE64_ENCODED_GRAFANA_CREDENTIALS>"
    }
}'

Replace {STREAM_ENDPOINT_URL} and <BASE64_ENCODED_GRAFANA_CREDENTIALS> with the correct values for your Grafana instance.

Step 2: Visualize Traces in Grafana

1. Go to Explore.

2. Select your tracing backend (Tempo).

3. Search for ThousandEyes trace entries by applying a filter such as:

   {resource.thousandeyes.account.id=~".*"}

ThousandEyes Traces in Grafana

Stream OpenTelemetry Logs to Grafana

Step 1: Create a Logs Integration

Using the ThousandEyes UI for Logs

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Select +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/logs to the end of your endpoint URL.

   For example, https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/logs.

6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Log.

10. Select a test.

    For more information on creating a test, see General Setup Instructions.

11. Select Save.

Using the ThousandEyes API for Logs

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/streams \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" -d '{
    "type": "opentelemetry",
    "endpointType": "http",
    "signal": "log",
    "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/logs",
    "customHeaders": {
      "Authorization": "Basic <BASE64_ENCODED_GRAFANA_CREDENTIALS>"
    }
}'

Replace {STREAM_ENDPOINT_URL} and <BASE64_ENCODED_GRAFANA_CREDENTIALS> with the correct values for your Grafana instance.

Step 2: Visualize Logs in Grafana

1. In Grafana Cloud, go to Explore.

2. Select your logs data source (typically Loki).

3. Search for ThousandEyes log entries by applying a filter such as:

   {service_name=~"unknown_service"} | thousandeyes_account_id=~".*" |= ""

   This filters logs that include a ThousandEyes account ID.

ThousandEyes Logs in Grafana