Grafana

Grafana is an open-source visualization and monitoring platform that lets you create customizable dashboards and analyze observability data from multiple sources. This guide explains how to integrate ThousandEyes metrics with Grafana using OpenTelemetry.

For more information about sending metrics to Grafana using OpenTelemetry, see Grafana Cloud: Send data using OpenTelemetry Protocol (OTLP).

Prerequisites

• You have created a ThousandEyes test.

• You have a Grafana Cloud account.

• You have created a Grafana Cloud OpenTelemetry connection:

  1. In the Grafana Cloud platform, go to Connections.

  2. Click Add new connection.

  3. Search for OpenTelemetry Collector.

  4. Scroll to the Generate OpenTelemetry Collector Configuration step.

  5. Enter an Access Policy token name.

  6. Click Create token.

  7. Copy the generated OpenTelemetry Collector Configuration for future use.

Grafana OpenTelemetry Collector Configuration

Stream OpenTelemetry Metrics to Grafana

Step 1: Create a Metrics Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/metrics to the end of your endpoint url.

   For example: https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/metrics.

   

   Grafana Endpoint URL
6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Metric.

10. For the Data Model Version, select v1.

11. Select a test.

    For more information on creating a test, see General Setup Instructions.

12. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H \
"Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/metrics",
  "customHeaders": {
    "Authorization": "Basic $(echo -n $GRAFANA_USER:$GRAFANA_PASSWORD | base64)"
  },
  "dataModelVersion": "v1"
}'

Replace $GRAFANA_USER, $GRAFANA_PASSWORD and {STREAM_ENDPOINT_URL} with the correct values for your Grafana instance.

ThousandEyes Dashboard in Grafana

To access the ThousandEyes Dashboard in Grafana, visit the official Grafana dashboard page ThousandEyes Dashboard on Grafana.

ThousandEyes Dashboard in Grafana

ThousandEyes Dashboard in Grafana

Stream OpenTelemetry Traces to Grafana

Step 1: Create a Traces Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/traces to the end of your endpoint url.

   For example, https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces.

6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Trace.

10. Select a test.

    For more information on creating a test, see General Setup Instructions.

11. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/stream \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" -d '{
    "type": "opentelemetry",
    "testMatch": [{
      "id": "281474976717575",
      "domain": "cea"
    }],
    "endpointType": "http",
    "signal": "trace",
    "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/traces",
    "customHeaders": {
      "Authorization": "Basic $(echo -n $GRAFANA_USER:$GRAFANA_PASSWORD | base64)"
    }
}'

Replace $GRAFANA_USER, $GRAFANA_PASSWORD and {STREAM_ENDPOINT_URL} with the correct values for your Grafana instance.

Step 2: Visualize Traces in Grafana

1. Go to Explore.

2. Select your tracing backend (Tempo).

3. Search for ThousandEyes trace entries by applying a filter such as:

   {resource.thousandeyes.account.id=~".*"}

ThousandEyes Traces in Grafana

Stream OpenTelemetry Logs to Grafana

Step 1: Create a Logs Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select OpenTelemetry Integration.

3. Enter a Name for the integration.

4. Set the Target to HTTP.

5. For the Endpoint URL, enter the endpoint from your generated OpenTelemetry Collector Configuration and add /v1/logs to the end of your endpoint url.

   For example, https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/logs.

6. For Preset Configurations, select Grafana.

7. For Auth Type, select Basic.

8. Enter the username and password from your OpenTelemetry Collector Configuration.

9. For the OpenTelemetry Signal, select Log.

10. Select a test.

    For more information on creating a test, see General Setup Instructions.

11. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

curl -v -XPOST https://api.thousandeyes.com/v7/stream \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BEARER_TOKEN" -d '{
    "type": "opentelemetry",
    "endpointType": "http",
    "signal": "log",
    "streamEndpointUrl": "{STREAM_ENDPOINT_URL}/v1/logs",
    "customHeaders": {
      "Authorization": "Basic $(echo -n $GRAFANA_USER:$GRAFANA_PASSWORD | base64)"
    }
}'

Step 2: Visualize Logs in Grafana

1. In Grafana Cloud, go to Explore.

2. Select your logs data source (typically Loki).

3. Search for ThousandEyes log entries by applying a filter such as:

   {service_name=~"unknown_service"} | thousandeyes_account_id=~".*" |= ""

   This filters logs that include a ThousandEyes account ID.

ThousandEyes Logs in Grafana