Dynatrace

Dynatrace is an observability and security platform that uses causal AI to deliver precise insights and automate DevSecOps at scale. This guide shows how to stream ThousandEyes metrics, traces, and logs to Dynatrace using OpenTelemetry.

For Dynatrace’s OpenTelemetry overview, see OpenTelemetry and Dynatrace. For the signals ThousandEyes supports, see Supported Signals.

Prerequisites

• You have created a ThousandEyes test.

• You have a Dynatrace API access token with scopes Ingest metrics, Ingest logs, Ingest OpenTelemetry traces and Ingest events:

  1. In Dynatrace, search for access token.

  2. Go to Access Tokens > Generate new token.

  3. Enter a token name and select the scopes above.

  4. Click Generate token, then copy the token for later use.

Stream OpenTelemetry Metrics to Dynatrace

Step 1: Create a Metrics Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and choose OpenTelemetry Integration.

3. Enter a Name.

4. Set Target to HTTP.

5. Set the Endpoint URL to https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/metrics (replace {INSTANCE}).

6. For Preset Configurations, select Dynatrace.

7. Set Auth Type to Custom and add "Authorization": "Api-Token {TOKEN}" (use the token from Prerequisites).

8. For Signal, select Metric.

9. For Data Model Version, select v2.

10. Select a test. For help, see General Setup Instructions.

11. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "signal": "metric",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/metrics",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'

Replace {TOKEN} and {INSTANCE} with your Dynatrace values.

Step 2: Enable ThousandEyes Resource Attributes in Dynatrace

Allow ThousandEyes resource attributes so they appear on metrics:

1. In Dynatrace, go to Settings and search for OpenTelemetry metrics.

2. Under Allow list: resource and scope attributes, add these keys (click Add Item for each):

   • thousandeyes.stream.id

   • thousandeyes.test.id

   • thousandeyes.test.name

   • thousandeyes.source.agent.id

   • thousandeyes.source.agent.name

3. For the full list, see common resource attributes.

4. Click Save Changes.

ThousandEyes Dashboard in Dynatrace

1. Download the dashboard template: ThousandEyes Dynatrace dashboard template.

2. In Dynatrace, go to Dashboards and click Upload.

3. Open the dashboard to visualize ThousandEyes data.

ThousandEyes Dashboard in Dynatrace

Stream OpenTelemetry Traces to Dynatrace

Step 1: Create a Traces Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select OpenTelemetry Integration.

3. Enter a Name.

4. Set Target to HTTP.

5. Set the Endpoint URL to https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/traces (replace {INSTANCE}).

6. For Preset Configurations, select Dynatrace.

7. Set Auth Type to Custom and add "Authorization": "Api-Token {TOKEN}".

8. For Signal, select Trace.

9. Select a test. See General Setup Instructions if needed.

10. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "signal": "trace",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/traces",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'

Replace {TOKEN} and {INSTANCE} with your Dynatrace values.

Step 2: Visualize Traces in Dynatrace

1. In Dynatrace, open Distributed Tracing.

2. Search for spans from ThousandEyes and apply filters (duration, errors, target URL, etc.).

ThousandEyes Traces in Dynatrace

Stream OpenTelemetry Logs to Dynatrace

Step 1: Create a Logs Integration

Using the ThousandEyes UI

1. In ThousandEyes, go to Manage > Integration 1.0.

2. Click +New Integration and select OpenTelemetry Integration.

3. Enter a Name.

4. Set Target to HTTP.

5. Set the Endpoint URL to https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/logs (replace {INSTANCE}).

6. For Preset Configurations, select Dynatrace.

7. Set Auth Type to Custom and add "Authorization": "Api-Token {TOKEN}".

8. For Signal, select Log.

9. Select a test. For details, see General Setup Instructions.

10. Click Save.

Using the ThousandEyes API

Create the HTTP stream programmatically:

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "endpointType": "http",
  "signal": "log",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/logs",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'

Replace {TOKEN} and {INSTANCE} with your Dynatrace values.

Step 2: Visualize Logs in Dynatrace

1. In Dynatrace, open Logs.

2. Search for ThousandEyes logs and apply filters (body, timestamp, account name, user name, user email, etc.).

ThousandEyes Logs in Dynatrace

Stream ThousandEyes Alerts to Dynatrace as Events

Step 1: Create a Webhook Connector and Operation

1. In ThousandEyes, go to Manage > Integration 2.0 > Integration Templates.

2. Click Custom Webhooks.

3. Enter a Name.

4. Set Target to https://{INSTANCE}.live.dynatrace.com/api/v2/events/ingest (replace {INSTANCE}).

5. Set Auth Type to Custom and add "Authorization": "Api-Token {TOKEN}" (use the token from Prerequisites).

6. Click Save & Assign Operation.

   

   Dynatrace Events Integration
7. Enter an Operation Name.

8. Set Custom Headers to "Content-Type": "application/json".

9. Set Body to the following template:

   {
     "eventType": "CUSTOM_ALERT",
     "title": "ThousandEyes: {{alert.rule.name}}",
     "startTime": {{alert.firstSeen.epochMilli}}{{#if alert.timeCleared}},
     "endTime": {{alert.timeCleared.epochMilli}}{{/if}},
     "timeout": 360,
     "properties": {
       "integration.source": "ThousandEyes",
       "thousandeyes.notification.id": "{{id}}",
       "thousandeyes.alert.id": "{{alert.id}}",
       "thousandeyes.alert.type": "{{alert.rule.alertType.id}}",
       "thousandeyes.account.id": "{{alert.rule.account.id}}",
       "thousandeyes.organization.id": "{{alert.rule.account.organization.id}}",
       "severity": "{{alert.severity.id}}",
       "thousandeyes.test.id": "{{alert.test.id}}",
       "thousandeyes.test.name": "{{alert.test.name}}",
       "thousandeyes.test.type": "{{alert.test.testType}}",
       "thousandeyes.permalink": "https://app.thousandeyes.com/network-app-synthetics/views/?testId={{alert.test.id}}"{{#if alert.test.description}},
       "thousandeyes.test.description": "{{alert.test.description}}"{{/if}}{{/if}}{{#if alert.targets.size}},
       "server.address": "{{#each alert.targets}}{{#if @first}}{{description}}{{/if}}{{/each}}"{{/if}},
       "thousandeyes.rule.id": "{{alert.rule.id}}",
       "thousandeyes.rule.name": "{{alert.rule.name}}",
       "thousandeyes.rule.expression": "{{formatExpression alert.rule.expression}}"{{#if alert.rule.notes}},
       "thousandeyes.rule.notes": "{{alert.rule.notes}}"{{/if}}
     }
   }

10. Click Save Integration to complete the setup.

Dynatrace Webhook Operation

Step 2: Associate the Webhook Integration with an Alert Rule

1. In ThousandEyes, go to Manage > Alert Rules.

2. Click the Alert Rule you want to associate with the webhook integration.

3. Click the Notifications tab.

4. In the Integrations section, add the webhook integration you created.

5. Click Save.

Dynatrace Alert Rule

Step 3: Visualize Alerts in Dynatrace

In Dynatrace Notebooks, run these DQL queries:

• Display the number of active alerts by test and severity:

fetch events
| filter event.type == "CUSTOM_ALERT"
| filter integration.source == "ThousandEyes"
| sort timestamp desc
| dedup thousandeyes.alert.id
| filter event.status == "ACTIVE"
| summarize active_alerts = count(),
    by: { thousandeyes.test.name, severity }
| sort thousandeyes.test.name asc, active_alerts desc

• Display the details of active alerts by test:

fetch events
| filter event.type == "CUSTOM_ALERT"
| filter integration.source == "ThousandEyes"
| sort timestamp desc
| dedup thousandeyes.alert.id
| filter event.status == "ACTIVE"
| fields
    thousandeyes.test.name,
    thousandeyes.rule.name,
    severity,
    server.address,
    thousandeyes.permalink,
    event.start
| sort thousandeyes.test.name asc, severity desc

Dynatrace Alerts Dashboard