> For the complete documentation index, see [llms.txt](https://docs.thousandeyes.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms/dynatrace.md). # Dynatrace Dynatrace is an observability and security platform that uses causal AI to deliver precise insights and automate DevSecOps at scale. This guide shows how to stream ThousandEyes metrics, traces, and logs to Dynatrace using OpenTelemetry. For Dynatrace’s OpenTelemetry overview, see [OpenTelemetry and Dynatrace](https://docs.dynatrace.com/docs/extend-dynatrace/opentelemetry). For the signals ThousandEyes supports, see [Supported Signals](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry#supported-signals). ## Prerequisites * You have [created a ThousandEyes test](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#create-a-test). * You have a Dynatrace API access token with scopes **Ingest metrics**, **Ingest logs**, **Ingest OpenTelemetry traces** and **Ingest events**: 1. In Dynatrace, search for **access token**. 2. Go to **Access Tokens > Generate new token**. 3. Enter a token name and select the scopes above. 4. Click **Generate token**, then copy the token for later use. ## Stream OpenTelemetry Metrics to Dynatrace ### Step 1: Create a Metrics Integration #### Using the ThousandEyes UI 1. In ThousandEyes, go to **Manage > Integration 1.0**. 2. Click **+New Integration** and choose **OpenTelemetry Integration**. 3. Enter a **Name**. 4. Set **Target** to **HTTP**. 5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/metrics` (replace `{INSTANCE}`). 6. For **Preset Configurations**, select **Dynatrace**. 7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"` (use the token from [Prerequisites](#prerequisites)). 8. For **Signal**, select **Metric**. 9. For **Data Model Version**, select **v2**. 10. Select a **test**. For help, see [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions). 11. Click **Save**. #### Using the ThousandEyes API Create the HTTP stream programmatically: ```curl curl -v -XPOST https://api.thousandeyes.com/v7/streams -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{ "type": "opentelemetry", "testMatch": [{ "id": "281474976717575", "domain": "cea" }], "endpointType": "http", "signal": "metric", "streamEndpointUrl": "https://.live.dynatrace.com/api/v2/otlp/v1/metrics", "customHeaders" : { "Authorization": "Api-Token " } }' ``` Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values. ### Step 2: Enable ThousandEyes Resource Attributes in Dynatrace Allow ThousandEyes resource attributes so they appear on metrics: 1. In Dynatrace, go to **Settings** and search for **OpenTelemetry metrics**. 2. Under **Allow list: resource and scope attributes**, add these keys (click **Add Item** for each): * `thousandeyes.stream.id` * `thousandeyes.test.id` * `thousandeyes.test.name` * `thousandeyes.source.agent.id` * `thousandeyes.source.agent.name` 3. For the full list, see [common resource attributes](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/metrics#common-resource-attributes). 4. Click **Save Changes**. ### ThousandEyes Dashboard in Dynatrace 1. Download the dashboard template: [ThousandEyes Dynatrace dashboard template](https://github.com/thousandeyes/thousandeyes-observability-dashboards/blob/main/dynatrace/ThousandEyesDashboard.json). 2. In Dynatrace, go to **Dashboards** and click **Upload**. 3. Open the dashboard to visualize ThousandEyes data. ![ThousandEyes Dashboard in Dynatrace](/files/6NmF6xj5KhwkgCpNSbQA) ## Stream OpenTelemetry Traces to Dynatrace ### Step 1: Create a Traces Integration #### Using the ThousandEyes UI 1. In ThousandEyes, go to **Manage > Integration 1.0**. 2. Click **+New Integration** and select **OpenTelemetry Integration**. 3. Enter a **Name**. 4. Set **Target** to **HTTP**. 5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/traces` (replace `{INSTANCE}`). 6. For **Preset Configurations**, select **Dynatrace**. 7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"`. 8. For **Signal**, select **Trace**. 9. Select a **test**. See [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions) if needed. 10. Click **Save**. #### Using the ThousandEyes API Create the HTTP stream programmatically: ```curl curl -v -XPOST https://api.thousandeyes.com/v7/streams -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{ "type": "opentelemetry", "testMatch": [{ "id": "281474976717575", "domain": "cea" }], "endpointType": "http", "signal": "trace", "streamEndpointUrl": "https://.live.dynatrace.com/api/v2/otlp/v1/traces", "customHeaders" : { "Authorization": "Api-Token " } }' ``` Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values. ### Step 2: Visualize Traces in Dynatrace 1. In Dynatrace, open **Distributed Tracing**. 2. Search for spans from ThousandEyes and apply filters (duration, errors, target URL, etc.). ![ThousandEyes Traces in Dynatrace](/files/ndE3CEj440DamJhKBNAI) ## Stream OpenTelemetry Logs to Dynatrace ### Step 1: Create a Logs Integration #### Using the ThousandEyes UI 1. In ThousandEyes, go to **Manage > Integration 1.0**. 2. Click **+New Integration** and select **OpenTelemetry Integration**. 3. Enter a **Name**. 4. Set **Target** to **HTTP**. 5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/logs` (replace `{INSTANCE}`). 6. For **Preset Configurations**, select **Dynatrace**. 7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"`. 8. For **Signal**, select **Log**. 9. Select a **test**. For details, see [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions). 10. Click **Save**. #### Using the ThousandEyes API Create the HTTP stream programmatically: ```curl curl -v -XPOST https://api.thousandeyes.com/v7/streams -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{ "type": "opentelemetry", "endpointType": "http", "signal": "log", "streamEndpointUrl": "https://.live.dynatrace.com/api/v2/otlp/v1/logs", "customHeaders" : { "Authorization": "Api-Token " } }' ``` Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values. ### Step 2: Visualize Logs in Dynatrace 1. In Dynatrace, open **Logs**. 2. Search for ThousandEyes logs and apply filters (body, timestamp, account name, user name, user email, etc.). ![ThousandEyes Logs in Dynatrace](/files/fmZLFAJbBUywF2ss4JyS) ## Stream ThousandEyes Alerts to Dynatrace as Events ### Step 1: Create a Webhook Connector and Operation 1. In ThousandEyes, go to **Manage > Integration 2.0 > Integration Templates**. 2. Click **Custom Webhooks**. 3. Enter a **Name**. 4. Set **Target** to `https://{INSTANCE}.live.dynatrace.com/api/v2/events/ingest` (replace `{INSTANCE}`). 5. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"` (use the token from [Prerequisites](#prerequisites)). 6. Click **Save & Assign Operation**. ![Dynatrace Events Integration](/files/7ELm87kbTt4Xu7Hepn8e) 7. Enter an **Operation Name**. 8. Set **Custom Headers** to `"Content-Type": "application/json"`. 9. Set **Body** to the following template: ```json { "eventType": "CUSTOM_ALERT", "title": "ThousandEyes: {{alert.rule.name}}", "startTime": {{alert.firstSeen.epochMilli}}{{#if alert.timeCleared}}, "endTime": {{alert.timeCleared.epochMilli}}{{/if}}, "timeout": 360, "properties": { "integration.source": "ThousandEyes", "thousandeyes.notification.id": "{{id}}", "thousandeyes.alert.id": "{{alert.id}}", "thousandeyes.alert.type": "{{alert.rule.alertType.id}}", "thousandeyes.account.id": "{{alert.rule.account.id}}", "thousandeyes.organization.id": "{{alert.rule.account.organization.id}}", "severity": "{{alert.severity.id}}", "thousandeyes.test.id": "{{alert.test.id}}", "thousandeyes.test.name": "{{alert.test.name}}", "thousandeyes.test.type": "{{alert.test.testType}}", "thousandeyes.permalink": "[https://app.thousandeyes.com/network-app-synthetics/views/?testId={{alert.test.id}}](https://app.thousandeyes.com/network-app-synthetics/views/?testId={{alert.test.id}})"{{#if alert.test.description}}, "thousandeyes.test.description": "{{alert.test.description}}"{{/if}}{{#if alert.targets.size}}, "server.address": "{{#each alert.targets}}{{#if @first}}{{description}}{{/if}}{{/each}}"{{/if}}, "thousandeyes.rule.id": "{{alert.rule.id}}", "thousandeyes.rule.name": "{{alert.rule.name}}", "thousandeyes.rule.expression": "{{formatExpression alert.rule.expression}}"{{#if alert.rule.notes}}, "thousandeyes.rule.notes": "{{alert.rule.notes}}"{{/if}} } } ``` 10. Click **Save Integration** to complete the setup. ![Dynatrace Webhook Operation](/files/g1Cb3TnYmrL3ka9bplKH) ### Step 2: Associate the Webhook Integration with an Alert Rule 1. In ThousandEyes, go to **Manage > Alert Rules**. 2. Click the **Alert Rule** you want to associate with the webhook integration. 3. Click the **Notifications** tab. 4. In the **Integrations** section, add the webhook integration you created. 5. Click **Save**. ![Dynatrace Alert Rule](/files/eHxH0pYs44uop7kY4FC8) ### Step 3: Visualize Alerts in Dynatrace In Dynatrace Notebooks, run these DQL queries: * Display the number of active alerts by test and severity: ```sql fetch events | filter event.type == "CUSTOM_ALERT" | filter integration.source == "ThousandEyes" | sort timestamp desc | dedup thousandeyes.alert.id | filter event.status == "ACTIVE" | summarize active_alerts = count(), by: { thousandeyes.test.name, severity } | sort thousandeyes.test.name asc, active_alerts desc ``` * Display the details of active alerts by test: ```sql fetch events | filter event.type == "CUSTOM_ALERT" | filter integration.source == "ThousandEyes" | sort timestamp desc | dedup thousandeyes.alert.id | filter event.status == "ACTIVE" | fields thousandeyes.test.name, thousandeyes.rule.name, severity, server.address, thousandeyes.permalink, event.start | sort thousandeyes.test.name asc, severity desc ``` ![Dynatrace Alerts Dashboard](/files/AgZZo33kOOR2W3J8V7Z0)