# Dynatrace

Dynatrace is an observability and security platform that uses causal AI to deliver precise insights and automate DevSecOps at scale. This guide shows how to stream ThousandEyes metrics, traces, and logs to Dynatrace using OpenTelemetry.

For Dynatrace’s OpenTelemetry overview, see [OpenTelemetry and Dynatrace](https://docs.dynatrace.com/docs/extend-dynatrace/opentelemetry). For the signals ThousandEyes supports, see [Supported Signals](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry#supported-signals).

## Prerequisites

* You have [created a ThousandEyes test](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#create-a-test).
* You have a Dynatrace API access token with scopes **Ingest metrics**, **Ingest logs**, **Ingest OpenTelemetry traces** and **Ingest events**:
  1. In Dynatrace, search for **access token**.
  2. Go to **Access Tokens > Generate new token**.
  3. Enter a token name and select the scopes above.
  4. Click **Generate token**, then copy the token for later use.

## Stream OpenTelemetry Metrics to Dynatrace

### Step 1: Create a Metrics Integration

#### Using the ThousandEyes UI

1. In ThousandEyes, go to **Manage > Integration 1.0**.
2. Click **+New Integration** and choose **OpenTelemetry Integration**.
3. Enter a **Name**.
4. Set **Target** to **HTTP**.
5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/metrics` (replace `{INSTANCE}`).
6. For **Preset Configurations**, select **Dynatrace**.
7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"` (use the token from [Prerequisites](#prerequisites)).
8. For **Signal**, select **Metric**.
9. For **Data Model Version**, select **v2**.
10. Select a **test**. For help, see [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions).
11. Click **Save**.

#### Using the ThousandEyes API

Create the HTTP stream programmatically:

```curl
 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "signal": "metric",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/metrics",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'
```

Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values.

### Step 2: Enable ThousandEyes Resource Attributes in Dynatrace

Allow ThousandEyes resource attributes so they appear on metrics:

1. In Dynatrace, go to **Settings** and search for **OpenTelemetry metrics**.
2. Under **Allow list: resource and scope attributes**, add these keys (click **Add Item** for each):
   * `thousandeyes.stream.id`
   * `thousandeyes.test.id`
   * `thousandeyes.test.name`
   * `thousandeyes.source.agent.id`
   * `thousandeyes.source.agent.name`
3. For the full list, see [common resource attributes](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/metrics#common-resource-attributes).
4. Click **Save Changes**.

### ThousandEyes Dashboard in Dynatrace

1. Download the dashboard template: [ThousandEyes Dynatrace dashboard template](https://github.com/thousandeyes/thousandeyes-observability-dashboards/blob/main/dynatrace/ThousandEyesDashboard.json).
2. In Dynatrace, go to **Dashboards** and click **Upload**.
3. Open the dashboard to visualize ThousandEyes data.

![ThousandEyes Dashboard in Dynatrace](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-9f4d30ce8066bd76fb7ce2e9d4c02f6ac6c12626%2Fdynatrace-thousandeyes-dashboard.png?alt=media)

## Stream OpenTelemetry Traces to Dynatrace

### Step 1: Create a Traces Integration

#### Using the ThousandEyes UI

1. In ThousandEyes, go to **Manage > Integration 1.0**.
2. Click **+New Integration** and select **OpenTelemetry Integration**.
3. Enter a **Name**.
4. Set **Target** to **HTTP**.
5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/traces` (replace `{INSTANCE}`).
6. For **Preset Configurations**, select **Dynatrace**.
7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"`.
8. For **Signal**, select **Trace**.
9. Select a **test**. See [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions) if needed.
10. Click **Save**.

#### Using the ThousandEyes API

Create the HTTP stream programmatically:

```curl
 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "testMatch": [{
    "id": "281474976717575",
    "domain": "cea"
  }],
  "endpointType": "http",
  "signal": "trace",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/traces",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'
```

Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values.

### Step 2: Visualize Traces in Dynatrace

1. In Dynatrace, open **Distributed Tracing**.
2. Search for spans from ThousandEyes and apply filters (duration, errors, target URL, etc.).

![ThousandEyes Traces in Dynatrace](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-20fd33e49455e55fd1a10fdcb3fdec220dff6a6e%2Fdynatrace-trace.png?alt=media)

## Stream OpenTelemetry Logs to Dynatrace

### Step 1: Create a Logs Integration

#### Using the ThousandEyes UI

1. In ThousandEyes, go to **Manage > Integration 1.0**.
2. Click **+New Integration** and select **OpenTelemetry Integration**.
3. Enter a **Name**.
4. Set **Target** to **HTTP**.
5. Set the **Endpoint URL** to `https://{INSTANCE}.live.dynatrace.com/api/v2/otlp/v1/logs` (replace `{INSTANCE}`).
6. For **Preset Configurations**, select **Dynatrace**.
7. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"`.
8. For **Signal**, select **Log**.
9. Select a **test**. For details, see [General Setup Instructions](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms#general-setup-instructions).
10. Click **Save**.

#### Using the ThousandEyes API

Create the HTTP stream programmatically:

```curl
 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "endpointType": "http",
  "signal": "log",
  "streamEndpointUrl": "https://<instance>.live.dynatrace.com/api/v2/otlp/v1/logs",
  "customHeaders" : {
    "Authorization": "Api-Token <TOKEN>"
  }
}'
```

Replace `{TOKEN}` and `{INSTANCE}` with your Dynatrace values.

### Step 2: Visualize Logs in Dynatrace

1. In Dynatrace, open **Logs**.
2. Search for ThousandEyes logs and apply filters (body, timestamp, account name, user name, user email, etc.).

![ThousandEyes Logs in Dynatrace](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-35c1f35e5c951edec7954686e609365da72326af%2Fdynatrace-logs.png?alt=media)

## Stream ThousandEyes Alerts to Dynatrace as Events

### Step 1: Create a Webhook Connector and Operation

1. In ThousandEyes, go to **Manage > Integration 2.0 > Integration Templates**.
2. Click **Custom Webhooks**.
3. Enter a **Name**.
4. Set **Target** to `https://{INSTANCE}.live.dynatrace.com/api/v2/events/ingest` (replace `{INSTANCE}`).
5. Set **Auth Type** to **Custom** and add `"Authorization": "Api-Token {TOKEN}"` (use the token from [Prerequisites](#prerequisites)).
6. Click **Save & Assign Operation**.

   ![Dynatrace Events Integration](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-55ecaeb0bcf918ebb3f59db5d889020a6cde3716%2Fdynatrace-webhook-connector.png?alt=media)
7. Enter an **Operation Name**.
8. Set **Custom Headers** to `"Content-Type": "application/json"`.
9. Set **Body** to the following template:

   ```json
   {
     "eventType": "CUSTOM_ALERT",
     "title": "ThousandEyes: {{alert.rule.name}}",
     "startTime": {{alert.firstSeen.epochMilli}}{{#if alert.timeCleared}},
     "endTime": {{alert.timeCleared.epochMilli}}{{/if}},
     "timeout": 360,
     "properties": {
       "integration.source": "ThousandEyes",
       "thousandeyes.notification.id": "{{id}}",
       "thousandeyes.alert.id": "{{alert.id}}",
       "thousandeyes.alert.type": "{{alert.rule.alertType.id}}",
       "thousandeyes.account.id": "{{alert.rule.account.id}}",
       "thousandeyes.organization.id": "{{alert.rule.account.organization.id}}",
       "severity": "{{alert.severity.id}}",
       "thousandeyes.test.id": "{{alert.test.id}}",
       "thousandeyes.test.name": "{{alert.test.name}}",
       "thousandeyes.test.type": "{{alert.test.testType}}",
       "thousandeyes.permalink": "https://app.thousandeyes.com/network-app-synthetics/views/?testId={{alert.test.id}}"{{#if alert.test.description}},
       "thousandeyes.test.description": "{{alert.test.description}}"{{/if}}{{/if}}{{#if alert.targets.size}},
       "server.address": "{{#each alert.targets}}{{#if @first}}{{description}}{{/if}}{{/each}}"{{/if}},
       "thousandeyes.rule.id": "{{alert.rule.id}}",
       "thousandeyes.rule.name": "{{alert.rule.name}}",
       "thousandeyes.rule.expression": "{{formatExpression alert.rule.expression}}"{{#if alert.rule.notes}},
       "thousandeyes.rule.notes": "{{alert.rule.notes}}"{{/if}}
     }
   }
   ```
10. Click **Save Integration** to complete the setup.

![Dynatrace Webhook Operation](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-5889261ea09d6b621ed98b2baa2511be21782da8%2Fdynatrace-webhook-operator.png?alt=media)

### Step 2: Associate the Webhook Integration with an Alert Rule

1. In ThousandEyes, go to **Manage > Alert Rules**.
2. Click the **Alert Rule** you want to associate with the webhook integration.
3. Click the **Notifications** tab.
4. In the **Integrations** section, add the webhook integration you created.
5. Click **Save**.

![Dynatrace Alert Rule](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-f420f19f3632f0c351a6e84d5918654183c33cdc%2Fdynatrace-webhook-alertrule-association.png?alt=media)

### Step 3: Visualize Alerts in Dynatrace

In Dynatrace Notebooks, run these DQL queries:

* Display the number of active alerts by test and severity:

```sql
fetch events
| filter event.type == "CUSTOM_ALERT"
| filter integration.source == "ThousandEyes"
| sort timestamp desc
| dedup thousandeyes.alert.id
| filter event.status == "ACTIVE"
| summarize active_alerts = count(),
    by: { thousandeyes.test.name, severity }
| sort thousandeyes.test.name asc, active_alerts desc
```

* Display the details of active alerts by test:

```sql
fetch events
| filter event.type == "CUSTOM_ALERT"
| filter integration.source == "ThousandEyes"
| sort timestamp desc
| dedup thousandeyes.alert.id
| filter event.status == "ACTIVE"
| fields
    thousandeyes.test.name,
    thousandeyes.rule.name,
    severity,
    server.address,
    thousandeyes.permalink,
    event.start
| sort thousandeyes.test.name asc, severity desc
```

![Dynatrace Alerts Dashboard](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-d44c6767f498adeecac3674ad3f4f91c8431a5a6%2Fdynatrace-alerts.png?alt=media)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/observability-platforms/dynatrace.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.