> For the complete documentation index, see [llms.txt](https://docs.thousandeyes.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thousandeyes.com/product-documentation/integration-guides/opentelemetry/tags-and-opentelemetry-streams.md).

# Tags and OpenTelemetry Streams

Tags can control which ThousandEyes test data is included in an OpenTelemetry stream. When a tag is eligible for export, ThousandEyes can also add it to exported telemetry as an OpenTelemetry resource attribute.

Use tags when you want a stream to follow a changing group of tests instead of a fixed list of test IDs. For example, you can configure a stream for `service:payments`, and then add or remove tests from that tag as your monitored services change.

For general tag creation, assignment, and management, see [Tags](/product-documentation/tags.md).

## How Tag Matching Works

When you select tags in **Test Data Configuration** or provide `tagMatch` in the API, ThousandEyes streams data from tests assigned to those tags.

Tag matching is dynamic:

* If you assign a Network & App Synthetics test or Endpoint Experience test to a selected tag after the stream is created, that test is automatically included in the stream.
* If you remove a selected tag from a test, that test is removed from the stream only if it no longer matches any selected tag and is not explicitly selected through `testMatch`.
* Tag assignment changes can take a short time to take effect while ThousandEyes updates its internal cache.

When you configure multiple entries in `tagMatch`, ThousandEyes uses the logical operator `OR`. For new API requests, use `tagId`. For backward compatibility, the API can resolve a `key` and `value` pair to a tag ID, but use only one representation in each `tagMatch` collection.

```json
"tagMatch": [
  {"tagId": "c6b78e57-81a2-4c5f-a11a-d96c3c664d55"},
  {"tagId": "0d9141fb-b7be-4e8b-b55a-69f43cc93f19"}
]
```

In this example, the stream includes tests assigned to either tag ID.

Tag matching controls which tests are included in the stream. It does not filter the tag keys or values exported as OpenTelemetry resource attributes. ThousandEyes can export supported tags associated with streamed tests and their agents, including tags that you did not select in **Test Data Configuration** or specify in `tagMatch`.

{% hint style="warning" %}
Tags specified in `tagMatch` must already exist in your account group. Requests that reference tags not found in the account are rejected.
{% endhint %}

## Tag Requirements for Streams

Tags remain valid for general ThousandEyes use even when they cannot be exported through OpenTelemetry. For example, you can still use those tags to organize resources, filter views, or manage assets in the ThousandEyes platform.

To be exported as OpenTelemetry resource attributes, tag keys must meet both of the following requirements:

* The tag key uses characters supported by [OpenTelemetry attribute naming requirements](https://opentelemetry.io/docs/specs/semconv/general/naming/#recommendations-for-application-developers).
* The tag key does not match a ThousandEyes reserved OpenTelemetry attribute name.

### Using Valid OpenTelemetry Attribute Names

Use tag keys that are valid OpenTelemetry attribute names when you plan to use the tags in OpenTelemetry streams.

The following tag pairs use compliant keys:

* `service:payments`
* `environment:prod`
* `team:netops`

Avoid spaces and unsupported characters in tag keys. Uppercase letters are allowed, although OpenTelemetry recommends lowercase attribute names.

Tags with non-compliant keys are not exported as OpenTelemetry resource attributes. In the ThousandEyes platform, non-compliant tags are not selectable for new stream configurations. If a non-compliant tag is already associated with an older stream, it can appear in the configuration so you can remove it.

### Avoiding Reserved OpenTelemetry Attribute Names

Do not create stream tags whose keys match ThousandEyes OpenTelemetry attribute names reserved for platform metadata. ThousandEyes compares tag keys with reserved attribute names case-insensitively.

For example, a tag key such as `thousandeyes.stream.id` conflicts with the reserved `thousandeyes.stream.id` attribute that ThousandEyes uses to identify the stream. The tag remains valid for general ThousandEyes use, but it is not exported as a tag attribute in OpenTelemetry streams.

Reserved attribute names include the fixed attributes documented in the data model:

* [Data Model v2 - Metrics](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/metrics.md#attributes)
* [Data Model v2 - Traces](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/traces.md#attributes)
* [Data Model v2 - Logs](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/logs.md#attributes)

If a tag attribute does not appear in streamed telemetry because its key conflicts with a reserved attribute name, rename the tag key, update the stream configuration if needed, and verify the stream output again.

### Using Tags with Empty Values

Tags with empty values can still be used for stream matching when the tag key is compliant. In the ThousandEyes platform, empty values appear as `(empty)` in muted text so they are not confused with a tag whose literal value is `empty`.

Some observability backends drop empty attribute values. In that case, ThousandEyes still sends the telemetry data, but the backend might not show the empty tag attribute.

## Endpoint Agent Tags and Local Network Data

For Endpoint Experience local network data, the [Stream API](https://developer.cisco.com/docs/thousandeyes/stream/) supports both `endpointAgentLabel` and `endpointAgentTag`.

* `endpointAgentLabel` uses Endpoint Agent label IDs and remains available for backward compatibility.
* `endpointAgentTag` uses Endpoint Agent tag IDs.
* `endpointAgentLabel` and `endpointAgentTag` represent the same selection data. Configure only one in a create or update request.

```json
{
  "type": "opentelemetry",
  "endpointType": "http",
  "streamEndpointUrl": "https://example.com/collector",
  "endpointAgentTag": [
    {"id": "91863f70-e5a6-4a1c-a266-caf02db3607d"}
  ]
}
```

After the stream is created, stream responses include both the label and tag representations for the same local network selection.

## How Tags Appear in OpenTelemetry Data

ThousandEyes converts supported tags associated with tests and agents into OpenTelemetry resource attributes. The tag key becomes the attribute name.

In data model v1, the tag value is represented as `array_value`, with an array of `string_value` elements, even when the key has only one value.

In data model v2, a tag key with one value is represented as `string_value`. If the same tag key has multiple values, the value is represented as `array_value`, with an array of `string_value` elements.

Data model v1 single-value example:

```json
{
  "key": "service",
  "value": {
    "array_value": {
      "values": [
        {"string_value": "payments"}
      ]
    }
  }
}
```

Data model v2 single-value example:

```json
{
  "key": "service",
  "value": {
    "string_value": "payments"
  }
}
```

If the same tag key has multiple values, the array includes each unique value:

```json
{
  "key": "service",
  "value": {
    "array_value": {
      "values": [
        {"string_value": "payments"},
        {"string_value": "checkout"}
      ]
    }
  }
}
```

For more examples, see [Data Model v1](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v1.md#tags-as-attributes), [Data Model v2 - Metrics](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/metrics.md#tags-as-attributes), and [Data Model v2 - Traces](/product-documentation/integration-guides/opentelemetry/data-model/data-model-v2/traces.md#tags-as-attributes).

## Troubleshooting Tags in OpenTelemetry Streams

Use the following checks when tags do not appear or do not behave as expected in an OpenTelemetry stream:

| Issue                                              | Checks                                                                                                                                                                                                                                                                                        |
| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A tag does not appear in the stream tag selector.  | Confirm that the tag key is a valid OpenTelemetry attribute name. Non-compliant tags that are already associated with an older stream can appear so you can remove them. Tags with reserved OpenTelemetry attribute names can remain selectable, but they are not exported as tag attributes. |
| A tag does not appear in exported telemetry.       | Confirm that the tag key is compliant and does not use a reserved attribute name. Allow time for tag assignment changes to propagate through the internal cache. If the tag value is empty, confirm whether your backend drops empty attribute values.                                        |
| A backend does not show a multi-value tag.         | Confirm whether the backend supports string arrays for OpenTelemetry attribute values. If the backend does not support string arrays, it might drop the tag attribute or the data point.                                                                                                      |
| An API request that uses `tagMatch` is rejected.   | Confirm that the tag ID exists in the account group and that the request uses the current `/v7/streams` schema.                                                                                                                                                                               |
| An API request for local network data is rejected. | Configure either `endpointAgentLabel` or `endpointAgentTag`, not both. Confirm that the label ID or tag ID exists and that the Endpoint Agent tag key is compliant.                                                                                                                           |