Tags and OpenTelemetry Streams
Tags can control which ThousandEyes test data is included in an OpenTelemetry stream. When a tag is eligible for export, ThousandEyes can also add it to exported telemetry as an OpenTelemetry resource attribute.
Use tags when you want a stream to follow a changing group of tests instead of a fixed list of test IDs. For example, you can configure a stream for service:payments, and then add or remove tests from that tag as your monitored services change.
For general tag creation, assignment, and management, see Tags.
How Tag Matching Works
When you select tags in Test Data Configuration or provide tagMatch in the API, ThousandEyes streams data from tests assigned to those tags.
Tag matching is dynamic:
If you assign a Network & App Synthetics test or Endpoint Experience test to a selected tag after the stream is created, that test is automatically included in the stream.
If you remove a selected tag from a test, that test is removed from the stream only if it no longer matches any selected tag and is not explicitly selected through
testMatch.If you edit a selected tag's key or value, the stream continues to select that tag by its stable ID. You do not need to update or recreate the integration.
When you configure multiple entries in tagMatch, ThousandEyes uses the logical operator OR. For new API requests, use tagId. For backward compatibility, the API can resolve a key and value pair to a tag ID, but use only one representation in each tagMatch collection.
"tagMatch": [
{"tagId": "c6b78e57-81a2-4c5f-a11a-d96c3c664d55"},
{"tagId": "0d9141fb-b7be-4e8b-b55a-69f43cc93f19"}
]In this example, the stream includes tests assigned to either tag ID.
Tag matching controls which tests are included in the stream. It does not filter the tag keys or values exported as OpenTelemetry resource attributes. ThousandEyes can export supported tags associated with streamed tests and their agents, including tags that you did not select in Test Data Configuration or specify in tagMatch.
Editing or Replacing a Selected Tag
A stream follows the selected tag's ID, which stays the same when you edit the tag's key or value. For example, if you rename a selected tag from service:payments to application:checkout, tests assigned to that tag remain selected. After the change propagates, exported telemetry uses application:checkout, subject to the tag requirements for streams.
Deleting a tag and creating another tag with the same key and value creates a new tag ID. To stream tests assigned to the replacement tag, select that new tag in the integration.
Allowing Changes to Propagate
Tag assignment and metadata changes take effect asynchronously. Allow several minutes for changes to appear in streamed telemetry. During this transition, telemetry can contain cached tag attributes, including both the previous and updated tag keys or values.
Verify fresh telemetry at the destination after changing a tag. You do not need to edit the integration to apply a tag rename.
Tag Requirements for Streams
Tags remain valid for general ThousandEyes use even when they cannot be exported through OpenTelemetry. For example, you can still use those tags to organize resources, filter views, or manage assets in the ThousandEyes platform.
To be exported as OpenTelemetry resource attributes, tag keys must meet both of the following requirements:
The tag key uses characters supported by OpenTelemetry attribute naming requirements.
The tag key does not match a ThousandEyes reserved OpenTelemetry attribute name.
Using Valid OpenTelemetry Attribute Names
Use tag keys that are valid OpenTelemetry attribute names when you plan to use the tags in OpenTelemetry streams.
The following tag pairs use compliant keys:
service:paymentsenvironment:prodteam:netops
Avoid spaces and unsupported characters in tag keys. Uppercase letters are allowed, although OpenTelemetry recommends lowercase attribute names.
Tags with non-compliant keys are not exported as OpenTelemetry resource attributes. In the ThousandEyes platform, non-compliant tags are not selectable for new stream configurations. If a non-compliant tag is already associated with an older stream, it can appear in the configuration so you can remove it.
Avoiding Reserved OpenTelemetry Attribute Names
Do not create stream tags whose keys match ThousandEyes OpenTelemetry attribute names reserved for platform metadata. ThousandEyes compares tag keys with reserved attribute names case-insensitively.
For example, a tag key such as thousandeyes.stream.id conflicts with the reserved thousandeyes.stream.id attribute that ThousandEyes uses to identify the stream. The tag remains valid for general ThousandEyes use, but it is not exported as a tag attribute in OpenTelemetry streams.
Reserved attribute names include the fixed attributes documented in the data model:
If a tag attribute does not appear in streamed telemetry because its key conflicts with a reserved attribute name, rename the tag key and allow the change to propagate. Then verify the stream output again. The tag remains selected in the integration because its ID has not changed.
Using Tags with Empty Values
Tags with empty values can still be used for stream matching when the tag key is compliant. In the ThousandEyes platform, empty values appear as (empty) in muted text so they are not confused with a tag whose literal value is empty.
Some observability backends drop empty attribute values. In that case, ThousandEyes still sends the telemetry data, but the backend might not show the empty tag attribute.
Endpoint Agent Tags and Local Network Data
For Endpoint Experience local network data, the Stream API supports both endpointAgentLabel and endpointAgentTag.
endpointAgentLabeluses Endpoint Agent label IDs and remains available for backward compatibility.endpointAgentTaguses Endpoint Agent tag IDs.endpointAgentLabelandendpointAgentTagrepresent the same selection data. Configure only one in a create or update request.
After the stream is created, stream responses include both the label and tag representations for the same local network selection.
How Tags Appear in OpenTelemetry Data
ThousandEyes converts supported tags associated with tests and agents into OpenTelemetry resource attributes. The tag key becomes the attribute name.
In data model v1, the tag value is represented as array_value, with an array of string_value elements, even when the key has only one value.
In data model v2, a tag key with one value is represented as string_value. If the same tag key has multiple values, the value is represented as array_value, with an array of string_value elements.
Data model v1 single-value example:
Data model v2 single-value example:
If the same tag key has multiple values, the array includes each unique value:
For more examples, see Data Model v1, Data Model v2 - Metrics, and Data Model v2 - Traces.
Troubleshooting Tags in OpenTelemetry Streams
Use the following checks when tags do not appear or do not behave as expected in an OpenTelemetry stream:
A tag does not appear in the stream tag selector.
Confirm that the tag key is a valid OpenTelemetry attribute name. Non-compliant tags that are already associated with an older stream can appear so you can remove them. Tags with reserved OpenTelemetry attribute names can remain selectable, but they are not exported as tag attributes.
A tag does not appear in exported telemetry.
Confirm that the tag key is compliant and does not use a reserved attribute name. Allow time for tag assignment and metadata changes to propagate. If the tag value is empty, confirm whether your backend drops empty attribute values.
Telemetry contains an old tag key or value after a rename.
Allow several minutes for metadata changes to propagate, then check fresh telemetry at the destination. Previous and updated tag attributes can appear together during this transition. You do not need to edit the integration to apply the rename.
A recreated tag does not select the expected tests.
Deleting and recreating a tag gives it a new ID, even if its key and value are the same. Select the replacement tag in the integration.
A backend does not show a multi-value tag.
Confirm whether the backend supports string arrays for OpenTelemetry attribute values. If the backend does not support string arrays, it might drop the tag attribute or the data point.
An API request that uses tagMatch is rejected.
Confirm that the tag ID exists in the account group and that the request uses the current /v7/streams schema.
An API request for local network data is rejected.
Configure either endpointAgentLabel or endpointAgentTag, not both. Confirm that the label ID or tag ID exists and that the Endpoint Agent tag key is compliant.
Last updated