Changelog

2021-11-16
New Endpoint Agent View
ThousandEyes has added a new view to simplify Endpoint Agent troubleshooting. This view gives you the ability to search for a specific agent and view all the information related to its scheduled tests, local networks, and browser session data.
For full documentation of this new view, see Agent Views.
Initially, this view is an opt-in feature. To access it, contact ThousandEyes Customer Engineering.
Bulk Edit for Tests
You can now edit tests assigned to multiple Enterprise Agents simultaneously. In Cloud & Enterprise Agents > Agent Settings, select multiple agents; then use the new Edit Tests option to edit the set of tests assigned to these agents.
Updated Dashboards and Reports Caching Mechanism
The caching mechanism for dashboards and reports has been updated to improve end-to-end performance. As part of this change, the cache time has been increased to 30 minutes. This will result in a delay in updated dashboards and reports when you change the ownership of a test to a different account or organization ID.
For example, if the test ownership moves from Account Group A to Account Group B, this change will only be reflected in your dashboard or report when the cache is updated - meaning there is a maximum potential delay of 30 minutes.
New Version of the Enterprise Agent Docker Image
A new version of the Enterprise Agent Docker image (0.14-1635966276) is released that includes important changes for future versions of BrowserBot. This version allows you to inspect your Docker Enterprise Agents' image versions via the Agents Settings page and plan an upgrade as necessary.
Change to the Agent ID Data Type in Web Transaction Scripts
The JavaScript data type of the agentId field, provided by the agent module, will change from Number to String in BrowserBot version 1.172.0 onwards. This change allows the Transaction scripting sandbox to accommodate agent IDs that are too large for the JavaScript Number type to represent.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Chengdu, China (Alibaba cn-chengdu)
Dublin, Ireland (IPv6) (Eircom)
Guangzhou, China (Alibaba cn-guangzhou)
Heyuan, China (Alibaba cn-heyuan)
Manila, Philippines (Alibaba ap-southeast-6)
Paraíba, Brazil (IPv6)
Rio de Janeiro, Brazil (IPv6)
Taipei, Taiwan 2 (IPv6)
Ulanqab, China (Alibaba cn-wulanchabu)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Deletion of Transaction Data from the ThousandEyes Platform
As part of completing the end-of-life for Transaction (Classic) tests, any remaining Transaction (Classic) data will be deleted from the platform on February 15, 2021.
Deprecated Features
BrowserBot is no longer supported for Enterprise Agents deployed on Amazon Linux. New Enterprise Agent installations using install_thousandeyes.sh will not be able to install BrowserBot on Amazon Linux. Existing Enterprise Agents will not be impacted immediately, though they may fail to upgrade to future versions of BrowserBot.
The Keep browser cache between tests agent setting has been deprecated. The setting will be visible, toggle-able, and respected by Enterprise Agents with the current BrowserBot version, but the setting will not be available in future releases. When the next version of BrowserBot is released, the setting will be removed from the Agent Settings page, and BrowserBot will reset the browser cache between all test rounds unconditionally.
2021-11-16 Bug Fixes
An issue was found where /v6/endpoint-agents API calls did not return the createdTime field. This has now been fixed.
An issue was found on macOS where TCP tests that run over certain VPN clients could cause the network interfaces to stop functioning. This issue has been reported to Apple, and the issue trigger is now avoided by ThousandEyes.
Previously, path trace alerts were triggered incorrectly for the following scenarios:
The alert condition was to trigger in event of path trace being incomplete. However, the alerts were triggered for tests with path trace completed.
The IP addresses provided were in the pre-defined range, yet the alerts were triggered.
These have now been fixed.
2021-11-09
Announcing Internet Insights Application Outages
ThousandEyes announced Application Outages availability on November 9th. Application Outages expands Internet Insights – the first ever collectively powered global view of Internet health – to give you visibility into outages affecting the top SaaS applications you rely on.
Application Outages brings the following new capabilities to ThousandEyes Internet Insights:
Overview screen: See both Application Outages and Network Outages on a color-coded map display.
Views screen:
Select Network Outages or Application Outages
Cross-layer visualization shows you what type of outage is occuring: network, application, or both.
Error Types tell you the type of errors encountered in the outage.
Catalog Settings screen: New SaaS type packages add more than 75 top application providers.
Application Outages data is available across the ThousandEyes platform: within snapshots, dashboard and reports, as well as outage alerts and alerts API. All of these capabilities leverage ThousandEyes' collective network intelligence using real data, not sentiment.
For more information, see the Announcing Internet Insights Application Outages blog post, or review the documentation here: Application Outages Documentation.
2021-11-08
CentOS 8.x End of Installation Support
The CentOS 8.1, 8.2, 8.3, and 8.4 operating systems have now transitioned into the End of Installation Support lifecycle phase. New installations of Enterprise Agents on these operating systems will no longer be supported.
Support for existing Enterprise Agents running on CentOS 8.1 systems will end on November 30th, 2021, while support for agents running on CentOS 8.2, 8.3, and 8.4 systems will end on December 31st, 2021.
After those dates, the operating systems will transition into the End of Support phase, in preparation for End of Life.
For ThousandEyes support lifecycle details, see Operating System Support Lifecycle.
If you need assistance migrating to a supported operating system, contact ThousandEyes Customer Engineering.
2021-10-28
New Webhook Timezone Variables
Two new variables have been added for ThousandEyes webhook payloads to replace the existing variables dateStart and dateEnd. Previously, the alert dateStart and dateEnd variables were based on the organization's timezone, but did not include a timezone identifier:
1
"dateStart": "2021-10-19 17:42:00",
Copied!
The new variables, dateStartZoned and dateEndZoned, include the time zone associated with the time of the alert trigger, as well as the alert clear.
Example for UTC timezone:
1
"dateStartZoned": "2021-10-19 17:42:00 UTC",
2
"dateEndZoned": "2021-10-19 17:47:39 UTC",
Copied!
The timezone is shown explicitly, based on your organization’s time zone setting. If your organization’s timezone is Eastern Daylight Time, the payload looks like this:
1
"dateStartZoned": "2021-10-19 15:20:00 EDT",
Copied!
In order to preserve current integrations, the existing dateStart and dateEnd variables will be maintained. However, these variables will eventually be deprecated. We ask that every customer begin to use the new variables moving forward.
Deprecated Features
Endpoint Agent Windows 7 Support
ThousandEyes no longer supports Windows 7 for the Endpoint Agent. For more information on supported operating systems, see Endpoint Agent System Requirements.
2021-10-28 Bug Fixes
Fixed a bug where instant tests were broken for long account names. This had been caused by base64 encoding, which by default was adding a newline character for names longer than 80 characters.
2021-10-22
Notifications for Device Data Alerts and Device Status Alerts
We have moved our device alert notifications to our standard notifier platform. This has resulted in the changes described below.
We recommend that you review the changes to see if any of your configured notifications are affected.
Device Data Alert Notifications
Webhooks
The webhook payload now includes both alert and deviceAlert as JSON keys. Moving forward, you should migrate to using just the alert JSON key, as the deviceAlert JSON key is now deprecated and will eventually be removed.
A permalink to the Device Topology page is now included in the webhook.
Date fields are timezone-converted dates instead of Epoch seconds timestamps.
Slack
Device Name is no longer in the title of the Slack message.
Device Name is added as a new field for the body of the Slack message.
Email
The column header for interfaces is now Interface instead of Interface name.
PagerDuty
Most custom detail field names are now in plain English rather than raw variable names.
Alert Rule and Rule Expression are now merged into one field instead of separate fields.
Type, No. of Interfaces, and Permalink have been added as new fields.
Interface Name has been removed from the incident title.
Device Status Alert Notifications
Webhooks
The webhook payload now includes both alert and deviceAlert as JSON keys. Moving forward, you should migrate to using just the alert JSON key, as the deviceAlert JSON key is now deprecated and will eventually be removed.
A permalink to the Device Topology page is now included.
Date fields are timezone-converted dates instead of Epoch seconds timestamps.
A dateEnd field has been added.
Slack
Rule expression is combined with rule name as the value for Alert Rule.
Email
The original email format contained links to devices for which a particular notification rule was triggered. The new email now gives a summary of all the devices that were triggered, with details such as alert ID, rule, and start date.
The new email includes the summary header and adds headers and brief information for each device.
PagerDuty
Alert Rule and Rule Expression are now merged into the Alert Rule field.
Alert type and permalink have been added as new fields.
Most custom detail field names are now in plain English rather than raw variable names
The header and summary in PagerDuty now only include the rule name.
2021-10-21
Path Trace Alert Display Change
As a result of migrating path trace metrics to our new alerter architecture, the display of alert data has changed:
When you set up an alert rule on any indexed hop condition from the destination direction, the hop index now displays the alert value from the source, rather than the destination.
For example:
Old alerter: Hop #1 from destination: (DSCP: Best Effort (DSCP 0))
New alerter: Path Trace #1: (Hop #18: (DSCP: Best Effort (DSCP 0)))
All alerting conditions with "No Hops" or "MPLS Label is empty" expressions will display the triggering condition as "No Hops: Path Violates Alerting Condition" instead of the specific rule expression.
For example:
Old alerter: Path Trace #1 - No Hop meets: rDNS in ["thousandeyes.com"]
New alerter: Path Trace #1: (No Hops: Path Violates Alerting Condition)
2021-10-15
Cisco Nexus Switch Support for Enterprise Agents
ThousandEyes now supports installing Enterprise Agents on Cisco Nexus 9000 series switches.
For installation instructions, see Installing Enterprise Agents on Cisco Nexus 9000 Series Switches.
For more information on the system requirements, see the Support Matrix.
Snapshot Scaling and Performance Improvements
As part of our performance and scaling improvements, Endpoint Agent snapshot data will be migrated to a new data platform. Whilst our migration project is in progress, snapshots that include Local Network data will be limited to a maximum of 12 hour windows.
Minor Enhancements
In the Endpoint Agents Instant Test view, the current view in the left hand navigation bar is now described as Instant Tests rather than as Scheduled Tests. This is now also the case when saving snapshots of instant tests.
Deprecated Features
The Organization filter in reports and dashboards widgets has been deprecated. This change only affects widgets that are configured with an Endpoint Agents > Browser Sessions metric that also filter by organizations. The filter will automatically be migrated to a Network filter in all affected widgets. No action is required.
2021-10-15 Bug Fixes
Previously, some macOS edge cases could result in the Endpoint Agent name not reflecting the correct hostname of the device. We have improved the way the Endpoint Agent obtains the hostname to eliminate these edge cases.
An issue was found where Endpoint Agent instant tests were breaking for accounts with long account names. This was caused by base64 encoding by default adding a new line character for names longer than 80 characters, and has been resolved.
The Endpoint Agent now shows traffic that has been configured in Zscaler Client Connect to bypass the VPN tunnel as going directly to the Internet in the path visualisation.
The Endpoint Agent now shows overlay network nodes for network tests using ICMP, when ZScaler Internet Access is used with tunnel version 2.0.
2021-10-11
Alerting Platform Updates
Over the next several weeks, our alerting platform will be updated to change the way we treat Enterprise Agents that are not reporting data.
Previously, a triggered alert would not clear, even if all violating/triggering agents had cleared the condition, if m or more agents were not reporting data (where m = the number of agents required to trigger an alert). For example, if an alert rule was set up to alert if two agents were violating a condition, the triggered alert would not clear if two or more agents did not report data, even if the previously violating agents were now cleared.
With the coming change, Enterprise Agents that do not send data to our system will now be treated as not violating the alert condition. Triggered alerts will now clear when violating/triggering agents have cleared the condition, even if m or more agents are not reporting data (where m = the number of agents required to trigger an alert). For example, if an alert rule was set up to alert if two agents are violating a condition, the alert will clear if less than two agents are violating the condition, even if two or more agents are not reporting data.
2021-10-07
Agent Alert Notifications Update
We have made several updates to our agent alert notifications in order to match our existing notification patterns for other alert types.
Email Notifications
Agent alert email notifications have been updated to match our other notification types. They now include:
Alert ID
Alert Rule
Agent Name
Hostname
IP Address
Start Date
Notes
Slack Notifications
Slack notifications will now include:
Agent Alert triggered for agent
Alert ID
Alert Rule
Agent ID
Agent Name
Hostname
IP Address
PagerDuty Notifications
PagerDuty notifications will now include:
Agent ID
Agent Name
Alert ID
Alert Rule
Hostname
IP Address
Rule ID
Start Date
Type
Webhook Notifications
Webhook notification will now include:
agentId
hostname
dateStart
ipAddress
ruleName
agentName
alertId
ruleExpression
dateEnd
ruleId
type
ruleAid
eventType
The following changes have also been made:
AGENT_ALERT_NOTIFICATION_TRIGGER has been changed to ALERT_NOTIFICATION_TRIGGER.
AGENT_ALERT_NOTIFICATION_CLEAR has been changed to ALERT_NOTIFICATION_CLEAR.
2021-09-29
Endpoint Agent Notifications Update
We have moved Endpoint Agent notifications to our new backend notifier. This results in several changes to notifications, including:
Webhooks
Added ruleName, violationCount, ruleAID, and type as new fields.
Changed browserSessionAlert to alert.
Added a permalink, but maintained the soon to be deprecated link field.
Most strings are now in human readable format.
These changes should not require customers to update their webhooks. However, you should review your downstream systems to ensure there is no impact.
Email
Changed from Browser Session (Application) to Browser Session Application.
The order of Start Date and Number of Sites Visited is flipped.
Changed from Scope to No. of Agents.
Pager Duty:
Added No. of Visited Sites.
Changed from AlertID to Alert ID.
Removed the dedicated field for ruleExpression and instead have the human readable form in the Alert Rule field.
Slack
Removed the dedicated field for ruleExpression and instead have the human readable form in the Alert Rule field.
Added Target and Test Names as fields.
Changed from Scope to No. of Agents.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Glasgow, Scotland (IPv6)
Louisville, KY, USA (IPv6)
New York, NY (IPv6)
New York, NY (Cogent) (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
2021-09-29 Bug Fixes
Previously, if an Endpoint Agent user was connected via a remote desktop, the temporary directory would change. This caused requests for the temporary directory location to fail, as system processes were still using the previously configured location. This has been resolved.
Previously, an issue was found where grouping by devices in a report or dashboard widget was not correctly populating individual device names. This has been resolved.
2021-09-16
Metric Calculation Improvements for Reports and Dashboards
We have improved the way we calculate the Availability metric for Device data in reports and dashboards. Availability is now calculated based on the number of interfaces that are up and running divided by the number of interfaces that the admin set as up. This is a slight change to the previous calculation, which also included in rare instances interfaces that were up and running even if they were set as down by the admin.
Additionally, customers can now query Device Availability by median, max, min, Nth percentile and standard deviation.
Changes to Update Email Limit
Effective immediately, ThousandEyes will be making an update to our profile logic so that requests to update the email address of a provisioned user will only be accepted a maximum of once per hour. Once that limit is reached, subsequent requests to update email will then be blocked by the system until the hour period has passed. This change will patch a security vulnerability. We do not anticipate this change to affect customer account access outside of the above limitation.
2021-09-14
Changes to the te-browserbot Package's Dependencies on RHEL 7 and Oracle Linux 7
In the coming months, as part of an architectural change to BrowserBot, ThousandEyes will add te-podman as a dependency of the te-browserbot package. This new package depends on some packages available in the Red Hat Enterprise Linux 7 Server - Extras and Oracle Linux Addons repositories for RHEL 7 and Oracle Linux 7, respectively.
Users with existing agents running on RHEL 7 should enable the rhel-7-server-extras-rpms repository. Users with existing agents running on Oracle Linux 7 should enable the ol7_addons repository. Otherwise, BrowserBot will not be able to upgrade to this forthcoming version.
On RHEL 7, you can enable this required repository by executing subscription-manager repos --enable=rhel-7-server-extras-rpms. On Oracle Linux 7, execute yum-config-manager -q -y --enable ol7_addons.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Madrid, Spain (IPv6)
Mumbai, India (JIO)
Mumbai, India (JIO) (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
2021-09-14 Bug Fixes
An issue was found in the Stacked Area graph widget for reports and dashboards when displaying a Page Load metric. The calculation for the metric was adding DOM load time to the page load time, which is incorrect, as DOM load time is a sub-component of page load time. This has now been resolved and the correct value is displayed.
2021-09-01
Changes to Webapp Session Timeout Duration
In the coming weeks, ThousandEyes will change the default session timeout duration from indefinite to 30 minutes. This change will patch a security vulnerability, as well as better align the timeout duration with industry best practices.
As part of this change, the keep_session_alive_on_auto-update permission will be automatically removed from all default built-in roles. This change could affect customers that continuously display ThousandEyes dashboards (for example, in Network Operations Centers).
If you would like to preserve the keep_session_alive_on_auto-update permission for certain users within your organization, we recommend that you create a custom role with this permission and assign users in the appropriate account group to that role.
For more information on creating custom roles, see Managing Roles.
Changes to Billing Page
We will be temporarily disabling the Billing History view in the Usage and Billing section of the web app while we work on a fix to correct some inaccuracies we've noticed with the data presented there.
While we implement this fix, please reach out to your account representative for data on billing and usage for previous periods.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Adelaide, Australia (IPv6)
Bangkok, Thailand (IPv6)
Chennai, India (JIO)
Chennai, India (JIO) (IPv6)
Denver, CO, USA
Denver, CO, USA (IPv6)
Denver, CO, USA (Cogent) (IPv6)
Falkenberg, Sweden (IPv6)
Frankfurt, Germany
Frankfurt, Germany (IPv6)
Hyderabad, India (IPv6)
Istanbul, Turkey (IPv6)
Manila, Philippines (IPv6)
Philadelphia, PA, USA (IPv6)
Wellington, New Zealand (IPv6)
In addition, we've added AWS, Azure, and Google Cloud Platform Cloud Agents in the locations listed below. For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
New Amazon Web Server Cloud Agents
Hong Kong (AWS ap-east-1)
Manama, Bahrain (AWS me-south-1)
Milan, Italy (AWS eu-south-1)
Osaka, Japan (AWS ap-northeast-3)
Stockholm, Sweden (AWS eu-north-1)
New Azure Data Center Cloud Agents
Berlin, Germany (Azure germanywestcentral)
Canberra, Australia (Azure australiacentral)
Cheyenne, Wyoming (Azure westcentralus)
Dubai, UAE (Azure uaenorth)
London, England (Azure uksouth)
Oslo, Norway (Azure norwayeast)
Zurich, Switzerland (Azure switzerlandnorth)
New Google Cloud Platform Cloud Agents
Delhi, India (GCP asia-south2)
Hong Kong (GCP asia-east2)
Jakarta, Indonesia (GCP asia-southeast2)
Las Vegas, NV (GCP us-west4)
Melbourne, Australia (GCP australia-southeast2)
Osaka, Japan (GCP asia-northeast2)
Salt Lake City, UT (GCP us-west3)
Seoul, South Korea (GCP asia-northeast3)
Toronto, Canada (GCP northamerica-northeast2)
Warsaw, Poland (GCP europe-central2)
Zurich, Switzerland (GCP europe-west6)
Updated Dashboards and Reports Caching Mechanism
The caching mechanism for dashboards and reports has been updated to improve end-to-end performance. As part of this change, the cache time has been increased from 5 minutes to 10 minutes. This will result in a delay in updated dashboards and reports when you change the ownership of a test to a different account or organization ID.
For example, if the test ownership moves from Account Group A to Account Group B, this change will only be reflected in a customer's dashboard or report when the cache is updated, meaning there is a maximum potential delay of 10 minutes.
2021-09-01 Bug Fixes
Previously, searching the Endpoint Agent Agent Settings filter list for values of any of four specific metrics (User, VPN Gateway Address, VPN Client Address, or VPN Client Network) would not work correctly. This has been resolved.
A display issue caused new Endpoint Agents to only display the username, but not the agent name for the host device. Both agent name and the username are now displayed again on the Agent Settings page.
Previously, some alerts were not clearing, even after the associated alert rule was disabled or deleted. This was due to a configuration propagation issue that has since been corrected.
An issue was fixed where report and dashboard widgets were incorrectly inheriting names of previously named widgets on save.