Changelog
We have improved the way we calculate the Availability metric for Device data in reports and dashboards. Availability is now calculated based on the number of interfaces that are up and running divided by the number of interfaces that the admin set as up. This is a slight change to the previous calculation, which also included in rare instances interfaces that were up and running even if they were set as down by the admin.
Additionally, customers can now query Device Availability by median, max, min, Nth percentile and standard deviation.
Effective immediately, ThousandEyes will be making an update to our profile logic so that requests to update the email address of a provisioned user will only be accepted a maximum of once per hour. Once that limit is reached, subsequent requests to update email will then be blocked by the system until the hour period has passed. This change will patch a security vulnerability. We do not anticipate this change to affect customer account access outside of the above limitation.
In the coming months, as part of an architectural change to BrowserBot, ThousandEyes will add te-podman as a dependency of the te-browserbot package. This new package depends on some packages available in the Red Hat Enterprise Linux 7 Server - Extras and Oracle Linux Addons repositories for RHEL 7 and Oracle Linux 7, respectively.
Users with existing agents running on RHEL 7 should enable the rhel-7-server-extras-rpms repository. Users with existing agents running on Oracle Linux 7 should enable the ol7_addons repository. Otherwise, BrowserBot will not be able to upgrade to this forthcoming version.
On RHEL 7, you can enable this required repository by executing subscription-manager repos --enable=rhel-7-server-extras-rpms. On Oracle Linux 7, execute yum-config-manager -q -y --enable ol7_addons.
New Cloud Agents have been added to the following locations:
Madrid, Spain (IPv6)
Mumbai, India (JIO)
Mumbai, India (JIO) (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
An issue was found in the Stacked Area graph widget for reports and dashboards when displaying a Page Load metric. The calculation for the metric was adding DOM load time to the page load time, which is incorrect, as DOM load time is a sub-component of page load time. This has now been resolved and the correct value is displayed.
In the coming weeks, ThousandEyes will change the default session timeout duration from indefinite to 30 minutes. This change will patch a security vulnerability, as well as better align the timeout duration with industry best practices.
As part of this change, the keep_session_alive_on_auto-update permission will be automatically removed from all default built-in roles. This change could affect customers that continuously display ThousandEyes dashboards (for example, in Network Operations Centers).
If you would like to preserve the keep_session_alive_on_auto-update permission for certain users within your organization, we recommend that you create a custom role with this permission and assign users in the appropriate account group to that role.
For more information on creating custom roles, see Managing Roles.
We will be temporarily disabling the Billing History view in the Usage and Billing section of the web app while we work on a fix to correct some inaccuracies we've noticed with the data presented there.
While we implement this fix, please reach out to your account representative for data on billing and usage for previous periods.
New Cloud Agents have been added to the following locations:
Adelaide, Australia (IPv6)
Bangkok, Thailand (IPv6)
Chennai, India (JIO)
Chennai, India (JIO) (IPv6)
Denver, CO, USA
Denver, CO, USA (IPv6)
Denver, CO, USA (Cogent) (IPv6)
Falkenberg, Sweden (IPv6)
Frankfurt, Germany
Frankfurt, Germany (IPv6)
Hyderabad, India (IPv6)
Istanbul, Turkey (IPv6)
Manila, Philippines (IPv6)
Philadelphia, PA, USA (IPv6)
Wellington, New Zealand (IPv6)
In addition, we've added AWS, Azure, and Google Cloud Platform Cloud Agents in the locations listed below. For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Hong Kong (AWS ap-east-1)
Manama, Bahrain (AWS me-south-1)
Milan, Italy (AWS eu-south-1)
Osaka, Japan (AWS ap-northeast-3)
Stockholm, Sweden (AWS eu-north-1)
Berlin, Germany (Azure germanywestcentral)
Canberra, Australia (Azure australiacentral)
Cheyenne, Wyoming (Azure westcentralus)
Dubai, UAE (Azure uaenorth)
London, England (Azure uksouth)
Oslo, Norway (Azure norwayeast)
Zurich, Switzerland (Azure switzerlandnorth)
Delhi, India (GCP asia-south2)
Hong Kong (GCP asia-east2)
Jakarta, Indonesia (GCP asia-southeast2)
Las Vegas, NV (GCP us-west4)
Melbourne, Australia (GCP australia-southeast2)
Osaka, Japan (GCP asia-northeast2)
Salt Lake City, UT (GCP us-west3)
Seoul, South Korea (GCP asia-northeast3)
Toronto, Canada (GCP northamerica-northeast2)
Warsaw, Poland (GCP europe-central2)
Zurich, Switzerland (GCP europe-west6)
The caching mechanism for dashboards and reports has been updated to improve end-to-end performance. As part of this change, the cache time has been increased from 5 minutes to 10 minutes. This will result in a delay in updated dashboards and reports when you change the ownership of a test to a different account or organization ID.
For example, if the test ownership moves from Account Group A to Account Group B, this change will only be reflected in a customer's dashboard or report when the cache is updated, meaning there is a maximum potential delay of 10 minutes.
Previously, searching the Endpoint Agent Agent Settings filter list for values of any of four specific metrics (User, VPN Gateway Address, VPN Client Address, or VPN Client Network) would not work correctly. This has been resolved.
A display issue caused new Endpoint Agents to only display the username, but not the agent name for the host device. Both agent name and the username are now displayed again on the Agent Settings page.
Previously, some alerts were not clearing, even after the associated alert rule was disabled or deleted. This was due to a configuration propagation issue that has since been corrected.
An issue was fixed where report and dashboard widgets were incorrectly inheriting names of previously named widgets on save.
In the coming weeks, ThousandEyes will change the default session timeout duration from indefinite to 30 minutes. This change will patch a security vulnerability, as well as better align the timeout duration with industry best practices.
As part of this change, the keep_session_alive_on_auto-update permission will be automatically removed from all default built-in roles. This change could affect customers that continuously display ThousandEyes dashboards (for example, in Network Operations Centers).
If you would like to preserve the keep_session_alive_on_auto-update permission for certain users within your organization, we recommend that you create a custom role with this permission and assign users in the appropriate account group to that role.
Endpoint Agents now support F5 VPN connections, and run network tests to these virtual networks.
As part of our continued efforts to scale our services globally, from this release, new agents will use a different hostname to reach the ThousandEyes cloud service. New agents will now use registry.agt.thousandeyes.com when registering for the first time upon boot. Subsequent connections will still use sc1.thousandeyes.com.
New Cloud Agents have been added to the following locations:
Portsmouth, England (IPv6)
Palermo, Italy (IPv6)
Santiago de Chile, Chile (IPv6)
Milan, Italy (IPv6)
Taipei, Taiwan (IPv6)
Ho Chi Minh City, Vietnam (IPv6)
Guadalajara, Mexico (Trial) (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
The Endpoint Agent Agent Settings page has been significantly improved for performance. For larger account groups, the Agent Settings page should be displayed around 10x faster, while filtering and searching for agents is now up to 50x faster.
For transaction tests, the default device emulation has been changed from Desktop Small to Desktop Large to better simulate a real user's environment.
An agent that has been deleted for 7 days is now removed from the list of assigned agents in the Endpoint Agent proxy settings. This brings this functionality in line with agent labels and scheduled tests, where agents are already removed 7 days after deletion. Agents that have been deleted but are still within the 7 days recovery window are shown with a '(deleted)' note in the list of assigned agents in the Endpoint Agent proxy settings.
A Deleted filter has been added to the Endpoint Agent drop-down list, to improve usability. This filter can be used to show all agents that have been deleted from the platform.
Previously, creating an existing user through SCIM could result in a runtime error, rather than the more accurate 409 response code. This has been corrected, and users will see a 409 error in this scenario.
Previously, responses from SCIM endpoints would include an additional element than requested when paging results. This has been resolved to return the exact amount requested.
An edge case where the Endpoint Agent filter could fail to be displayed above the agent search box in the Agent Settings page has been resolved.
Previously, the list of agents displayed in the Scheduled Tests view's agent filter did not include agents that were using computerName to identify themselves, and had no unique name set. Agents that use computerName as the identifier are now also included.
The Endpoint node is now correctly colored green when selecting the CPU / Memory metric in the Local Networks view.
Previously, a system metrics parameters are not available message was returned when a call was made to https://api.thousandeyes.com//v6/endpoint-data/tests/web/http-server/{testId}.json. The correct metrics data is now returned.
An issue was found where alerts were incorrectly firing during an alert suppression window. This was related to a maintenance window that we created for bootstrapping the configuration for the new alerter, and has been fixed.
Previously, some alerts were not being cleared after deletion of the alert rule. This was due to a metadata issue that has since been corrected.
As part of this release, ThousandEyes now supports deploying Enterprise Agents on Cisco Catalyst 8000 series and ISR 4000 series routers. For full support details and installation instructions, see:
​Support Matrix​
Starting with this agent release, ThousandEyes supports browser tests on Catalyst 9300 and 9400 switches that have an SSD. For more information, see the Support Matrix.
As transaction and page load tests are run from a ThousandEyes Agent, all HTTP requests for those tests include a custom header, X-ThousandEyes-Agent: yes. This header is necessary to identify requests as originating from a ThousandEyes agent.
In certain edge cases (for example, when an agent or test is configured to use a PAC file), the inclusion of this identifying header will trigger a CORS policy violation for some cross-origin requests.
To circumvent this issue, HTTP requests can alternatively be made identifiable by extending the User Agent request header with a specific identifier text. Customers whose transaction or page load tests are affected by cross-origin request failure will be provided with a dropdown labeled "Agent ID Strategy" in the advanced settings for those test types. This dropdown contains two options:
x-thousandeyes-agent Header: The default selection for transaction and page load tests where the custom header is used as the identifier.
Extend User Agent Header: Selecting this option will ensure that the problematic custom header is not included in Agent HTTP requests, but also requires that the
(ThousandEyes Agent)identifier text is included in the User Agent request header. When selected, the User Agent dropdown options will update to display "(TE)" after every option name to signify that this identifier text is appended to the User Agent request header. Whenever the User Agent text input is visible the identifier text will be included at the end of the displayed text. If the identifier text is subsequently modified or removed from the text input while the "Extend User Agent Header" option is selected, the test configuration will not be valid and saving the test configuration will be prevented until the identifier text is restored.
Agents assigned to transaction or page load tests which use the User Agent identifier strategy must have:
a minimum agent version of 1.120.0.
a minimum Browserbot version of 1.168.0.
The agent and Browerbot version can be seen in the configuration side panel for an agent in Agent Settings.
In the coming weeks, ThousandEyes will change the default session timeout duration from indefinite to 30 minutes. This change will patch a security vulnerability, as well as better align the timeout duration with industry best practices.
As part of this change, the keep_session_alive_on_auto-update permission will be automatically removed from all default built-in roles. This change could affect customers that continuously display ThousandEyes dashboards (for example, in Network Operations Centers).
If you would like to preserve the keep_session_alive_on_auto-update permission for certain users within your organization, we recommend that you create a custom role with this permission and assign users in the appropriate account group to that role.
Previously, HTTP scheduled tests on macOS Endpoint Agents would fail when the "Verify SSL Certificate" flag was off, even if a valid certificate was installed. This has been fixed.
An issue was found where any data collected while an Endpoint Agent device was connected to mobile broadband may not appear in the Endpoint Agent views. This included scheduled test, browser session, and local network data. This issue has been resolved, and the Endpoint Agent now recognizes when a mobile broadband adapter is used as the primary network interface correctly.
An issue was found that caused outage alert rules to be shown and selectable in the test settings Alert Rules list, despite not having a corresponding test type. The outage alert rules have now been removed from the list.
The "Country" column header under the "Cloud Agents" tab in agent settings has been updated to "Country and Region" to ensure the column header captures all agents and their associations.
Added a warning message on the alert list page to provide additional context as to why certain alerts may not be clearing due to lack of data.
Previously, the datapoint sent to the alerting platform used to include all markers that were present in a script, regardless of if they were present in the actual measurement or not. This changed when we upgraded our alerting platform, which resulted in an issue where the transaction test alert rule expression Marker \ is not present was not being evaluated correctly. This has been fixed.
A rare case was found where a DNS query performed on a particular test caused an instance on some of our cloud agents to crash. This has been resolved.
An issue was found that prevented large PDFs from being downloaded. This has been resolved, and customers can now download all reports that can be loaded and rendered in the application as PDFs.
An issue was found where the "measure" header for Internet Insights data was populated incorrectly for when downloading a CSV file for a report. This has been fixed.
An issue was found where the "Show in device layer" hyperlink was not correctly redirecting users to the appropriate device layer views from the path visualization. This has been resolved.
An issue was found where agent uptime was not reported correctly in the Agent Statistics tab in agent settings. This was caused by a MySQL error, and has been fixed.
An issue was found where the Endpoint Agent browser sessions Waterfall view was not showing the correct messages when the user hovered over alert messages. This has been resolved.
An error message is now displayed if the user tries to save a label with the deprecated VPN Address attribute.
A visual error caused labels to be removed from the display when an Endpoint Agent test was saved. This has been fixed.
Fixed a rare issue that prematurely ended some alert suppression windows, causing alerts to be fired during this suppression.
As part of ThousandEyes' commitment to improving security within the product, the OAuth and Access Token fields are no longer visible within the User Profile. Token values are presented when they are initially created, but will not be available afterwards to protect the confidentiality of the information.
As of Endpoint Agent client version 1.83.0, the Endpoint Agent has been tested and verified against the new Apple M1 chipset. Additionally, the browser extension for Chrome and Edge now declares support for the new mac/arm64 platform.
By default, browser extensions are now an opt-in feature during installation. Each extension must be explicitly mentioned in the ADDLOCAL section of the msiexec command. This change has been made to avoid clashes between the ExtensionInstallForceList set at the user-level by group policy versus the ExtensionInstallForceList set at the machine level set by the Endpoint Agent. Although multiple levels of policies can exist, Google Chrome only takes into account one policy. See https://support.google.com/chrome/a/answer/9037717?hl=en for further details, and Endpoint Agent Installation Reference for installation details.
Existing installations of Endpoint Agent will not be affected, as upgrade installs respect the existing settings unless additional command line arguments are used.
The Internet Insights default alert rule for outages has been renamed the Default Network Outage Alert Rule.
New Cloud Agents have been added to the following locations:
Markham, Canada (IPv6)
Montevideo, Uruguay (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
An issue was found where alerts were either not showing any active location alerts, or not populating at all, preventing them from being correctly displayed on the Alert List page. This has been resolved.
In some rare cases, alert suppression windows would end prematurely, resulting in alerts firing during the intended suppression. This has been resolved.
An issue was found where shared tests that had previously been disabled in the shared account were triggering alerts. This behavior has been corrected.
An issue was found where assigning specific agents to a web-transaction test caused alerts to fail to trigger, even if the alert conditions were met. This has been resolved.
An edge case was found when Cloud Agent vantage points were removed from tests that had an Internet Insights rule attached. In this case, a small subset of Cloud and Enterprise Agent alert rules were disassociated from the tests. This has been fixed.
Previously, API queries to browser sessions data were not using the userSessionIds field correctly. This has been resolved.
An issue was found in Endpoint Agent label filtering where the text string used was not filtering out all non-matching agents. After the fix, only agents matching the filter text will be displayed.
In cases where there was an unknown path to a valid destination, the packet loss displayed in the Endpoint Agent path visualization was not consistent with that displayed in the map view. This has been corrected, and the two views now display the same value.
All PagerDuty integrations are now compliant with their updated "Simple Install Flow" method for authentication, which can be found here: Events Integration.
This change will allow you to select multiple PagerDuty services to integrate with ThousandEyes. If multiple services are selected on the PagerDuty side, multiple integrations will be created on the ThousandEyes side.
Starting with this release, we have increased the per-agent limit for monitored interfaces to assist in the adoption of monitoring devices using SNMP.
Prior to this release, there was a limit of 4000 interfaces per agent. This required users to deploy additional agents to monitor more interfaces, and thereby more devices using the device layer. The new behavior increases the supported interfaces per agent to around 20,000. The exact number of interfaces supported per agent may depend on local network conditions, and users can leverage the agent utilization information that shows device utilization to gauge the ability to monitor additional interfaces.
Previously, connection failures could occur when Endpoint Agent TCP network tests were run, because the proxy server port was not taken into account. This has been fixed in Endpoint Agent Client version 1.82.0.
An issue was found where unexpected network traffic in Endpoint Agent TCP tests resulted in an erroneous calculation of impossibly long delay values from hop 0. This has been resolved in Endpoint Agent Client version 1.82.0 by dropping any incoming packet that we cannot match against a source packet.
The "Exclude data from Cloud Agents with Local Problems" tooltip in reports and dashboards has been updated to no longer identify the specific agents that are having issues. A generic message is now provided. To view which agents are having issues, see the individual test view for information.
Alert conditions now appear in alphabetical order when setting up an alert rule, rather than order of creation.
An issue was found that prevented alerts from being shown correctly in the Alert List page. Some alerts were not showing any active location alerts, while some where not populating at all. This has been resolved.
An issue was found that prevented certain alerts from clearing. This was caused by an ungraceful reset of the alerting platform. The impacted alerts have been cleared, and improvements have been made in order to prevent this from happening again.
In some cases, the geolocation calculation of an Endpoint Agent could be overwritten by less reliable measurements of location. The accuracy of our calculations has been improved by ensuring we always preserve successful resolutions of RDNS to a specific location.
An issue was found where Endpoint Agent snapshots with a filter could sometimes take longer to generate than the timeout period configured, resulting in the snapshot failing to be created. The timeout period has been extended to resolve this issue.
An issue was found in the usage calculation for Endpoint Agent accounts where, when an account was using 0 agents, the usage page incorrectly displayed the last usage. This has been corrected, and 0 Endpoints is now displayed as expected.
An issue was found in the permissions framework for Endpoint Agents that caused users with the 'View endpoint tests' permission but not the 'View tests' permission to be unable to view Endpoint tests. This has been resolved.
The Google Chrome browser extension has been redesigned so that configuration changes to the set of monitored domains are now received by the extension faster. Previously, these changes were only received when the agent transitioned from an actively recording page or domain to one that was not recorded (or vice versa). Now, configuration changes are received and applied as soon as they are delivered to the agent.
This improvement also fixes inconsistencies in the data display and UI extension status for Windows agents using Microsoft Edge and Internet Explorer.
The Agent Settings page now contains an icon to indicate whether an Endpoint Agent is enabled for TCP tests.
All Mac agents, and all Windows agents with the npcap library installed, support TCP, provided the agent is up to date.
Previously, when an agent received a network change event, it cleared its scheduled test assignments, checked in with the ThousandEyes backend, and waited for new scheduled test assignments. For agents that were unable to connect via websockets, the agent would wait for one minute before polling the backend for its assignments. During this window, the agent would not run scheduled tests, occasionally resulting in short gaps in test data. This window has been decreased to thirty seconds in Endpoint Agent client version 1.81.0 to reduce the chance of this issue occurring.
Previously, network changes for Zscaler VPNs would cause the list of scheduled tests to be refreshed immediately. However, the network would cycle again before the tests had been run, causing tests to fail. This has been resolved in Endpoint Agent client version 1.81.0 by ignoring rapid changes for up to a five minute period, and the Endpoint Agent is now able to continue testing when running a Zscaler VPN that is cycling frequently between a range of configured IP addresses.
An issue was found where the list of domains that should bypass the Zscaler tunnel, when used in "tunnel with local proxy" mode, was incorrect, resulting in the path visualization not clearly showing that traffic was bypassing the Zscaler network, as it should. This has been corrected in Endpoint Agent client version 1.81.0, and the list of domains is now obtained from both the "App profile PAC" and the "Forwarding PAC" files.
New Cloud Agents have been added to the following locations:
Baltimore, Maryland, USA (IPv6)
Columbus, Ohio, USA (IPv6)
Athens, Greece (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Previously, widgets in reports and dashboards would occasionally show the incorrect amount of tests based on the filters selected. We've updated the functionality to only show the number of tests that are expressed in the data point.
Note: If multiple filters are selected that have overlapping tests, or in instances where a test is deleted after the widget is created, the number shown on the widget may be less than the number of tests selected in an individual filter.
Added an error message when a user tries to save a dashboard with < or > as characters in the name to provide more context as to why the dashboard isn't saving (these characters prevent saving due to information security purposes).
An issue was found where the list of providers for the Default Outage Alert Rule was not being correctly updated. This could prevent the rule from triggering, and has been resolved.
An issue was found where the selected Endpoint Agent "grouping" option in the UI did not persist after navigating somewhere else on the timeline, resulting in the default values overriding the selected ones. This has been fixed.
An issue was found where customers were unable to access wireless metrics saved events that were older than 30 days. This bug only affected saved events (not public snapshots), and has been resolved.
An issue was found that caused more informative Endpoint Agent error messages (for example, "The internet connection has been lost") to be displayed as "Other" when a page load fails. This has been corrected, and the full error messages will now be displayed.
An issue was found where the Y axis for some report and dashboard widgets had a truncated name. This has been fixed.
An issue was found where the error count metrics in reports and dashboards were incorrectly showing as "No Data", rather than the correct value of "0 Errors". This has been fixed.
An issue was found where some alerts were triggering incorrectly during customer defined alert suppression windows. This has been resolved.
An issue was found where alerts were incorrectly firing due to stale test prefix metadata, which was leading to alerts firing on deleted IP prefixes. This has been fixed.
An issue was found where some report and dashboard widgets were unable to filter by target agent. This has been resolved.
An issue was found where certain location alerts were presented in the UI as part of the trigger condition that violated the alert condition before the global alert actually triggered. This issue has been fixed.
Note: This fix changes the behavior of other potential alert trigger instances. As of this release, the only location alerts that will be displayed in the UI at a global alert trigger will be the location alerts active at the time of trigger. This could lead to scenarios where a flapping agent was involved in the evaluation criteria of an alert being fired, but has since cleared before the global alert fires. However, this will only occur if the alert conditions have multiple agents that need to fire multiple times in a row.
As part of our migration to our new alerter infrastructure, we have made a change to the way transaction metrics are reported in our Alert List page. We have moved from string text to key value pairs for both Metrics at Alert Start and Metrics at Alert End. This will be rolled out to 100% of customers over the coming weeks.
In the coming weeks, ThousandEyes will add te-podman as a dependency of the te-browserbot package. Podman version 3.1.0 or greater will be required for page load and transaction tests.
If your Enterprise Agent has an existing compatible version of Podman, Browserbot will configure it to execute tests. Otherwise, Podman version 3.1.0 will be downloaded from the ThousandEyes public repositories (or a configured repository with a compatible version of Podman).
New Cloud Agents have been added to the following locations:
Eindhoven, Netherlands (KPN) (IPv4 and IPv6)
Kansas City, MO, USA (IPv6)
Lima, Peru (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
The Endpoint Agent is now signed with an Extended Validation (EV) certificate. This smooths the installation process by providing immediate reputation with Microsoft SmartScreen in the event that the certificate used to sign the Endpoint Agent installer package is changed (for example, after it expires or is revoked).
This is implemented in Endpoint Agent Client version 1.80.0.
Several improvements have been made to the alert integration notifications:
The property name ID has been updated to Alert ID.
The Test Name has been added to the notification.
The URL or Server target has been added to the notification.
The Scope field has been changed to Number of Agents.
Property names have been updated to title case rather than camel case (i.e. Rule ID instead of ruleId).
The alert rule expression now appears in the Alert Rule field.
The Clear Date value has been removed.
The word Triggered has been added as part of the email subject line for triggered alerts.
Previously, an alert with the error matches or error does not match criteria could trigger even if no errors were present. For example, if an alert condition was set that stated "Alert when: error does not match 401", an alert could trigger even if there were no errors, as a 401 error was also not present. We have updated the way we evaluate these two alert criteria to only trigger an alert when at least one actual error is present. If no errors are present, these two alert criteria will not trigger.
An issue was found where Metrics at Alert End was showing as N/A - Agent Removed from Test when an Endpoint Agent was removed from a test, after the alert had triggered. The agent is now persisted in the the Alert List and Alert History page, even if the agent has been removed.
An issue was found where updating an existing test with Cloud Agents did not immediately reflect the configuration change. This has been fixed.
New IPv4 and IPv6 Cloud Agents have been added to Eindhoven, Netherlands (Ziggo). For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Previously, the VPN node in a path visualization merged with the first hop of the overlay. This caused some confusion with customers. We've updated the view so that the VPN node will stand alone in the path, and the first hop of the overlay will appear as a regular node.
An issue was found where, for some disabled alerts, the alert would still fire if the conditions were met. This has been resolved.
An issue was found where location alerts were failing to persist due to the metrics being reported as empty. This lead to location alerts not being cleared when they should have been, and has been resolved.
An issue was found where the SSL certificate expiry alert rule was interpreted incorrectly, and caused an alert to fire when no certificates were expiring. This has been fixed.
An issue was found where the API response to /v6/tests/\.json did not return all account IDs and their associated account group names if the user was associated with multiple organizations. This has been fixed.
An Internet Insights issue was found where it appeared that some outages shown in the Views timeline were not being displayed in the Overview dashboard. This was because the Overview and Views timeline did not previously refresh at the same interval, resulting in windows where they would be briefly out of sync. The default refresh interval for the Overview has been lowered to match the Views timeline, and this resolves the issue.
An issue was found where updating the test interval to a higher frequency (for example, changing from 5 minutes to 2 minutes) would cause the timeline to show gaps in data. This has been resolved.
An issue was found where the alerting system was incorrectly interpreting the alert configuration, resulting in alerts firing when the conditions were not met. Alert configuration validation has been improved to resolve this issue.
An issue was found where the Availability metric in reports and dashboards was showing as milliseconds instead of percentage. This has been fixed.
An issue was found where, in some cases, the system was not properly appending agent details to alerts. This resulted in the UI incorrectly showing 0 triggered agents against an alert, and has been fixed.
An issue was found where the global Alert Start Time for some alerts was showing as significantly before the Alert Start Time for the location alerts. This was caused by logic in the platform that carried the first instance of a condition violation for an agent, rather than the most recent violation. This logic has been updated, and the issue is now fixed.
An issue was found where, if ICMP was blocked and preventing end-to-end metrics from being measured, a 100% packet loss alert would be triggered. This issue has been resolved.
An issue was found where the number of agents shown to be running a scheduled test could differ between the Timeline and Table views. This occurred in instances where there were multiple data points for the same machine in the same round. The way the agent count is done has been adjusted, and only unique machines in a round will now be counted.
As of browserbot version 1.163.0 and ThousandEyes Recorder IDE version 1.1.0, the transaction test scripting sandbox now supports importing functionality from the Node.js crypto module.
One way to leverage this new functionality is to compute an HMAC:
import { credentials } from 'thousandeyes';import { createHmac } from 'crypto';​const dataToSign = 'hello, world!';const hmac = createHmac('sha256', credentials.get('some-secret-key')).update(dataToSign).digest('base64');
We have added a "Prefer TCP" option to Endpoint Agent scheduled tests that will ensure the test will fall back to ICMP + TCP Connect if no driver is detected, so that tests are still performed. This option is to avoid situations where Endpoint Agents are not included in TCP tests due to either the Windows driver missing, or agents running versions earlier than 1.75.0.
In addition, the TCP Connect checkbox has been removed, and replaced by additional Protocol options:
ICMP
ICMP + TCP Connect
Prefer TCP (fallback to ICMP)
TCP (Driver required)
Starting this release, all Catalyst switching customers entitled to ThousandEyes will see a new onboarding workflow when visiting their newly created ThousandEyes organization. The workflow will guide them through creating agents, tests, and surface common use cases.
The alert rules write API now supports the following alert rule metrics: certificate validity, TLS version, and weak cipher metrics.
A new Cloud Agent has been added to Johannesburg, South Africa (Azure southafricanorth). For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
An issue was found where some users were receiving emails on cleared alert events even when they deselected the option to receive cleared alert emails in the UI. This has been resolved.
An issue was found where gaps in data were incorrectly displayed on the timeline. This has been resolved.
An issue was found in the v6 reports API where incorrect results were returned for certain 'groupValues' and 'groupProperty'. This has been corrected.
An issue was fixed where changes to a filter on a report or dashboard widget weren't persisting, and reverted back to the previous filter. This has been corrected.
API requests to /v6/endpoint-tests/http-server/new with the groupId missing now return a 400 rather than a 500 error. This is more explanatory, since the error is with the input rather than the server.
An issue was found when an Endpoint Agent was uninstalled and later re-installed, where the Last Contact field would not accurately reflect the new Endpoint Agent installation date/last contact time. This has been corrected.
An issue was found where newly created organizations were not able to use the AppDynamics Dash Studio integration due to a 0 API rate limit. This has been corrected and fixed for all customers.
In the coming weeks, ThousandEyes will add te-podman as a dependency of the te-browserbot package. This package is required for page load and transaction tests, and will be automatically installed.
te-podman is a wrapper of the Podman package. If your Enterprise Agents have existing installations of Podman, this dependency may cause conflicts. We suggest removing Podman before installing this new version of BrowserBot.
Joint ThousandEyes and AppDynamics customers can now ingest ThousandEyes Cloud and Enterprise test data directly into their AppDynamics Dash Studio instance as either a number card widget or a time series widget. For more information, see ThousandEyes Integration with AppDynamics.
The Endpoint Agent now allows scheduled tests to be executed using TCP as well as ICMP protocols. This is available in Endpoint Agent version 1.75.0 and above. For more information, see Endpoint Agent TCP Support.
The average response time per hop is now shown in the Endpoint Agents > Views > Local Networks view.
The Endpoint Agent API response now includes DNS server test ping data instead of ICMP ping data for DNS probes. This avoids cases where no ICMP data was available for DNS probes.
Agent-to-server network tests can now be created and read with DSCP parameters (dscp and dscpId) when using the API.
Previously, some of the advanced settings options for shared tests were selectable. This gave users the impression that those settings could be modified within the shared account group, which is incorrect. All test settings options are now unselectable to ensure users are aware that shared tests cannot be edited from the shared account group.
The SNMP timeout and number of retry attempts for device availability SNMP requests has been increased to account for devices with high CPU utilization that either take a long time (over 2 seconds) for an SNMP or may ignore the SNMP request altogther.
Endpoint network topologies can now be grouped by interface. The available options to group by are IP Address, Network, Network & Location and Location.
An edge case was found when using the Endpoint Agent public API to get test data, where if there was exactly 1000 data points returned, the next page URL for pagination was not set. This has been resolved.
An issue was found that caused attempts to create Endpoint Agent snapshots with a date range of more than 12 hours to fail. This has been resolved, and snapshots can now be created with a date range of up to 2 days.
An issue was found where the hop slider in a multi-service path visualization view was not showing every hop, even when the slider was set to show maximum hops. This has been resolved.
An issue was found where the traceroute style output in the path visualization layer showed the test target IP address instead of the terminating hop's IP address. This has been resolved.
An issue was found that caused sorting Enterprise Agents by Utilization to incorrectly sort the list of agents. This has been fixed.
An issue was found that caused test names that included '<' or '>' to be incorrectly rejected due to a silent validation error. This has been corrected.
The ThousandEyes Recorder IDE for Windows will be signed by "Cisco Systems, Inc" from version 1.0.5 onwards. Customers running version 1.0.4 will be able to auto-update without issue.
Customers running version 1.0.3 or earlier may need to reinstall the IDE to upgrade to version 1.0.5.
The latest version of the IDE is always linked from the Test Settings page when the transaction test option is chosen.
We have placed a hard render limit on our Cloud and Enterprise and Device data as it relates to our dashboards and reports. Widgets that have a high number of tests, a large number of agents running those tests, and long timespans may hit this hard render limit. For more information, see Working with Reports.
The Activity Log now provides the option to display more details related to "create" and "update" events for transaction tests. For these events, a View Change button is displayed in the event row. This button opens a modal that provides a diff showing any transaction script changes. A separate tab shows any credential selection changes for the test.
We have made several updates to our webhook alert notifications around outage alerts:
The link field has been deprecated and replaced by the permalink field. The link field will remain for the time being, but will eventually be removed. ThousandEyes recommends using the permalink field going forward.
The ruleAid field has now been added to all alert types that use the new notifier. Previously, it was only included in outage alerts.
A dateEnd field has been added.
A type field has been added.
The ruleRefId field has been removed.
Endpoint Agents can now be transferred in bulk via the public API. Users can upload a CSV file containing the agentId, the source account and the target account. For large transfers (several thousands of agents) it may take a few minutes for the UI to show the agents as transferring.
We've redesigned the look of test labels for scheduled tests.
Fixed an issue where a small number of alerts were being inadvertently suppressed due to faulty "agents with local problems" logic in our backend system.
Fixed a UI bug where the system was not rendering error messages correctly around OAuth authentication for our webhook integrations. Previously, clicking Add New Webhook would close the modal dialog even if the save operation failed. This issue has been fixed to ensure the proper error message is shown before a webhook is saved.
Fixed an issue where the alerter system was not correctly updating alert configurations based on updates to IP prefix subnet ranges. This impacted a very small number of customers and all spurious alerts have been cleared.
In Views > Scheduled Tests > Table, an agent that appeared when no filters were applied would not be displayed when filtered by agent using its own name. This was because we were only considering agents that submitted data for the round if a filter was applied. This is now fixed.
Fixed an issue where, when a report based on Internet Insights metrics had a duration set to 12 hours or less, some widgets displayed "No Data" instead of the expected 0 outages.
Starting April, 2021, the Google Chrome and Microsoft Edge browser extensions are installed by default in new installations of Endpoint Agent on Windows endpoints. Users can opt-out of the installation as part of the Endpoint Agent installer process.
If you opt not to install the extensions as part of the Endpoint Agent installer process, you can still install them separately. See Install the Browser Extension.
The ThousandEyes Endpoint Agent is now able to identify the presence of Zscaler Internet Access (ZIA) on customer machines allowing us to gather metrics and performance measurements for traffic transiting the Zscaler network. Metrics and measurements are displayed in the UI in the same way as other supported VPN providers. Some ZIA deployment models may not be fully supported. For more information see Endpoint Agent VPN Support.
ThousandEyes has introduced a new image for its Docker Enterprise Agents that attaches an anonymous Docker volume to each container running the Enterprise Agent. This change supports a future enhancement that will improve the consistency and performance of browser-based tests.
The endpoint scheduled test table now displays a dash rather than 0% loss if the loss is in fact null, which may happen if ICMP measurements could not be completed.
The ThousandEyes Endpoint API has been extended to allow existing agent labels to be updated.
The API /endpoint-agents now correctly respects the "deleted" flag when returning a list of agents. By default this flag is false - only non-deleted agents are returned.
The Endpoint Agent basic service set identifiers (BSSID) table now shows a list of all of the BSSIDs detected in a test round.
The location of the Endpoint Agent is no longer updated if insufficient WiFi data is obtained from the end user machine, to avoid occasional "flapping" between agent locations that occurred due to operating system inconsistencies in obtaining the WiFi data used for obtaining geolocation data.
In cases where the speed score cannot be calculated due to an error gathering data for the page, the experience score is now set to 0 rather than -.
The default widget time on reports has been updated from 7 days to 1 day. This value can still manually be changed by the customer at the report level or the individual widget level.
Effective March 30, ThousandEyes will cease support for Internet Explorer 11 for https://app.thousandeyes.com. Support for all other browsers remains unchanged.
Fixed an improper handling of an aborted request that could result in an occasional failure to display an Endpoint Agent Path Visualisation.
An issue was found where creating an Endpoint agent scheduled test via the API could throw a 500 Internal Server Error. This has been resolved.
An issue occurred where our third-party provider of geolocation data was wrongly assigning an IP address to a location in the UK rather than in New York City, USA. This is now resolved.
An edge case was found where, when an Endpoint alert rule was applied to a scheduled test, the test would show the assigned rule, but the alert rule would show 0 assigned tests. This issue has been resolved.
API endpoint /v6/endpoint-data/user-sessions no longer returns 500 when only the "from" field is specified. In this case, the end date of the query will default to the current time.
API endpoint /v6/endpoint-data/network-topology no longer returns 500 when specifying only the "from" field is specified. In this case, the end date of the query will default to the current time.
Previously, unresolved VPN overlay targets were displayed on the path visualization as 0.0.0.0, which was misleading. This has been corrected, and they are no longer displayed.
An issue was found where a deleted Endpoint Pulse Agent was still being returned in API calls. This has been resolved.
Previously, an issue occurred where a combination of filters was interpreted as OR rather than AND. The path visualisation agent filter now correctly filters the list of agents.
Previously, it was possible to see different colours when clicking a group to expand it versus when you applied the equivalent filter manually. The path visualisation now shows consistent colours for groups.
In a small number of cases, test data was not collected as scheduled tests data points were discarded if the tests couldn’t finish within their expected time bounds. Data points are no longer discarded in this case.
Fixed an issue where the interactive Endpoint Agent Windows installer could fail installing a COM component and return an error "There is a problem with this Windows installer package."
Fixed an issue where a handful of no longer valid BGP alerts were incorrectly being included in reporting and dashboarding widgets due to an incorrect clear time on the alerts.
An issue was found where enterprise agents running shared tests were not showing up as a filter option in the shared to account group while using dashboards and reports. This has been resolved.
An issue was found where the hard render limits put in place for report and dashboard widgets were not transparent to users in our multi-metric table widget. Previously, if a multi-metric table request was too large, we would simply say "No Data." We have now added a tooltip that explains that the data cannot be rendered because the request is too large.
Fixed an issue that prevented test labels from being used in reports and dashboards in relation to our BGP hard render limits. A user can now select a test label and the hard render limit calculation will correctly apply. For more information on our render limits, see Working with Reports.
Fixed an issue that occurred when a user tried to deselect a filter selection in a dashboard or report widget and the filter inadvertently closed instead of letting the user deselect multiple options at a time.
Fixed an issue where a small number of users with the correct edit permissions were not able to edit existing alert rules.