Changelog

2022-05-12
ThousandEyes Recorder IDE v1.3.0
Transaction scripting through the Recorder IDE now supports the ability to make API requests through a proxy by using fetch() and an HTTP proxy agent. You can refer the documentation to learn more and examples can be found in the scripting repository.
This version also resolves the issue where, the Recorder IDE for Windows was not cleaning up scoped_dir files in the temp directory after running a test round in the IDE.
Additional Dynamic Baseline Metrics
Additional metrics have been added to our dynamic baseline alerting capabilities:
Web FTP Server Metrics
Web – FTP Server – Connect Time
Web – FTP Server – FTP Negotiation Time
Web – FTP Server – Response Time
Web – FTP Server – Transfer Time
Web – FTP Server – Wait Time
Voice SIP Metrics
Voice – Sip Server – Connect Time
Voice – Sip Server – Invite Time
Voice – Sip Server – Options Time
Voice – Sip Server – Register Time
Voice – Sip Server – Response Time
Voice – Sip Server – Wait Time
Network Metrics
Network – Agent to Server – Jitter
Network – Agent to Agent – Jitter
Network – Agent to Server – Proxy Jitter
Network – Agent to Server – Proxy Latency
Enhanced API Output to support Automated Solutions
For Enterprise Agents, we have added a new element to the output of the agent status in the ThousandEyes API that contains a hash of the interfaces available to that agent, with the IPs assigned to those interfaces. This allows customers building automation to deterministically select which IP and interface combination should be used for testing. For situations where the unordered nature of the IP Addresses element is an issue, this presents a better path.
Minor Enhancements
Automated Session Tests on API - The Endpoint Agent API now allows customers to extract end-to-end metrics, path visualization, and detailed path trace from specific automated session test.
New and Updated Cloud Agents
New Cloud Agents have been added to the following locations:
Riyadh, Saudi Arabia
Riyadh, Saudi Arabia (IPv6)
Halifax, Canada
Boise, Idaho (IPv4)
Boise, Idaho (IPv6)
Seoul, South Korea (IPv4)
In addition, the following Cloud Agent locations been renamed:
Kyiv, Ukraine
Kyiv, Ukraine (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
New Webex Cloud Agents
New Webex Cloud Agents have been added to the following locations:
Amsterdam, Netherlands
Chicago, IL
Montreal, Canada
New York, NY
Sydney, Australia
Tokyo, Japan
Toronto, Canada
For more information on how to use these new Webex Cloud Agents, see the product documentation.
2022-05-12 General Bug Fixes
For automated session tests, an issue has been resolved that was causing inconsistencies in the aggregated metrics for the path view. Customers can now see consistent data (packet loss, latency, jitter) along the different hops of the path view.
Fixed an issue that displayed empty graphs for VPN metrics when no data was collected. Customers will only see graphs with VPN metrics in the Endpoint Agent Views when meaningful data is obtained.
Fixed an issue that was duplicating the system proxy in the path visualization of the browser sessions tests.
Fixed an issue in the Dashboards API (v6), where the ThousandEyes built-in dashboards were not being returned as part of the response payload.
Fixed an issue where the Dashboards drop down list was incorrectly showing in alphabetical order instead of by last seen.
An issue has been resolved in the visualization of BGP path changes which caused a path that was withdrawn and re-announced within the same round to be incorrectly rendered as just being withdrawn.
2022-05-12 Endpoint Agent Client version 1.117.0 Bug Fixes
An issue has been resolved that would occur on a network change event, which caused the Endpoint Agent to stop running automated session tests until the tests were updated or the agent was restarted.
Fixed an issue where the Endpoint Agent was not running Automated session tests for the whole duration of a meeting.
An issue was found where the CPU metrics (CPU, memory) were not being collected for cases where the Local Network probe failed or completed too quickly. This has been resolved.
2022-04-27
Kazan, Russia Cloud Agent Decommission
The ThousandEyes Cloud Agent in Kazan, Russia is currently unavailable.
At this time we have no estimate for when this Cloud Agent will return to operation. If you need help transferring your tests to other agents, contact [email protected]. The nearest available agents are:
Moscow, Russia
St. Petersburg, Russia
Hamina, Finland
Helsinki, Finland
Tallinn, Estonia
Kharkiv, Ukraine
Kyiv, Ukraine
Warsaw, Poland
2022-04-26
New Cloud Agents
New Cloud Agents have been added to the following locations:
New Delhi, India (JIO)
New Delhi, India (JIO) (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Additional Dynamic Baseline Metrics
Additional metrics have been added to our dynamic baseline alerting capabilities:
Network
Agent-to-server - Latency
Voice
RTP stream - Latency
SIP server - DNS time
SIP server - Total time
Web
FTP server - DNS time
FTP server - SSL negotiation time
FTP server - Total time
HTTP server - Connect time
HTTP server – DNS time
HTTP server – Receive time
HTTP server – SSL negotiation time
HTTP server – Total time
HTTP server - Wait time
Page load - Page load time
Page load - Response time
2022-04-26 General Bug Fixes
An issue has been resolved that was preventing customers from downloading a PDF or CSV of a report if an individual widget wasn't fully configured. Customers can now download a PDF or CSV even with widgets that are not fully configured.
An issue has been resolved where, in some cases, the Internet Insights provider selector did not alphabetize the list of available providers.
An issue has been resolved that, in rare cases, prevented the correct sorting of values in descending or ascending order in our stacked bar chart widget for reports and dashboards.
2022-04-18
New Cloud Agents
New Cloud Agents have been added to the following locations:
Indore, India (JIO)
Indore, India (JIO) (IPv6)
Irvine, CA, USA (Century Link)
Irvine, CA, USA (Century Link) (IPv6)
Phoenix, AZ, USA (CenturyLink)
Phoenix, AZ, USA (CenturyLink) (IPv6)
Phoenix, AZ, USA (Cox)
Phoenix, AZ, USA (Cox) (IPv6)
Plano, TX, USA
Plano, TX, USA (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Minor Enhancements
During a DNS trace test, it is possible that a non-authoritative answer is received from a server in the path. Previously, we returned a general error message in this situation. Now, a more descriptive error message similar to "ERROR: Received non-authoritative response (AA flag not set) to query for www.example.com. A" is returned.
2022-04-18 General Bug Fixes
An issue was found where, in some edge cases, Endpoint Agents were able to fire alerts after the organization that they were associated to had been disabled. This has been resolved.
An issue was found where the alert count wasn't being passed correctly via the report and dashboard APIs, and was instead always returning a 'No Data' response. This has been fixed, and the APIs now pass the correct number of alerts present in the appropriate widget.
An issue has been resolved that was preventing the Endpoint Agents drill down filter from loading in report and dashboard widgets.
2022-04-18 Endpoint Agent Client version 1.108.2 Bug Fixes
An issue has been resolved where the Endpoint Agent was generating test data for an application after disabling the corresponding automated session test.
2022-04-04
Alert Severity
You can now set a severity level for your alerts and alert rules. An alert's severity is shown both in the ThousandEyes UI and in any notifications that you have configured, such as email.
If you have existing alert rules that predate this feature, they will be assigned the default Info severity. You can change this default to the severity level that suits your organization's needs.
For more information, see Alert Rule Severity.
2022-03-30
Automated Session Testing
ThousandEyes has added automated session tests to monitor the network performance of selected applications running on end-user machines. Automated session tests are configured and run using ThousandEyes Endpoint Agents. Note that these tests capture information only when the application is running an active session. Initially, the following applications are supported:
Webex
Microsoft Teams
Zoom
Learn more about automated session tests here:
​Automated Session Tests​
​Endpoint Agent Automated Session Tests View​
Cisco Catalyst 8500 and Cisco ASR 1000 Container Installation Support
The ThousandEyes Enterprise Agent for Cisco Application Hosting now supports the Catalyst 8500 and ASR 1000 routing platforms. See the Support Matrix for a complete list of supported agent platforms.
Additional Metrics for Dynamic Baselines
Per our release notes on March 3rd under Changes to Dynamic Baselines, we have now migrated all existing dynamic baseline alert rules to a 24-hour dynamic baseline. Any rules using standard deviation will now look at a 3-hour window for standard deviation. No customer action is required.
We have also added two additional metrics to our dynamic baseline alerting capability:
HTTP - Connect Time
HTTP - Wait Time
For more information, see Dynamic Baselines.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Fort Worth, TX, USA
Fort Worth, TX, USA (IPv6)
Indianapolis, IN, USA
Indianapolis, IN, USA (IPv6)
Milwaukee, WI, USA
Milwaukee, WI, USA (IPv6)
Montreal, Canada (IPv6)
Oklahoma City, OK, USA
Oklahoma City, OK, USA (IPv6)
Ottawa, Canada
Ottawa, Canada (IPv6)
Rotterdam, Netherlands
Rotterdam, Netherlands (IPv6)
Saint Paul, MN, USA
Seoul, South Korea (KT Telecom)
Seoul, South Korea (KT Telecom) (IPv6)
Seoul, South Korea (SK Telecom)
Seoul, South Korea (SK Telecom) (IPv6)
Tucson, AZ, USA
Tucson, AZ, USA (IPv6)
Utrecht, Netherlands
Utrecht, Netherlands (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Minor Enhancements
Endpoint Agent:
We have improved the security of the public installer download links for the Endpoint Agent by replacing the current naming scheme (which uses 5-9 lowercase alphabetic characters) with one that uses 40 characters and considers a wider range of possible characters (upper case, lower case and numbers). The old public installer download links will remain valid for a further 28 days, after which they will not be functional.
A single account group can now contain up to 50,000 Endpoint agents.
A new geo-location algorithm has been deployed to help identify users more accurately, particularly for users connected behind VPNs.
To improve customer experience when being logged out of the application due to a timeout, the ThousandEyes platform will now redirect the browser to the ThousandEyes login page automatically. Previously, the platform would require the user to click the OK button on a popup before being redirected to the login page in order to reauthenticate.
2022-03-30 Bug Fixes
Fixed an issue where some BGP origin verification alerts were not firing appropriately, due to a mismatch in the active vs. inactive state of the monitored prefix in the ThousandEyes control plane. This has been corrected.
Fixed an issue with test labels in alert rules, where the alert webhook was incorrectly sending additional test labels via the JSON payload that were not associated with the alert or test.
Fixed an issue in the sankey diagram on the Application Outages view, Topology tab. Previously, the link hover-panel in Internet Insights Application Outages displayed information in the context of all tests seen by the agent group. This has been corrected to display information only for the affected tests seen by the agent group.
2022-03-24
Introducing Webex Cloud Agents
ThousandEyes has now deployed Webex-specific Cloud Agents at the front door of several Webex data centers responsible for serving customer traffic. These agents are available for general use by all ThousandEyes customers and are a tool to better monitor and troubleshoot your Webex performance.
For more information on how to use these new Webex Cloud Agents, see the product documentation.
2022-03-16
Provider Labels for Internet Insights
ThousandEyes provider labels are now available for use with Internet Insights. Customers can label providers (including by region), and use these labels to customize the Internet Insights overview, views, alerts, dashboards and reports, as well as track and maintain labels within catalog settings.
Improved Snapshot Experience for Reports
The snapshot and download experience for ThousandEyes reports has been improved. Snapshot options can now be accessed via the Camera button on the top configuration bar in Reports. This will allow you to create a snapshot, as well as view all the most recent snapshots in your account group and all of the related snapshots to the selected report.
In addition, we have moved the Download button to the top configuration bar in Reports.
API
The Dashboard object in v6 of the publicly facing API can now be accessed by customers:
GET /dashboards: Returns object with list of dashboards available for the user.
GET /dashboards/{dashboardId}: Returns object with the requested dashboard ID.
GET /dashboards/{dashboardId}/{widgetId}: Returns object with dashboard widget data for requested widget and dashboard ID.
POST /dashboards/{dashboardId}/delete: Deletes the given dashboard.
Minor Enhancements
Improved the documentation for the Endpoint Agent update label API to include descriptions of the endpointAgents structure.
A new column, called Data Type, has been added to the provider coverage map in the Internet Insights Catalog Settings > Packages and Catalog Settings > Providers tab so that customers can distinguish whether individual covered providers in the catalog will appear in network or application outages.
The Test Table widget search box is now always visible in the widget, and has moved to the left hand side, to the right of the widget title.
2022-03-16 Bug Fixes
An issue that caused the Alert List widget to not populate configuration options correctly has been resolved.
Previously, the Last Contact column of the Device Settings table was incorrectly displaying the time the last connection attempt was made, instead of the time the device was last seen online. This has been resolved. In addition, the last contact attempt information has been added to the Device Details side bar.
An issue was found where Endpoint snapshots were missing page load and response times after a data migration to a higher-performance data store. This was resolved and all relevant data is now available.
An issue that prevented data being loaded in the Local Network view when Endpoint Agent tests had larger numbers of Endpoint Agents in the path visualization has been resolved.
An issue was found where some Endpoint Agent time periods were incorrectly displayed as "01:00" during winter, specifically for the Brazil/Sao Paulo timezone. This has been resolved.
An issue where alerts were incorrectly fired during an active alert suppression window has been resolved.
An issue where the public IP visible in the Web UI did not match the sourceAddr attribute in the user sessions list API when network traffic is proxied has been resolved.
An issue was found where a focus handling issue on the Endpoint Agent Scheduled Tests view caused the drop-down selection list of scheduled tests to disappear when scrolling through the list using the scrollbar. This has been fixed.
An issue was found in the Endpoint Agent Browser Sessions view, where a Connection Failed error message was displayed when showing results for a visited site, if the network traffic was proxied on the server side. This has been corrected.
2022-03-03
Changes to Dynamic Baselines
We are in the process of improving our dynamic baselining functionality for alert rules. Part of this improvement is simplifying the configuration process and ensuring that our alerts are firing on real issues and not false positives.
In order to do this, we are removing the option to select a sliding window for your dynamic baseline. Dynamic baselines will be based on a 24-hour period. If a user selects a dynamic alert rule based on a standard deviation, this standard deviation will always be based on a 3-hour period. You will still be able to select a percentage, standard deviation, or absolute value above the dynamic baseline in order to trigger an alert.
For the time being, all existing alert rules will maintain their existing configuration. However, if a user attempts to create a new alert rule or edit an existing alert rule with dynamic baselines, this new 24-hour baseline and 3-hour standard deviation will be required.
30 days after the release of this feature (approximately March 30), we will migrate all alert rules to this new configuration. No customer action is required.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Charlotte, NC, USA (IPv6)
Cleveland, OH, USA
Cleveland, OH, USA (IPv6)
Geneva, Switzerland
Geneva, Switzerland (IPv6)
Irvine, CA, USA
Irvine, CA, USA (IPv6)
Nashville, TN, USA
Nashville, TN, USA (IPv6)
Pittsburgh, PA, USA
Pittsburgh, PA, USA (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Private BGP Collectors
The ThousandEyes private BGP collectors listed below will be retired on March 30th, 2022 at 18:00 UTC, and all private BGP monitors currently associated with these collectors will be removed.
BGP Collector
IPv4
IPv6
bgpc1.thousandeyes.com
192.150.160.138
2620:c2:4000:128::138
bgpc2.thousandeyes.com
192.150.160.135
2620:c2:4000:128::135
bgpc3.thousandeyes.com
192.150.160.136
2620:c2:4000:128::136
bgpc4.thousandeyes.com
192.150.160.137
2620:c2:4000:128::137
bgpc5.thousandeyes.com
192.150.160.139
2620:c2:4000:128::139
bgpc6.thousandeyes.com
192.150.160.140
2620:c2:4000:128::140
bgpc7.thousandeyes.com
192.150.160.144
2620:c2:4000:128::144
bgpc8.thousandeyes.com
192.150.160.145
2620:c2:4000:128::145
bgpc9.thousandeyes.com
192.150.160.146
2620:c2:4000:128::146
bgpc10.thousandeyes.com
192.150.160.147
2620:c2:4000:128::147
bgpc11.thousandeyes.com
192.150.160.148
2620:c2:4000:128::148
bgpc12.thousandeyes.com
192.150.160.149
2620:c2:4000:128::149
bgpc13.thousandeyes.com
192.150.160.150
2620:c2:4000:128::150
bgpc14.thousandeyes.com
192.150.160.151
2620:c2:4000:128::151
bgpc15.thousandeyes.com
192.150.160.152
2620:c2:4000:128::152
bgpc16.thousandeyes.com
192.150.160.153
2620:c2:4000:128::153
bgpc17.thousandeyes.com
192.150.160.154
2620:c2:4000:128::154
bgpc18.thousandeyes.com
192.150.160.155
2620:c2:4000:128::155
If you are currently using the private BGP collectors above, you should have received multiple emails starting in December 2021 with instructions on how to update your configuration to peer with our new BGP collectors. If you did not receive the emails or need help making the change, contact ThousandEyes Customer Engineering.
2022-03-03 Bug Fixes
Fixed an issue where our alerter was incorrectly using the page load test interval for HTTP test data when a page load test was executed, although these intervals can be set up independently. This was causing alerts to fire incorrectly due to an incorrect round count.
We upgraded the Endpoint Agent client to support changes in how WiFi metrics are exposed in macOS 12 devices. [Endpoint Agent client version 1.100.0]
Fixed an issue with the 1.99 Endpoint Agent client release, which had led to some customers observing increases in latency metrics when running TCP tests (but not ICMP tests). This issue is resolved in the 1.100 release. [Endpoint Agent client version 1.100.0]
Fixed an error in calculating the link delay between two nodes on the local network visualization. The link delay is now correctly calculated and displayed as delay(hop) - delay(hop-1).
2022-02-15
Intel NUC 11 Support
ThousandEyes now supports installing Enterprise Agents on Intel NUC 11 devices. For more information on installation processes and hardware requirements, see Installing a Physical Appliance.
Intel NUC 8 No Longer Supported
Intel NUC 8th generation and older devices are no longer supported. For assistance with upgrading, contact ThousandEyes Support.
Support for 100 KB Transaction Scripts
Transaction test users can now author scripts that are up to 100,000 characters long (approximately 100 KB). Previously the limit was 64,000 characters.
Report Server IPs for Connection Errors
ThousandEyes now reports server IPs in cases that observe HTTP failures during the TCP connection phase. Also, for failures detected at Layer 4, the server IP is set to the last attempted last IP. This IP can be one of following:
the network test target URL
the proxy IP
or one of the IPs that failed to connect on the last redirect.
New Cloud Agents
New Cloud Agents have been added to the following locations:
Ahmedabad, India (JIO)
Ahmedabad, India (JIO) (IPv6)
Hyderabad, India (JIO)
Hyderabad, India (JIO) (IPv6)
Omaha, NE, USA
Omaha, NE, USA (IPv6)
Wan Chai, Hong Kong (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Minor Enhancements
For Endpoint Agent path visualization, in order to better represent an unknown number of hops between the penultimate node displayed in the path trace and the destination, we now display the following between the penultimate node and the destination:
? when the pathtrace reached the destination.
x when the pathtrace did not reach the destination.
2022-02-15 Bug Fixes
An issue was found in the Endpoint API where the pagination of result sets for user sessions was incorrect. This has now been fixed and made consistent with other data types.
Fixed an issue where the Alerts API was not responding with the Agent Name in the Agent array section of the response.
2022-02-01
Minor Enhancements
In order to improve our user experience, the drag icon and the administrative icons on dashboard and report widgets are now only displayed on hover, instead of by default. Additionally, customers can now tab through the administrative icons with keyboard controls.
In Endpoint Agents > Agent Views, when there is no agent to display, the "Getting Starting with Endpoint Agents" page is now displayed.
For Endpoint Agent path visualization, in order to better represent an unknown number of hops between the penultimate node displayed in the pathtrace and the destination, we now display the following between the penultimate node and the destination:
? when the pathtrace reached the destination.
x when the pathtrace did not reach the destination.
2022-02-01 Bug Fixes
An issue was resolved that caused HTTP 500 errors to be thrown for a user when they tried to delete their default dashboard.
An issue was resolved where the Endpoint Agent Browser Sessions view displayed a blank window when viewing agent-specific browser session data, rather than the relevant data for the agent accessing those websites.
An issue was found in device-layer monitoring, where a device was submitting interface discard metrics, but was not unresponsive to device status calls that the Enterprise Agent was making to the device. This resulted in a discrepancy in Device Layer views where the interface discard metrics were displayed on the timeline, but not in the Table view. This has been resolved by allowing the interface metrics to be displayed in the table, but noting to users that the interface status is "Unknown".
An issue has been resolved that caused some values to incorrectly show as "No Data" rather than "O" in Snapshot report widgets.
An Endpoint Agent installation issue that prevented installation on macOS if Google Chrome or Microsoft Edge were not installed has been resolved.
An issue was found where the "Go to views" option in the Endpoint Agent > Test Settings page was not displayed for users with the "Regular User" role. This has been resolved.
An issue where an Endpoint Agent TCP test could show jitter and latency values despite 100% packet loss has been resolved. This occurred because in SACK mode, the round-trip times from the TCP handshake counted towards the reported latency values.
An issue was resolved where dashboard and report widgets incorrectly showed no data when an agent label filter that only included Cloud Agents was selected.
Several BGP alerts that were stuck in a triggered state due to stale BGP monitors have been cleared.