Changelog
ThousandEyes' BrowserBot now runs its Chromium browser in headless mode for page load, transaction (classic), and transaction tests. This change was made to remove the dependency on an X11 server and reduce resource utilization on Cloud and Enterprise Agents. In addition, the Chromium browser will now be restarted between test rounds, rather than BrowserBot attempting to reuse the same browser instance across multiple test rounds.
Two test configurations will still result in Chromium launching the browser without headless mode:
Kerberos is used for HTTP authentication.
A PAC script URL is used for proxy configuration.
All other configurations will run in headless mode.
A future release will remove te-browserbot's dependency on an X11 server. This release maintains the dependency, but transitions ownership fo the X11 server from the te-browserbot package to te-xvfb (a new package added for this purpose). In future releases, te-xvfb will be an optional package.
While this change will not impact most users, two use-cases are known to be impacted:
Transaction scripts that use the "Switch to Next Tab" example will need to be updated: switchToNextTab.js​
Transaction scripts that use Chromium's in-browser PDF viewer will need to use downloads.waitForDownload() to have their script wait for the PDF download to complete.
For more information or assistance with these use cases, contact ThousandEyes Customer Engineering.
ThousandEyes has replaced the existing ten-character key that identifies an embedded widget with a 128-bit universally unique identifier (UUID). This change was made to provide additional security to our embedded report and dashboard widgets, and will impact all widgets created from this update onwards.
Existing embedded widgets will not be impacted by the change.
To update an existing widget to use a 128-bit UUID, uncheck the "Allow anyone to embed this widget to external sites" checkbox on the widget's configuration panel, and then re-check it.
ThousandEyes platform health announcements through https://success.thousandeyes.com will be discontinued on October 15th, 2020. Future health and maintenance announcements will all be made through the new https://status.thousandeyes.com site.
You can receive notifications via e-mail, SMS, RSS, and Twitter by subscribing at https://status.thousandeyes.com.
For more information, contact ThousandEyes Customer Engineering.
To improve performance when a report or dashboard widget is created with a large Endpoint Agent data set (either due to no applied filters or too large of a time span), users are now asked to either apply an endpoint test filter or reduce the time span in order to render the data.
This change impacts existing widgets as well as new widgets, but it does not impact existing snapshots.
For details, see [the product documentation[(https://docs.thousandeyes.com/product-documentation/reports/working-with-reports#render-limit).
An issue was found in the Browser Sessions Network view where the domain filter drop-down menu incorrectly opens when clicking on a destination domain that has been grouped by domain. This has been resolved.
An issue was found where the path visualization for an Endpoint Agent scheduled test displayed inconsistent data compared to other views. This has been resolved.
The Endpoint Agent ping test timeout has been increased from 11 seconds to 15 seconds to resolve a couple of edge cases.
Several edge cases were found that prevented Endpoint Agents from terminating when requested. These have been resolved.
An issue was found where changing a Map widget to display in two columns instead of one distorted the widget configuration panel. This has been resolved.
Previously, users that entered multiple domains when configuring DNS alert rules did not receive an error message if they used an unsupported delimiter. This has been resolved, and customers can now use space and comma delimiters when entering values into the following alert conditions: OriginAsn, NextHopAsn, Prefix, BgpHopAnyAsn, DnsMapping, Domain, ASN, IP, RDNS, and TargetIP.
An issue was found where the platform treated the same domain name as two separate domains if the capitalization was different. This has been fixed, and capitalization no longer impacts the domain name.
ThousandEyes now supports installing Enterprise Agents on Red Hat Enterprise Linux (RHEL) and CentOS 8.1+ operating systems.
By the end of the fourth quarter of 2020, ThousandEyes will complete a relocation of our infrastructure to Amazon Web Services. Some ThousandEyes customers have access control lists (ACLs) or firewall rules which permit Enterprise Agents, Endpoint Agents or other resources to communicate specifically with the ThousandEyes infrastructure. These ACLs/rules will need to be updated with new IP addresses.
Customers who use such ACLs or rules should watch this changelog for future instructions on updating their rules. Additionally, each customer's account mangagement team will will reach out to ensure a smooth transition. In the meantime, feel free to contact ThousandEyes Customer Engineering with questions or concerns.
ThousandEyes uses publicly available BGP monitors to provide data for our BGP tests. To achieve a higher granularity of visibility, we have increased the number of public monitor vantage points accessible for tests to 80. For more information on public BGP monitor data and route view collectors, see Working with Raw BGP Data.
Several updates have been made to the Internet Insights filters:
Users can now filter by server location and server prefix, in addition to the existing interface location and interface IP address filters.
Filters are now applied at an affected interface level, rather than at an outage level.
A topology filter has been added for the Server Location to allow users to filter the topology view by location (for example, by country).
In addition, the Add a filter menu has been reorganized to clarify the following options:
The Affected Interfaces, Destinations, and Other sections are now the Network, Application, and Outage sections respectively.
The In Catalog option has been removed.
The default outages filter is now Outage Scope, and listed under Outages.
The Catalog Provider filter has been renamed Affected Provider.
The Location and IP Address filters in the Network section have been renamed as Interface Location and Interface IP Address to distinguish them from the new server location and server prefix options.
The Affected Test filter has been moved to the Application section.
The Domain and Network sections have been renamed Affected Hostname/Domain and Server Network respectively.
Existing snapshots have been updated to include the new filter options and functionality. Depending on the filters selected in the existing snapshot, views may appear to be more narrowly scoped due to interface-level filtering, but the snapshot data is unchanged.
The voice call test type has now been removed. Voice call tests can be migrated to separate SIP and RTP tests without incurring additional unit consumption. For more information about SIP and RTP test configuration, see SIP Server Test Settings and RTP Stream Test Settings. Contact Support for more information.
An issue was found after adding the render limit pop up notification to reports and dashboard widgets, where if both the render limit notification and the existing BGP size notification were triggered, only one would display. This has been resolved.
Previously, the Service Now alert integration was incorrectly avaiable for all alert types. This has been corrected, and the alert integration is only visible for the supported alert types: Cloud and Enterprise Agent, BGP, Endpoint, and Internet Insights.
Data from Cloud Agents with local problems can now be excluded from report and dashboard widgets by clicking the Exclude Data from Cloud Agents with Local Problems checkbox in the widget's configuration.
For more information, see Customizing your Dashboard.
AppDynamics is now supported as an alert integration type for all Cloud and Enterprise Agent and BGP alert data sources. For more information, see AppDynamics Integration.
The following Endpoint Agent scheduled test alerts are now available in the Alert Grid widget:
End-to-end (server)
Endpoint HTTP server
Endpoint path trace
Two new filter options are now available when searching for interfaces or devices:
The IP Address filter uses the mapped IP address to find associated interfaces and devices.
The Test filter allows customers to find interfaces and devices that were traversed by the selected test(s) over the last 30 days.
An issue was found where the estimated usage for BGP tests was not calculated correctly for the current and next billing cycles. This has been resolved.
Important: The actual BGP test usage was still captured and reported correctly.
An issue was found where the loss, jitter, and latency for an Endpoint Agent network connection displayed inconsistent data if the connection had failed. The loss would show 9%, while the jitter and latency were shown to be < 1ms. This has been resolved, and the correct values are now displayed if the connection fails.
An issue was found where Endpoint Agent browser session tests were not gathering network information when a proxy server was part of the network. This has been resolved.
An error was found where a log entry was incorrectly referencing an "unsupported device". The correct system error is now logged instead.
An issue was found where Endpoint Agents running macOS 10.9 were not reporting wireless information. This has been resolved.
In some instances, the Download button for report snapshots was not working. This has been fixed.
An issue occurred when creating a report snapshot with the v7 API, where the incorrect time range was shown in the UI when compared to the inputted request. This has been fixed.
ThousandEyes now allows customers to configure higher-frequency browser-based tests:
Page load test intervals can now be reduced to one minute.
Transaction test intervals can now be reduced to two minutes.
For more information, see Working with Test Settings.
ThousandEyes has added the ability to disable viewing of stored transaction test credentials by any user, regardless of permissions.
Previously, any user with the View sensitive data in web transaction scripts permission (including, by default, an Organization Admin or Account Admin user) was able to reveal stored credential values in the Credentials Repository.
Users with the Edit security and authentication settings permission (by default, only those in the Organization Admin role) can now select a new Disable global credential retrieval checkbox to override the View sensitive data in web transaction scripts permission, and disallow user credential access via the web app and the API.
To configure the new option:
Navigate to Account Settings > Organization Settings > Security and Authentication.
Scroll down to the Credentials Repository section, and select Disable global credential retrieval:
Changes to the Disable global credential retrieval setting are logged under Account Settings > Activity Log.
An issue occurred on the Scheduled Tests view for Endpoint Agents, where the settings displayed when hovering over an agent label were inconsistent with the agent's settings. This has been resolved.
An error occurred when validating the proxy setting URL for Endpoint Agents. This has been resolved, and now any URL can be used as the PAC URL target.
In some instances, the Download button for report snapshots wasn't working. This has been resolved.
You can now configure up to 10 scheduled tests per Endpoint Agent. This higher limit applies to both Endpoint Agent and Endpoint Agent Pulse.
When you edit a transaction script, you can now expand the editor window, for a more ergonomic coding experience.
When you create a report or dashboard widget that uses BGP data and does not use filters, you are now asked to confirm that you want to render the data. This change is made to improve performance when a widget selects a large data set.
This change impacts existing widgets as well as new widgets, but it does not impact existing snapshots.
The following Cloud Agent locations now have IPv6 support:
Mississauga, Canada
Brno, Czech Republic
For a complete list of ThousandEyes Cloud Agent locations, see https://www.thousandeyes.com/product/cloud-agents.
Changing the alert conditions on an existing alert rule inadvertently removed the assigned tests from the alert rule. This issue has been resolved.
In Agent Settings, when you sort the list by Last Contact in ascending order, the list now displays disabled agents last.
An issue sometimes occurred in Internet Insights where the Select All checkboxes in filters could reach a selected state without all options being selected. This issue has been resolved.
In some rare cases, the Endpoint Agent failed to check in when connected over VPN and when a specific HTTP error code had been returned for previous connection requests. This issue has been resolved. With this fix, the Endpoint Agent is now always able to check in when there is a clear Internet connection (VPN or otherwise).
Previously, some transaction tests could time out spuriously on certain agents without any apparent cause. ThousandEyes has determined that the root cause of the issue was variability in certain phases of transaction test setup, and has modified BrowserBot to account for time differently, such that these spurious timeouts cease.
An issue was found where changing the alert conditions on an existing alert rule inadvertently removed all of the assigned tests from the alert rule. This issue has been fixed.
Webhooks using HTTPS endpoints now require that the TLS SNI matches one of the webserver certificate's subject alternative names (SANs). ThousandEyes will reach out to customers that we believe will be directly impacted by this change.
Custom static key-value pairs can now be added to the webhook payload as part of the Alert Key in the Body section of a supported webhook. These custom payloads can only be used on the Cloud or Enterprise Agent alert type, and will be specific to the selected alert rules. This change also adds two new fields to the webhook payload:
testLabels: The test label name and ID.
testTargetsDescription: The description of the test target as defined in the test configuration.
In addition, the Authorization Code grant type has been added to the OAuth workflow for webhooks.
In order to better support Palo Alto and Juniper firewalls in the path visualization, ThousandEyes has added the capability to perform a path trace "in session" within an agent to agent test. This initiates a TCP session with the target server, and sends path trace packets within that session.
The new option can be configured by selecting the In Session checkbox on the Basic Configuration page of the test, under the Path Trace Mode setting:
The Test Table widget has been updated to exclude disabled tests by default. Additionally, a checkbox option has been added to the widget configuration that allows users to include disabled tests in the Test Table widget if needed.
A new button has been added to the script editor in the transaction test settings that will add a sleep command to the transaction script where the cursor is positioned.
An issue occurred when an Endpoint Agent user tried to access a share link immediately after creating it. In some instances, this would result in the user seeing gaps in the data, as they were viewing the share link before all the data had been indexed. This issue has been resolved.
An issue occurred where some Endpoint Agents connected via WiFi showed no wireless information in the Local Networks views. This has been resolved.
Previously, the best-match logic used when searching the Internet Insights catalog on the Catalog Settings page resulted in unexpected results when searching for short terms (for example, ISP). This has been resolved.
When configuring Internet Insights outage alert rules, in some instances, the validation errors did not provide enough information to identify the field that did not validate. This has been resolved.
An issue occurred where many transaction tests would periodically appear to complete successfully (reporting a full waterfall and duration), but end up reporting as having timed out. This was caused by a step in the inter-test setup running over the allocated duration, which counted against the user's configured test timeout time. This has been corrected.
BrowserBot 1.132, which is used by Cloud and Enterprise Agents to conduct page load and transaction tests, includes an upgrade of the te-chromium browser from version 68 to version 80. Users may observe changes in test behavior due to the browser upgrade.
Red Hat Enterprise Linux and CentOS environments now require java-1.8.0-openjdk rather than java-1.7..0-openjdk.
Debian-based agents are not affected by this update as they already use java-1.8.0-openjdk.
The COVID-19 virus is putting a huge strain on IT and network teams all around the world. To assist with managing your end-user experience, ThousandEyes is offering free access to our End User Monitoring until July 31, 2020. If this is of interest to you, reach out soon — enrollment for this offer ends on June 30, 2020. For more information, please see our blog or contact the ThousandEyes team.
The default BGP alert rule has been updated to fire when 10% of monitors have less than 100% reachability.
Note: This change will be applied moving forward and retroactively, but will only impact the default BGP alert rules that have not been updated by a customer.
Note: These changes will occur in sequence; initially, the change will be made for new rules, and then in the next couple of weeks, the changes will be made retroactively.
Transaction scripts can now contain up to 200 defined markers.
The Waterfall tab now includes a Show More link for when there are more than 10 markers to display.
Internet Insights
The number and ID of affected tests reported in outage alerts are now available via the alerts API. See v7 API documentation.
Added more specific tooltip titles to the metrics displayed in the timeline for Endpoint Agent local networks.
An issue occurred with the v7 Dashboard API that was allowing users to create, update, or delete dashboards using the GET method instead of the appropriate HTTP method POST. This has been fixed.
An issue was found where gaps could appear between the page bars shown in the Waterfall view when browser-based tests targeted certain pages. This was caused by missing start/end time requests for those pages, and has been resolved.
An issue occurred where transaction tests would periodically appear to complete successfully (reporting a full waterfall and duration), but would also report as having timed out. This was caused by part of the inter-test setup overrunning its allocated duration, and counting against the user's configured test timeout. This has been resolved.
An error occurred when transaction scripts that used the waitForDownload() function were configured to use an HTTP proxy. In these circumstances, the test would fail. This was caused by the way waitForDownload() had been implemented, and has been resolved.
An error occurred when transaction scripts were configured to use an HTTP proxy where the proxy would receive periodic HTTP requests on port 9999 that were not intended to leave the agent. This was caused by the way the waitForDownload() function had been implemented, and has been resolved.
Previously, the outage swimlane tooltip would sometimes cover the Internet Insights Views timeline selection, making it difficult to click on the timeline. The tooltip positioning has been changed and the issue is now resolved.
An issue occurred that required users to have the View alert rules permission in order to view the Endpoint Agent Views and Test Settings pages. This has been resolved.
An issue occurred where agents remained disabled after a customer's trial period was extended. This has been corrected.
The COVID-19 virus is putting a huge strain on IT and network teams all around the world. To assist with managing your end-user experience, ThousandEyes is offering free access to our End User Monitoring until July 31, 2020. If this is of interest to you, reach out soon — enrollment for this offer ends on June 30, 2020. For more information, please see our blog or contact the ThousandEyes team.
Previously, the scheduled tests assigned to an agent were randomly selected from the list of pre-assigned tests. Now, tests containing a label that matches specifically to an agent are prioritized ahead of all other tests when they are assigned execution slots.
The URL and network target can now be edited in the Scheduled Test Settings form when you edit a test.
Reachable VPN nodes are now displayed on the path visualizations created for scheduled tests.
Endpoint Agent browser session alerts are now supported in the Alert Grid and Alert List widgets.
An issue occurred where the Markers filter was showing duplicate markers if a saved event shared the same name as the test it was created from. This duplication has been resolved.
An issue occurred that caused public dashboards and reports to be deleted when the original creator was deleted from the platform. Dashboards and reports now remain within the appropriate account group even when a user is deleted.
An issue caused embedded links to be deleted when a user updated a dashboard or a report through the API. This has been resolved.
In some instances, the Download button for report snapshots was grayed out and non-functional. This has been resolved.
An issue in the registration process caused the process to abort when it encountered network errors. This has been resolved.
The Headers link was missing under the Response code column in the Table view for Endpoint Agent HTTP server tests. It is now displayed correctly.
An issue sometimes occurred where the Affected Tests filter in Internet Insights would display a "loading" animation indefinitely and the test list would never complete loading. This issue has been resolved.
The COVID-19 virus is putting a huge strain on IT and network teams all around the world. To assist with managing your end-user experience, ThousandEyes is offering free access to our End User Monitoring until July 31, 2020. If this is of interest to you, reach out soon — enrollment for this offer ends on June 30, 2020. For more information, please see our blog or contact the ThousandEyes team.
ThousandEyes has expanded our Enterprise Agent deployment options to include support for Raspberry Pi 4. A new download option is now available, along with installation instructions, on the Agent Settings page.
Browser-based tests are not supported when running an Enterprise Agent on Raspberry Pi 4.
Updated Agent Statistics Tab
The Agent Statistics tab in the Agent Settings view has been revamped to provide per test contribution to agent utilization. The tab now includes the host account group, test type, average utilization, and max utilization of each test to provide greater context.
In addition, agent uptime and utilization metrics are available over a longer time period (up to 30 days).
The location filter on the Internet Insights Views page has been updated. Locations are now grouped by area and country, allowing users to search more effectively.
ThousandEyes has added a new Cloud Agent with IPv6 capabilities in Buenos Aires. For a complete list of our Cloud Agent locations, see https://www.thousandeyes.com/product/cloud-agents.
ThousandEyes is deprecating the voice call test type on August 31, 2020.
Voice call tests simulate a full Voice over Internet Protocol (VoIP) call sequence, starting with the session initiation using the Session Initiation Protocol (SIP), and then the voice data transmission using the Real-time Transport Protocol (RTP) between one or more ThousandEyes agents and a target agent, acting as the VoIP user agents (or the call endpoints).
A voice call test can be migrated to separate SIP and RTP tests without incurring additional unit consumption. For more information about SIP and RTP test configuration, see SIP Server Test Settings and RTP Stream Test Settings. Contact support for more information.
An issue occurred where device views would become unresponsive when navigating from the Alerts List page, or from alert notification emails. This has been resolved.
Under certain conditions, the certification path displayed was not the certification path used by the test's TLS connection. The way the certificate chain is shown in the Response Details panel has been changed to show the more commonly used certificate chain in the case of cross-signed certificates.
An issue in the Test Settings page was found where the bypass list configured in proxy settings would bleed across the test settings box if there were greater than 4 entries. This has been resolved.
An issue was found in Device views where the "interface" filter selection did not auto-populate all available options a user would typically be able to select. This has been fixed.
An issue occurred where inconsistent times were displayed on report widgets based on the users timezone. This has been updated to accurately reflect the time regardless of timezone.
In rare production circumstances, a previously identified and fixed issue that resulted in a report snapshot being emailed twice instead of once would reappear. Additional checks have been added to ensure the issue is now resolved.
An issue with test table widgets existed where user-created labels were not showing up as filterable options. This has been corrected.
The color grid widget filter option was not working properly, and has been fixed.
An issue occurred where the filter values selector for individual widgets was enabled before a metric was selected. The filter values selector is now only available when a metric has been selected.
An error occurred that resulted in an agent being displayed multiple times in a filter. This has been resolved.
An issue occurred where Endpoint Agents without data reported as 100% data loss in dashboards. This has been resolved.
A race condition prevented setting changes from being applied to some Endpoint Agent tests. This has been resolved.
An issue occurred where changing the Endpoint Agent network topology view grouping could cause the web UI to hang. This has been resolved.
An error occurred when clicking the "Show Network Topology of this Agent" link for an Endpoint Agent that resulted in a perpetual loading icon. This has been resolved.
An issue occurred where an error was displayed when attempting to modify a label. This has been fixed.
We have resolved an issue in Internet Insights where in some cases, when creating an outage alert rule, the "Catalog Providers" dropdown menu had a blank entry.
An issue causing animations on the Internet Insights Overview page to occasionally consume 100% CPU has been resolved.