Changelog
As previously announced, ThousandEyes is transitioning its infrastructure to Amazon Web Services. ThousandEyes customers may have access control lists (ACLs), firewall rules, or other configurations (for example, /etc/hosts files with hard-coded IP addresses) that permit Enterprise Agents, Endpoint Agents, or other resources to communicate with the ThousandEyes infrastructure. These configurations must be updated with new IP addresses by November 15th, 2020. For instructions, see the article ThousandEyes Infrastructure Changes.
If you have questions or concerns, contact ThousandEyes Customer Engineering.
The Proxy Loss, Proxy Latency, and Proxy Jitter metrics are now available for view in dashboards and reports.
In addition, users can now see additional proxy details such as Proxy Name, Proxy IP, where the proxy setting is applied, and the type of proxy in the HTTP Server view.
The ThousandEyes Raspberry Pi installation image has been updated to support the 8GB version of the device.
The ThousandEyes virtual appliance installation image has been updated to use Ubuntu Bionic 18.04.5 LTS as the base operating system. All new virtual appliance installs will run using Ubuntu Bionic.
A new IPv6 Cloud Agent has been added to Indore, India. For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Widgets can now be removed from a report or dashboard by dragging them directly into a deletion zone (i.e., a trash can). This action can be undone immediately after deletion.
An alerting logic issue prevented BGP alerts from firing correctly, as setting the trigger threshold to 100% of monitors was detecting either IPv4 monitors or IPv6 monitors only, rather than the combined total. This has been corrected.
The Color Grid widget in reports and dashboards was sometimes displaying the wrong color based on the scale provided. This has been resolved.
General HTTP tests were not offering network alerts even if the setting to collect network measurements was on. This has been corrected.
A broken “Show Traces using this VPN" link in Endpoint Agent network tests has been fixed.
An issue prevented newly created Endpoint Agent test labels from showing up in the Endpoint Agent test list, and made them unavailable for searches. This has been resolved.
In instances where no data was collected when looking at Endpoint Agent wireless data, the data loading icon would not disappear. This has been resolved, and the correct message that no data was found is displayed.
An issue was found where clicking the Show Only This Agent link while interacting with the path visualization would not display pagination correctly. This has been corrected.
The API response format for web layer data requests was inadvertently changed in a previous release. The previous documented response format has been restored.
An issue was found where customers were unable to install Enterprise Agents in a VMware ESXi environment due to an incompatibility in the compression algorithm used. This is now resolved, and customers can install as expected.
ThousandEyes has updated the reports and dashboards user interface, widget icons, and workflow to substantially improve user experience, and ensure that widget addition and configuration is as streamlined as possible:
Widgets can now be added by clicking the + Add Widget button to open the Add a Widget panel, and then either selecting a widget to configure, or by dragging it to the desired location in the report or dashboard.
The Add a Widget side panel is now movable and minimizable, allowing users to review how the added widget looks before saving.
Changes to a widget can be previewed before saving. Clicking Cancel will revert the widget to its pre-configuration state, while clicking Save will confirm the new configuration.
As part of this update, the Manage Widgets section has been removed.
For more information on working with reports and dashboards, see:
​Working with Reports​
Internet Insights customers can now add widgets to reports and dashboards to leverage Internet Insights network outage metrics. The network outage metrics provided are:
Affected Tests: Test data within your organization that is affected by outages.
Interfaces: Interface data affected by outages.
Locations: Location data affected by outages.
Outages: Overall outage data.
Widgets built on Internet Insights metrics can apply the following filter conditions: Catalog Provider, Outage Scope, and Network; Interface Location and Interface IP Address; Affected Test and Affected Domain; and Server Location, Server Prefix, and Server Network.
To add a widget with Internet Insights metrics:
In Reports or Dashboards, click + Add Widget and select a supported widget.
Select Internet Insights as the datasource.
Select Network Outages as the category.
Select a metric and measure (and filters if desired).
Internet Insights metrics are not available for use with the Stacked widget types (Stacked Bar, Stacked Area, and Pie).
A new pre-built dashboard that captures weekly and monthly outage trends has been added for Internet Insights customers. The dashboard, called Internet Insights Built-In, also provides an up-to-date view of the most impacted tests, locations, and providers.
The default dashboard can be duplicated and used as a starting point for building custom dashboards.
The Endpoint Agent now integrates with the Pulse Secure VPN, providing end-to-end visibility of network nodes and metrics for traffic that traverses the network. When present, the client will populate the VPN Vendor attribute visible in Endpoint Agent views and filters.
Device discoveries now report devices that failed discovery due to a lack of working credentials. The Edit Device modal has also been updated to show the used credentials on the device, and provide users with easy access to new credential creation:
A hard limit of 700 (for example, 50 tests over 14 days, or 100 tests over 7 days) has been added to improve performance when a report or dashboard widget is created with a large BGP data set (either due to no applied filters or too large of a time span). Users are now asked to either apply a BGP test or monitor filter, or to reduce the time span in order to render the data, if it would otherwise exceed the hard limit.
The Internet Insights alert rules functionality has been updated for when both Catalog Providers and Affected Tests are set to Any. Previously, this configuration would only trigger if a customer test was impacted by an outage. The configuration has now been updated to trigger if a subscribed provider is impacted, in addition to if a customer test is affected by an outage in a subscribed provider.
To alert on outages affecting only customer tests (including in providers the customer is not subscribed to), customers should set Affected Tests to Specific, and select all tests, or the specific set of tests you want the rule to match.
This change has been made to all existing alert rules.
In order to simplify test selection in Cloud and Enterprise Agent views, saved events have been removed from the test selector. Saved events can be accessed from the Sharing > Saved Events page in the main menu.
An issue was found where path trace data was not appearing in Endpoint Agent sharelinks. This has been resolved.
You can now gather network data and path-visualization data about proxy servers.
In the past, if your target was behind a proxy, your ThousandEyes tests would not collect network data. Workarounds existed (adding the target to the proxy's bypass list, or creating separate agent-to-server tests to the proxy), but these were cumbersome to configure. And with the popularity of cloud proxy solutions such as Zscaler, Netskope, and Palo Alto Networks, you may want to understand network health to the cloud proxy.
With the ability to run network tests to the proxy, you can more easily correlate data between the application and network layers. The new ThousandEyes proxy capabilities allow you to see
Proxy loss
Proxy latency
Proxy jitter
A visualization of the network path to the proxy
To explore the new proxy metrics:
Go to Cloud & Enterprise Agents > Test Settings.
Select or create an HTTP, page load, or transaction test.
On the Advanced Settings tab, find the Proxy Settings section.
Select Perform network measurements to the proxy.
​
​
Several improvements have been made to how Endpoint VPN data is visualized:
VPN loss and VPN latency metrics have been added to the Scheduled Tests, Browser Sessions, and Local Networks views.
The following filters have been added to Endpoint Agent Settings and Endpoint Agent labels:
VPN Vendor: The VPN vendor.
VPN Gateway IP address: The public IP Address of the VPN gateway/server.
VPN Client Address: The private IP via which clients route packets to the VPN tunnel.
VPN Client Network: The private network that the VPN client belongs to.
The old VPN Address label filter is still supported, but is marked for deprecation. This label filter is only relevant for filtering data in the views. In Browser Sessions, it will return pages that were accessed through a VPN where the gateway IP address is equal to the filter value. In Local Networks > Network Access, it will return probes to the VPN server where the gateway IP address of the VPN is equal to the filter value.
It will be fully removed from the product in a future release.
From te-browserbot version 1.146 and later, installation of an X11 server (via the te-xvfb package) is no longer mandatory. Existing Enterprise Agents will continue to have te-xvfb installed, unless users take action to remove the package. New Enterprise Agents will have the package installed by default, but this can be avoided using one of the following processes:
For Linux package installations: install_thousandeyes.sh now has a new command-line flag to skip te-xvfb package installation.
For Docker installations: Users may create new images based on the official ThousandEyes image that run
apt remove te-xvfb.For TEVA installations: Users may unlock their appliances and run
apt remove te-xvfb.
While it is now optional, users are still encouraged to keep the te-xvfb package installed. Some page load and transaction test configurations require starting Chromium without the --headless flag, and these situations will require te-xvfb.
Due to a change in service provider, customers with tests assigned to the IPv6 Cloud Agent in Dublin, Ireland will observe a corresponding change in path-visualization and source IP. The Cloud Agent ID used within the ThousandEyes API will remain unchanged. The new IPV6 IP is: 2001:4d40:4003::/48.
Endpoint Agent labels can now be updated via the API.
Users can now create Endpoint Agent scheduled tests via the API.
Previously, the Account group list API endpoint would only return the organizationName if the user was associated with multiple organizations. This has been changed, and the organizationName will now always be included.
The Import from JS File button in Transaction Test Settings has been replaced with a Create New dropdown menu. This dropdown retains the option to choose a local file to import, but also provides a selection of predefined template scripts. Clicking on an option will update the editor with the contents of the selected template script.
An issue was found where duplicate alerts were inadvertently being created, causing one of the alerts to fail to clear after resolution of the issue. This has been resolved.
An issue was found where updating proxy configuration would refresh disabled tasks, causing them to continue to run despite being disabled. This has been resolved. .
An issue was found where Cloud and Enterprise Agents reported no data on the Map view, even if data was being reported as expected. This has been resolved.
The digital certificates for app.thousandeyes.com and downloads.thousandeyes.com, which were previously issued by the DigiCert certificate authority, will be replaced with certificates from the QuoVadis certificate authority on October 13th, 2020. The replacement certificates will use certificate chains with larger key lengths (4096 bits) than the current certificate chain (2048 bits).
Customers who access these servers with clients (browsers, file transfer utilities, custom programs) that use older cryptographic libraries which cannot support 4096-bit key lengths should update their clients. Customers who use supported browsers or other current clients do not need to take any action.
Due to a change in service provider, customers with tests assigned to the Cloud Agent in Paris, France will observe a corresponding change in path-visualization and source IP.
We do not anticipate downtime or impact on test performance. The Cloud Agent ID used within the ThousandEyes API will remain unchanged.
The following Cloud Agent locations now have IPv6 support:
Paris, France
For a complete list of ThousandEyes Cloud Agent locations, see https://www.thousandeyes.com/product/cloud-agents.
Transaction test users may now use a variety of functions that enable them to generate files for upload to applications under test, and to inspect the contents of files downloaded from the same location.
Users can now import an uploads library from the thousandeyes module and use the following functions:
uploads.generateFile(filename, fileContents): Create a file with the given filename and contents, returning a path to the generated file.
uploads.generateRandomTextFile(sizeInBytes, filename): Create a file of the given size with the given filename, returning a path to the generated file. The generated file will be filled with randomly-generated alphanumeric characters.
uploads.generateRandomBinaryFile(sizeInBytes, filename): Create a file of the given size with the given filename, returning a path to the generated file. The generated file will be filled with randomly-generated bytes.
Users can import downloads from thousandeyes and use the following functions:
downloads.waitForDownload(filename): Return the path of a downloaded file once the download is complete.
downloads.getContents(path): Return the contents of the specified downloaded file's path.
downloads.getSize(path): Return the size of the specified downloaded file's path.
downloads.getMD5(path), downloads.getSHA1(path), and downloads.getSHA256(path): Return the hash of the specified downloaded file's contents.
downloads.getPdfTextContents(path): Extract all text from a downloaded PDF file. Customers may use this to verify that the contents of a downloaded PDF file match their expectations.
The AppDynamics integration has been updated to provide a more intuitive user experience. We have updated the AppDynamics Instance URL field to specify that we are looking for protocol://hostname:[port], and have added input validation to this field to strip out any unnecessary information. Additionally, we have added a direct link to our AppDynamics integration documentation to the form itself.
ThousandEyes UI users can now configure whether they want the HTTP layer of page load tests to follow redirects. Previously, this setting was only configurable via the API.
Calling markers.start("x") twice with the same marker value will now cause transaction tests to throw an error.
Transaction test scripts can now
import { test } from 'thousandeyes'and calltest.getSettings()to obtain an object containing certain test settings, including:timeout: The test's timeout in seconds.
interval: The test's interval in seconds.
url: The test's target URL.
proxy: An object describing the test's proxy settings.
Transaction test scripts can now
import { agent } from 'thousandeyes'and callagent.getInfo()to obtain an object describing the agent that is currently executing the script. This object includes the following parameters:
agentName: The agent's name.
agentId: A unique identifier for the physical agent.
hostName: The agent's hostname.
ipAddress: The agent's public IP address.
isRunningInAgent: Boolean value determining whether the script is running in the agent or the Recorder IDE.
User agent strings and window sizes for HTTP, page load, and transaction tests have been updated to include modern and popular mobile devices.
Added tooltips to describe the reasons why the Endpoint Agent experience score is not displayed when a metric is unavailable in the UI.
304 redirects are no longer shown as errors in the Browser Sessions view. They also display the Cached label in the Entry Size column, rather than a warning triangle icon.
Probes directed at an Endpoint Agent VPN gateway are now sent out via a physical network interface (bypassing the virtual interface), to determine the nodes between the agent and the VPN gateway.
An issue was found where the "DNS" timing of a request in the Waterfall view of page load and transaction tests was beeing added to the "Connect" timing instead. This has been resolved, and now a DNS lookup is only represented in the "DNS" timing.
An issue was found where the VPN node in the path visualization for Endpoint Agents was not visible when the agent filter was removed. This has been corrected.
In some cases, transaction tests were found to run longer than their configured timeouts without timing out. This has been resolved, and tests should timeout when intended.
Previously, different criteria was used to filter out Endpoint Agent data points for the timeline and the round data of a scheduled test's network layer. This sometimes resulted in inconsistencies in the UI, and has been resolved.
An issue was found where the metrics summary was occasionally not being updated when switching between rounds of data in Endpoint Agent views. This has been resolved.
An issue was found where the ethernet link speed for very high-speed network interfaces (>= GBit/s network adapters) was incorrectly determined. This has been resolved.
ThousandEyes' BrowserBot now runs its Chromium browser in headless mode for page load, transaction (classic), and transaction tests. This change was made to remove the dependency on an X11 server and reduce resource utilization on Cloud and Enterprise Agents. In addition, the Chromium browser will now be restarted between test rounds, rather than BrowserBot attempting to reuse the same browser instance across multiple test rounds.
Two test configurations will still result in Chromium launching the browser without headless mode:
Kerberos is used for HTTP authentication.
A PAC script URL is used for proxy configuration.
All other configurations will run in headless mode.
A future release will remove te-browserbot's dependency on an X11 server. This release maintains the dependency, but transitions ownership of the X11 server from the te-browserbot package to te-xvfb (a new package added for this purpose). In future releases, te-xvfb will be an optional package.
While this change will not impact most users, two use-cases are known to be impacted:
Transaction scripts that use the "Switch to Next Tab" example will need to be updated: switchToNextTab.js​
Transaction scripts that use Chromium's in-browser PDF viewer will need to use downloads.waitForDownload() to have their script wait for the PDF download to complete.
For more information or assistance with these use cases, contact ThousandEyes Customer Engineering.
ThousandEyes has replaced the existing ten-character key that identifies an embedded widget with a 128-bit universally unique identifier (UUID). This change was made to provide additional security to our embedded report and dashboard widgets, and will impact all widgets created from this update onwards.
Existing embedded widgets will not be impacted by the change.
To update an existing widget to use a 128-bit UUID, uncheck the "Allow anyone to embed this widget to external sites" checkbox on the widget's configuration panel, and then re-check it.
ThousandEyes platform health announcements through https://success.thousandeyes.com will be discontinued on October 15th, 2020. Future health and maintenance announcements will all be made through the new https://status.thousandeyes.com site.
You can receive notifications via e-mail, SMS, RSS, and Twitter by subscribing at https://status.thousandeyes.com.
For more information, contact ThousandEyes Customer Engineering.
To improve performance when a report or dashboard widget is created with a large Endpoint Agent data set (either due to no applied filters or too large of a time span), users are now asked to either apply an endpoint test filter or reduce the time span in order to render the data.
This change impacts existing widgets as well as new widgets, but it does not impact existing snapshots.
For details, see the product documentation.
An issue was found in the Browser Sessions Network view where the domain filter drop-down menu incorrectly opens when clicking on a destination domain that has been grouped by domain. This has been resolved.
An issue was found where the path visualization for an Endpoint Agent scheduled test displayed inconsistent data compared to other views. This has been resolved.
The Endpoint Agent ping test timeout has been increased from 11 seconds to 15 seconds to resolve a couple of edge cases.
Several edge cases were found that prevented Endpoint Agents from terminating when requested. These have been resolved.
An issue was found where changing a Map widget to display in two columns instead of one distorted the widget configuration panel. This has been resolved.
Previously, users that entered multiple domains when configuring DNS alert rules did not receive an error message if they used an unsupported delimiter. This has been resolved, and customers can now use space and comma delimiters when entering values into the following alert conditions: OriginAsn, NextHopAsn, Prefix, BgpHopAnyAsn, DnsMapping, Domain, ASN, IP, RDNS, and TargetIP.
An issue was found where the platform treated the same domain name as two separate domains if the capitalization was different. This has been fixed, and capitalization no longer impacts the domain name.
ThousandEyes now supports installing Enterprise Agents on Red Hat Enterprise Linux (RHEL) and CentOS 8.1+ operating systems.
By the end of the fourth quarter of 2020, ThousandEyes will complete a relocation of our infrastructure to Amazon Web Services. Some ThousandEyes customers have access control lists (ACLs) or firewall rules which permit Enterprise Agents, Endpoint Agents or other resources to communicate specifically with the ThousandEyes infrastructure. These ACLs/rules will need to be updated with new IP addresses.
Customers who use such ACLs or rules should watch this changelog for future instructions on updating their rules. Additionally, each customer's account mangagement team will will reach out to ensure a smooth transition. In the meantime, feel free to contact ThousandEyes Customer Engineering with questions or concerns.
ThousandEyes uses publicly available BGP monitors to provide data for our BGP tests. To achieve a higher granularity of visibility, we have increased the number of public monitor vantage points accessible for tests to 80. For more information on public BGP monitor data and route view collectors, see Working with Raw BGP Data.
Several updates have been made to the Internet Insights filters:
Users can now filter by server location and server prefix, in addition to the existing interface location and interface IP address filters.
Filters are now applied at an affected interface level, rather than at an outage level.
A topology filter has been added for the Server Location to allow users to filter the topology view by location (for example, by country).
In addition, the Add a filter menu has been reorganized to clarify the following options:
The Affected Interfaces, Destinations, and Other sections are now the Network, Application, and Outage sections respectively.
The In Catalog option has been removed.
The default outages filter is now Outage Scope, and listed under Outages.
The Catalog Provider filter has been renamed Affected Provider.
The Location and IP Address filters in the Network section have been renamed as Interface Location and Interface IP Address to distinguish them from the new server location and server prefix options.
The Affected Test filter has been moved to the Application section.
The Domain and Network sections have been renamed Affected Hostname/Domain and Server Network respectively.
Existing snapshots have been updated to include the new filter options and functionality. Depending on the filters selected in the existing snapshot, views may appear to be more narrowly scoped due to interface-level filtering, but the snapshot data is unchanged.
The voice call test type has now been removed. Voice call tests can be migrated to separate SIP and RTP tests without incurring additional unit consumption. For more information about SIP and RTP test configuration, see SIP Server Test Settings and RTP Stream Test Settings. Contact Support for more information.
An issue was found after adding the render limit pop up notification to reports and dashboard widgets, where if both the render limit notification and the existing BGP size notification were triggered, only one would display. This has been resolved.
Previously, the Service Now alert integration was incorrectly avaiable for all alert types. This has been corrected, and the alert integration is only visible for the supported alert types: Cloud and Enterprise Agent, BGP, Endpoint, and Internet Insights.