Changelog
In the coming weeks, ThousandEyes will add te-podman as a dependency of the te-browserbot package. Podman version 3.1.0 or greater will be required for page load and transaction tests.
If your Enterprise Agent has an existing compatible version of Podman, Browserbot will configure it to execute tests. Otherwise, Podman version 3.1.0 will be downloaded from the ThousandEyes public repositories (or a configured repository with a compatible version of Podman).
New Cloud Agents have been added to the following locations:
Eindhoven, Netherlands (KPN) (IPv4 and IPv6)
Kansas City, MO, USA (IPv6)
Lima, Peru (IPv6)
For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
The Endpoint Agent is now signed with an Extended Validation (EV) certificate. This smooths the installation process by providing immediate reputation with Microsoft SmartScreen in the event that the certificate used to sign the Endpoint Agent installer package is changed (for example, after it expires or is revoked).
This is implemented in Endpoint Agent Client version 1.80.0.
Several improvements have been made to the alert integration notifications:
The property name ID has been updated to Alert ID.
The Test Name has been added to the notification.
The URL or Server target has been added to the notification.
The Scope field has been changed to Number of Agents.
Property names have been updated to title case rather than camel case (i.e. Rule ID instead of ruleId).
The alert rule expression now appears in the Alert Rule field.
The Clear Date value has been removed.
The word Triggered has been added as part of the email subject line for triggered alerts.
Previously, an alert with the error matches or error does not match criteria could trigger even if no errors were present. For example, if an alert condition was set that stated "Alert when: error does not match 401", an alert could trigger even if there were no errors, as a 401 error was also not present. We have updated the way we evaluate these two alert criteria to only trigger an alert when at least one actual error is present. If no errors are present, these two alert criteria will not trigger.
An issue was found where Metrics at Alert End was showing as N/A - Agent Removed from Test when an Endpoint Agent was removed from a test, after the alert had triggered. The agent is now persisted in the the Alert List and Alert History page, even if the agent has been removed.
An issue was found where updating an existing test with Cloud Agents did not immediately reflect the configuration change. This has been fixed.
New IPv4 and IPv6 Cloud Agents have been added to Eindhoven, Netherlands (Ziggo). For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
Previously, the VPN node in a path visualization merged with the first hop of the overlay. This caused some confusion with customers. We've updated the view so that the VPN node will stand alone in the path, and the first hop of the overlay will appear as a regular node.
An issue was found where, for some disabled alerts, the alert would still fire if the conditions were met. This has been resolved.
An issue was found where location alerts were failing to persist due to the metrics being reported as empty. This lead to location alerts not being cleared when they should have been, and has been resolved.
An issue was found where the SSL certificate expiry alert rule was interpreted incorrectly, and caused an alert to fire when no certificates were expiring. This has been fixed.
An issue was found where the API response to /v6/tests/\.json did not return all account IDs and their associated account group names if the user was associated with multiple organizations. This has been fixed.
An Internet Insights issue was found where it appeared that some outages shown in the Views timeline were not being displayed in the Overview dashboard. This was because the Overview and Views timeline did not previously refresh at the same interval, resulting in windows where they would be briefly out of sync. The default refresh interval for the Overview has been lowered to match the Views timeline, and this resolves the issue.
An issue was found where updating the test interval to a higher frequency (for example, changing from 5 minutes to 2 minutes) would cause the timeline to show gaps in data. This has been resolved.
An issue was found where the alerting system was incorrectly interpreting the alert configuration, resulting in alerts firing when the conditions were not met. Alert configuration validation has been improved to resolve this issue.
An issue was found where the Availability metric in reports and dashboards was showing as milliseconds instead of percentage. This has been fixed.
An issue was found where, in some cases, the system was not properly appending agent details to alerts. This resulted in the UI incorrectly showing 0 triggered agents against an alert, and has been fixed.
An issue was found where the global Alert Start Time for some alerts was showing as significantly before the Alert Start Time for the location alerts. This was caused by logic in the platform that carried the first instance of a condition violation for an agent, rather than the most recent violation. This logic has been updated, and the issue is now fixed.
An issue was found where, if ICMP was blocked and preventing end-to-end metrics from being measured, a 100% packet loss alert would be triggered. This issue has been resolved.
An issue was found where the number of agents shown to be running a scheduled test could differ between the Timeline and Table views. This occurred in instances where there were multiple data points for the same machine in the same round. The way the agent count is done has been adjusted, and only unique machines in a round will now be counted.
As of browserbot version 1.163.0 and ThousandEyes Recorder IDE version 1.1.0, the transaction test scripting sandbox now supports importing functionality from the Node.js crypto module.
One way to leverage this new functionality is to compute an HMAC:
import { credentials } from 'thousandeyes';import { createHmac } from 'crypto';​const dataToSign = 'hello, world!';const hmac = createHmac('sha256', credentials.get('some-secret-key')).update(dataToSign).digest('base64');
We have added a "Prefer TCP" option to Endpoint Agent scheduled tests that will ensure the test will fall back to ICMP + TCP Connect if no driver is detected, so that tests are still performed. This option is to avoid situations where Endpoint Agents are not included in TCP tests due to either the Windows driver missing, or agents running versions earlier than 1.75.0.
In addition, the TCP Connect checkbox has been removed, and replaced by additional Protocol options:
ICMP
ICMP + TCP Connect
Prefer TCP (fallback to ICMP)
TCP (Driver required)
Starting this release, all Catalyst switching customers entitled to ThousandEyes will see a new onboarding workflow when visiting their newly created ThousandEyes organization. The workflow will guide them through creating agents, tests, and surface common use cases.
The alert rules write API now supports the following alert rule metrics: certificate validity, TLS version, and weak cipher metrics.
A new Cloud Agent has been added to Johannesburg, South Africa (Azure southafricanorth). For a full list of Cloud Agents, see ThousandEyes Cloud Agent Locations.
An issue was found where some users were receiving emails on cleared alert events even when they deselected the option to receive cleared alert emails in the UI. This has been resolved.
An issue was found where gaps in data were incorrectly displayed on the timeline. This has been resolved.
An issue was found in the v6 reports API where incorrect results were returned for certain 'groupValues' and 'groupProperty'. This has been corrected.
An issue was fixed where changes to a filter on a report or dashboard widget weren't persisting, and reverted back to the previous filter. This has been corrected.
API requests to /v6/endpoint-tests/http-server/new with the groupId missing now return a 400 rather than a 500 error. This is more explanatory, since the error is with the input rather than the server.
An issue was found when an Endpoint Agent was uninstalled and later re-installed, where the Last Contact field would not accurately reflect the new Endpoint Agent installation date/last contact time. This has been corrected.
An issue was found where newly created organizations were not able to use the AppDynamics Dash Studio integration due to a 0 API rate limit. This has been corrected and fixed for all customers.
In the coming weeks, ThousandEyes will add te-podman as a dependency of the te-browserbot package. This package is required for page load and transaction tests, and will be automatically installed.
te-podman is a wrapper of the Podman package. If your Enterprise Agents have existing installations of Podman, this dependency may cause conflicts. We suggest removing Podman before installing this new version of BrowserBot.
Joint ThousandEyes and AppDynamics customers can now ingest ThousandEyes Cloud and Enterprise test data directly into their AppDynamics Dash Studio instance as either a number card widget or a time series widget. For more information, see ThousandEyes Integration with AppDynamics.
The Endpoint Agent now allows scheduled tests to be executed using TCP as well as ICMP protocols. This is available in Endpoint Agent version 1.75.0 and above. For more information, see Endpoint Agent TCP Support.
The average response time per hop is now shown in the Endpoint Agents > Views > Local Networks view.
The Endpoint Agent API response now includes DNS server test ping data instead of ICMP ping data for DNS probes. This avoids cases where no ICMP data was available for DNS probes.
Agent-to-server network tests can now be created and read with DSCP parameters (dscp and dscpId) when using the API.
Previously, some of the advanced settings options for shared tests were selectable. This gave users the impression that those settings could be modified within the shared account group, which is incorrect. All test settings options are now unselectable to ensure users are aware that shared tests cannot be edited from the shared account group.
The SNMP timeout and number of retry attempts for device availability SNMP requests has been increased to account for devices with high CPU utilization that either take a long time (over 2 seconds) for an SNMP or may ignore the SNMP request altogther.
Endpoint network topologies can now be grouped by interface. The available options to group by are IP Address, Network, Network & Location and Location.
An edge case was found when using the Endpoint Agent public API to get test data, where if there was exactly 1000 data points returned, the next page URL for pagination was not set. This has been resolved.
An issue was found that caused attempts to create Endpoint Agent snapshots with a date range of more than 12 hours to fail. This has been resolved, and snapshots can now be created with a date range of up to 2 days.
An issue was found where the hop slider in a multi-service path visualization view was not showing every hop, even when the slider was set to show maximum hops. This has been resolved.
An issue was found where the traceroute style output in the path visualization layer showed the test target IP address instead of the terminating hop's IP address. This has been resolved.
An issue was found that caused sorting Enterprise Agents by Utilization to incorrectly sort the list of agents. This has been fixed.
An issue was found that caused test names that included '<' or '>' to be incorrectly rejected due to a silent validation error. This has been corrected.
The ThousandEyes Recorder IDE for Windows will be signed by "Cisco Systems, Inc" from version 1.0.5 onwards. Customers running version 1.0.4 will be able to auto-update without issue.
Customers running version 1.0.3 or earlier may need to reinstall the IDE to upgrade to version 1.0.5.
The latest version of the IDE is always linked from the Test Settings page when the transaction test option is chosen.
We have placed a hard render limit on our Cloud and Enterprise and Device data as it relates to our dashboards and reports. Widgets that have a high number of tests, a large number of agents running those tests, and long timespans may hit this hard render limit. For more information, see Working with Reports.
The Activity Log now provides the option to display more details related to "create" and "update" events for transaction tests. For these events, a View Change button is displayed in the event row. This button opens a modal that provides a diff showing any transaction script changes. A separate tab shows any credential selection changes for the test.
We have made several updates to our webhook alert notifications around outage alerts:
The link field has been deprecated and replaced by the permalink field. The link field will remain for the time being, but will eventually be removed. ThousandEyes recommends using the permalink field going forward.
The ruleAid field has now been added to all alert types that use the new notifier. Previously, it was only included in outage alerts.
A dateEnd field has been added.
A type field has been added.
The ruleRefId field has been removed.
Endpoint Agents can now be transferred in bulk via the public API. Users can upload a CSV file containing the agentId, the source account and the target account. For large transfers (several thousands of agents) it may take a few minutes for the UI to show the agents as transferring.
We've redesigned the look of test labels for scheduled tests.
Fixed an issue where a small number of alerts were being inadvertently suppressed due to faulty "agents with local problems" logic in our backend system.
Fixed a UI bug where the system was not rendering error messages correctly around OAuth authentication for our webhook integrations. Previously, clicking Add New Webhook would close the modal dialog even if the save operation failed. This issue has been fixed to ensure the proper error message is shown before a webhook is saved.
Fixed an issue where the alerter system was not correctly updating alert configurations based on updates to IP prefix subnet ranges. This impacted a very small number of customers and all spurious alerts have been cleared.
In Views > Scheduled Tests > Table, an agent that appeared when no filters were applied would not be displayed when filtered by agent using its own name. This was because we were only considering agents that submitted data for the round if a filter was applied. This is now fixed.
Fixed an issue where, when a report based on Internet Insights metrics had a duration set to 12 hours or less, some widgets displayed "No Data" instead of the expected 0 outages.
Starting April, 2021, the Google Chrome and Microsoft Edge browser extensions are installed by default in new installations of Endpoint Agent on Windows endpoints. Users can opt-out of the installation as part of the Endpoint Agent installer process.
If you opt not to install the extensions as part of the Endpoint Agent installer process, you can still install them separately. See Install the Browser Extension.
The ThousandEyes Endpoint Agent is now able to identify the presence of Zscaler Internet Access (ZIA) on customer machines allowing us to gather metrics and performance measurements for traffic transiting the Zscaler network. Metrics and measurements are displayed in the UI in the same way as other supported VPN providers. Some ZIA deployment models may not be fully supported. For more information see Endpoint Agent VPN Support.
ThousandEyes has introduced a new image for its Docker Enterprise Agents that attaches an anonymous Docker volume to each container running the Enterprise Agent. This change supports a future enhancement that will improve the consistency and performance of browser-based tests.
The endpoint scheduled test table now displays a dash rather than 0% loss if the loss is in fact null, which may happen if ICMP measurements could not be completed.
The ThousandEyes Endpoint API has been extended to allow existing agent labels to be updated.
The API /endpoint-agents now correctly respects the "deleted" flag when returning a list of agents. By default this flag is false - only non-deleted agents are returned.
The Endpoint Agent basic service set identifiers (BSSID) table now shows a list of all of the BSSIDs detected in a test round.
The location of the Endpoint Agent is no longer updated if insufficient WiFi data is obtained from the end user machine, to avoid occasional "flapping" between agent locations that occurred due to operating system inconsistencies in obtaining the WiFi data used for obtaining geolocation data.
In cases where the speed score cannot be calculated due to an error gathering data for the page, the experience score is now set to 0 rather than -.
The default widget time on reports has been updated from 7 days to 1 day. This value can still manually be changed by the customer at the report level or the individual widget level.
Effective March 30, ThousandEyes will cease support for Internet Explorer 11 for https://app.thousandeyes.com. Support for all other browsers remains unchanged.
Fixed an improper handling of an aborted request that could result in an occasional failure to display an Endpoint Agent Path Visualisation.
An issue was found where creating an Endpoint agent scheduled test via the API could throw a 500 Internal Server Error. This has been resolved.
An issue occurred where our third-party provider of geolocation data was wrongly assigning an IP address to a location in the UK rather than in New York City, USA. This is now resolved.
An edge case was found where, when an Endpoint alert rule was applied to a scheduled test, the test would show the assigned rule, but the alert rule would show 0 assigned tests. This issue has been resolved.
API endpoint /v6/endpoint-data/user-sessions no longer returns 500 when only the "from" field is specified. In this case, the end date of the query will default to the current time.
API endpoint /v6/endpoint-data/network-topology no longer returns 500 when specifying only the "from" field is specified. In this case, the end date of the query will default to the current time.
Previously, unresolved VPN overlay targets were displayed on the path visualization as 0.0.0.0, which was misleading. This has been corrected, and they are no longer displayed.
An issue was found where a deleted Endpoint Pulse Agent was still being returned in API calls. This has been resolved.
Previously, an issue occurred where a combination of filters was interpreted as OR rather than AND. The path visualisation agent filter now correctly filters the list of agents.
Previously, it was possible to see different colours when clicking a group to expand it versus when you applied the equivalent filter manually. The path visualisation now shows consistent colours for groups.
In a small number of cases, test data was not collected as scheduled tests data points were discarded if the tests couldn’t finish within their expected time bounds. Data points are no longer discarded in this case.
Fixed an issue where the interactive Endpoint Agent Windows installer could fail installing a COM component and return an error "There is a problem with this Windows installer package."
Fixed an issue where a handful of no longer valid BGP alerts were incorrectly being included in reporting and dashboarding widgets due to an incorrect clear time on the alerts.
An issue was found where enterprise agents running shared tests were not showing up as a filter option in the shared to account group while using dashboards and reports. This has been resolved.
An issue was found where the hard render limits put in place for report and dashboard widgets were not transparent to users in our multi-metric table widget. Previously, if a multi-metric table request was too large, we would simply say "No Data." We have now added a tooltip that explains that the data cannot be rendered because the request is too large.
Fixed an issue that prevented test labels from being used in reports and dashboards in relation to our BGP hard render limits. A user can now select a test label and the hard render limit calculation will correctly apply. For more information on our render limits, see Working with Reports.
Fixed an issue that occurred when a user tried to deselect a filter selection in a dashboard or report widget and the filter inadvertently closed instead of letting the user deselect multiple options at a time.
Fixed an issue where a small number of users with the correct edit permissions were not able to edit existing alert rules.