Step-by-Step Guide to Troubleshooting Endpoint Agent Problems
Last updated
Last updated
A prerequisite to Endpoint Agent troubleshooting is physical or remote access to the end-user machine and for some steps an un-restricted user account on the end-user machine.
To resolve any Endpoint Agent problem follow the path in the Endpoint Agent troubleshooting flow diagram below (see Figure 1.0). This document contains many links to make switching between the flow diagram, sections, diagrams and troubleshooting steps as easy as possible. The section directly below the flow diagram contains the page links to relevant Endpoint Agent troubleshooting steps. If the sequence in the flow diagram is followed in the correct order, most Endpoint Agent issues will be resolved. It is important not to skip questions, sections or the steps provided in this document. The green boxes contain a series of questions to capture an Endpoint Agent issue and the teal boxes informs you which troubleshooting steps you need to follow.
If you cannot find the information you are looking for in this Article check the "Reference Articles" section at the bottom of this page; it contains links to the Endpoint Agent frequently asked questions articles. Or contact the Customer Engineering Team.
Figure-1.0
Endpoint Agents periodically check in every 10 minutes for configuration updates with ThousandEyes controller https://c1.eb.thousandeyes.com. If an Endpoint Agent switches networks, the agent checks in again. If the agent is not able to connect to the controller, the agent will not appear on the Agent Settings page (see Figure 1.1).
Figure-1.1
Troubleshooting Endpoint Agent deployments and checkin steps:
1.1. To check if an agent is checking in, head to the Endpoint Agent > Agent Settings page and find the agent by name or use the filters to track down the agent (see Figure 1.1). If you have recently installed a new agent, wait 10 minutes for the agent to appear.
1.2. If the agent is not present on the Agent Settings page despite a successful software package installation, then re-installing the package is not the solution. Every time the Endpoint Agent package is removed and installed again a new ID is generated a new Endpoint Agent instance can be created, an Endpoint licence can be consumed and multiple Endpoint Agents with the same name may appear in the Agent Settings page which can create unnecessary administrative overhead.
1.3. If the deployment is working in one network but not on another, then it is most likely a problem with the end-user's deployment environment. Using the correct Endpoint Agent deployment guide article for your Operating System check no steps were missed during the installation process. The deployment guides contain information about how to set-up firewalls, VPNs, proxies and end-user machine permissions so the Endpoint Agent software operates correctly.
1.4. Check when the agent last checked in on the Agent Settings page (see Figure 1.1). If you know for certain the end-user system is online but the Endpoint Agent has not checked in for 10 minutes or more, check the Endpoint Agent services are running:
If you are using a Windows machines, check via the Services Control Manager the “ThousandEyes Endpoint Agent” service is running (see Figure 1.2)
Figure-1.2
If you are using a Mac OSX machine, check via a terminal application the “ThousandEyes Endpoint Agent” service is running (see Figure 1.3)
Figure-1.3
Using the command-line application (e.g. iTerm for Mac OSX and CMD for Windows) on the end-user's machine check you get a response when you run the PING command to the ThousandEyes controller “c1.eb.thousandeyes.com” (see Figure 1.4)
Figure-1.4
Check that the controller URL https://c1.eb.thousandeyes.com loads in the end-user's browser (see Figure 1.5)
Figure-1.5
Try restarting the end-user's machine
1.5. Check the Endpoint Agent is not disabled (see Figure 1.1). If the Endpoint Agent is disabled, try to enable it on the Agent Settings page. If you cannot enable an agent, check the Endpoint Agent licence usage on the Usage and Billing page and ensure the licences in your plan are not all used (see Figure 1.6). If all licences are used then free up a licence by disabling another Endpoint Agent back on the Agent Settings page (see Figure 1.1).
1.6. If the agent is now checking in, revert back to where you left off on the Endpoint Agent troubleshooting flow diagram to continue troubleshooting. If the agent is still not checking in, contact the Customer Engineering Team for assistance.
It is important Endpoint Agents deployments are updated when new releases are made available as they contain product enhancements and may also include bugs fixes. We publish Release Notes in the Customer Engineering Centre along with every release or you can subscribe to the regular Release Notes via email. The Release Notes contain a summary of all new Endpoint Agent release changes. New Endpoint Agent packages are released by the ThousandEyes Engineering team every other Tuesday. Once the new Endpoint Agent package is released it is made available to organisations randomly over the following 5 day period immediately after a release. It is possible you'll receive the Release Notes before the update is made available to your organisation. The larger the Endpoint Agent deployment the longer it takes to fully deploy as deployments within an organisation are staggered. For large organisation please expect updates to be completed over a 2-3 day period after your first agent has updated. Endpoint Agents continually check every 3-4 hours for version updates and provided the download from thousandeyes.com domain is not being blocked no user intervention should be required for agent to update themselves.
Troubleshooting Endpoint Agent update steps:
1.1. Check whether the Endpoint Agent is running the latest package release by heading over to the Agent Settings page. If a warning sign appears next to any Endpoint Agent, the agent is not up-to-date (see Figure 1.1). In the agent table, agent row on the Agent Settings page there is a column displaying what version the agent is actively running (see Figure 1.1).
1.2. In order for package updates to work, the agent has to be able to check in regularly. Complete the steps in the "Troubleshooting Endpoint Agent Deployments and Checkin Problems" section.
1.3. If the Endpoint Agent has been manually installed and the environment does not allow for automatic updates, contact the organization's senior ThousandEyes system administrators to get the agent updated.
1.4. If, after you have followed the previous steps, the agent has updated, return to where you left off on the Endpoint Agent troubleshooting flow diagram to continue troubleshooting. If the Endpoint Agent did not update, contact the Customer Engineering Team for assistance.
Scheduled Tests are configured to run at set intervals and run in the background on the agent's end-user machine. Scheduled Tests are assigned to agents according to the agent label matching criteria. During Scheduled Test runs, Network Access layer probes also run. At this stage of troubleshooting, ThousandEyes users and administrators may decide to refresh their knowledge on Scheduled Test configuration and operations using the Scheduled Tests article.
Troubleshooting Scheduled Tests steps The following steps cover Scheduled Test and Agent Settings checks, end-user machine checks, environment devices such as proxy and firewall device checks and a brief note on Network Access data.
3.1. Ensure an agent Label has been created and contains active agent (see Figure 3.1).
3.2. Check the following:
The correct agent Label has been applied to the test
The correct proxy setting has been set (proxy settings applied here take precedence over system and agent proxy settings)
There are agent actively matching the agent Label
The test is enabled (see Figure 3.2)
3.3. If the test appears to be configured correctly, check you have not oversubscribed tests to the Endpoint Agents. An Endpoint Enterprise or Pulse Agent runs a maximum of 10 Scheduled Tests at a time. Check you have not oversubscribed an agent to tests; ensure the same agent has not been accidentally matched by several agent Labels matching conditions or the same label or multiple labels have not been applied to more tests than the Endpoint Agent is capable of running.
3.4. When a test is first configured it may take up to 10 minutes for test data to start appearing.
3.5. On the end-user machine check the Scheduled Test target is reachable from the end-user machine:
From the end-user's machine command-line application (iTerm on Mac OSX or CMD on Windows Operating Systems) check you get a response when you run the PING command to the Scheduled Test target (see Figure 3.3)
If the target is web-based check it will load from the end-user Web Browser (see Figure 3.4)
3.6. If the end-user is behind a proxy and the agent type is Endpoint Enterprise, Scheduled Tests by default detect and use the end-user's proxy settings. Endpoint Agent Scheduled Tests do not support HTTPS proxies. However, most HTTPS proxies support clients connecting via HTTP (see section "Proxying HTTPS Requests" in our blog post about "Measuring Performance with HTTP Proxies" for details on how HTTP and HTTPS proxies work). You may want to check the full details on Endpoint Enterprise proxy configuration in Endpoint Agent proxy configuration for Scheduled Tests.
If you deployed Endpoint Pulse on a Windows machine, then Scheduled Tests will use the system level proxy setting. It is common the system level proxy is not set and be aware that changing the system proxy setting could affect other application traffic. Changing the system proxy setting on a Windows machine requires administrator privileges. If setting a system proxy is a problem the best approach is to deploy Endpoint Enterprise on the end-user machine. To check the Windows end-user machines system proxy settings using the example screenshots in (Figure 3.5) below and ensure you have administrator access prior to starting.
3.7. If the Endpoint Agent end-user is switching networks (such as between 2 different wireless networks in 2 different geographical locations), the services in each of those networks may prevent the scheduled tests from running. Double-check that the following access control devices or services in each location are not blocking access to the scheduled test target:
Firewalls
Proxies
VPNs
3.8. If Network Access layer data is not appearing during periods when Scheduled Tests are being run, ensure that you have configured a Monitored Network (see Figure 3.6).
3.9. In some circumstances, no data will be collected in a test round, and the message Agent has not collected data in this round will be seen when hovering over an Endpoint Agent, as shown in (Figure 3.7) below:
This may occur when:
The Endpoint Agent stops communicating with the ThousandEyes platform, either because the host is sleeping or is shutting down. The agent remains assigned to the scheduled test for one hour before being removed. During this time, the agent does not collect data, and the above message is observed, until either the agent comes back online, or is removed from the scheduled test. However, if the Endpoint Agent loses network connectivity or has an issue communicating with the ThousandEyes controller, the agent continues to run for one hour and caches the collected data for 24 hours.
The network connection being used by the Endpoint Agent is changed. In this case, data collection may be skipped for a test round while the agent refetches the assigned scheduled tests.
The Endpoint Agent experiences performance issues that cause test collection deadlines to be missed.
3.10. If, after you have followed the previous steps, the scheduled test is still not working, contact the Customer Engineering Team for further assistance.
To start a manual Browser Session recording: the end-user selects a browser tab to record from, clicks on the ThousandEyes Browser Sessions recorder extension icon in the end-user's browser (see figure 10) and then in the active tab the recording starts when the user loads a new web page or refreshes the current web page. The recording stops when the end-user activates another tab, after 30 seconds of inactivity or after 5 minutes of continuous recording. When a recording starts or stops: in Google Chrome, the browser extension icon status is updated, in the IE browser, a system notification is displayed. Browser Sessions data appears on the Endpoint Agents > Views page and is grouped together in 5 minute intervals. After a new Endpoint Agent installation Browser Sessions recordings may take up to 10 minutes to appear on the Views page but in most cases data appears within 60 seconds from when a recording is started.
Troubleshooting manual Browser Sessions recordings steps: For troubleshooting Google Chrome Browser Session recordings start at step 4.1.1 or if you are using IE start at section 4.2.1 Once you have worked through the browser specific troubleshooting steps, continue to step 4.3.1.
4.1.1.(Chrome) If you have access to the end-user machine, check the ThousandEyes Browser Sessions recorder extension icon appears in the end user's browser (see Figure 4.1).
4.1.2. (Chrome) If the ThousandEyes Browser Sessions recorder extension icon does not appear in the Chrome's toolbar, check that the extension is not hidden (see Figure 4.2).
4.1.3. (Chrome) To check if the browser extension is installed, type the special URL “chrome://extensions” in Chrome’s search bar and look for the ThousandEyes Endpoint Agent extension package (see Figure 4.3). If the extension is not installed, you can perform a manual installation by opening this link to the Chrome web store on the end-user machine or follow the full installation in our article Installing the Endpoint Agent for Mac OS X.
4.1.4. (Chrome) If the browser extension icon is covered with a small covering red warning sign (see Figure 4.4), click the icon and the Repair button to resolve the issue. If the repair is unsuccessful restart the browser. Also ensure that you covered everything in sections 1 and 2 of the troubleshooting process; then try running the repair again. If the alert status persists, contact the Customer Engineering Team for further assistance. If the extension alert status has been fixed, continue to work through the non-browser specific troubleshooting steps as step 4.3.1.
4.2.1. (IE) If the ThousandEyes Browser Sessions recorder extension icon does not appear in the IE Command bar (see Figure 4.5), check that the Command bar is visible (see Figure 4.6).
4.2.2. (IE) Check that the IE Add-on is enabled (see Figure 4.7). If the Add-on does not appear in IE Add-ons after following all the previous steps and the main installation package completed successfully, contact the Customer Engineering Team for further assistance.
4.3.1. Check the Endpoint Agent's browser extension status by navigating to the Agent Settings page (see Figure 4.8). On the Agent Settings page Endpoints deployed on a Mac OSX platform show an Apple Logo next to the them and Windows Endpoint deployments show a Windows logo next to them. All Endpoint Agent Windows platforms with Internet Explorer installed show the Internet Explorer logo and Endpoint platforms with Google Chrome installed should show the Chrome icon. A green dot over any browser extension icon in the Agent Settings page means the extension is installed and working. The browser logo and icon status can take up to 10 minutes to appear and switch from amber to green status.
4.3.2. If the browser extension appears to be working, the agent is checking in but no waterfall data appears on the Endpoint Agents > Views page try to restart the machine or browser.
4.3.3. If after following the previous steps the Browser Sessions recordings are now working then revert back to where you left off on the Endpoint Agent troubleshooting flow diagram for the next troubleshooting steps. If the Browser Sessions recordings are still not working, contact the Customer Engineering Team for assistance.
Automatic Browser Session recordings begin when a user browses to a configured target domain from a specified network. The automatic Browser Sessions configuration can be referred to as a “target domain and network set”.
Troubleshooting Automatic Browser Sessions recordings steps:
5.1. If you have not already done so, follow the steps in the previous section (see section 4) as it covers most problems that could occur when trying to perform automatic Browser Sessions recordings.
5.2. Check that the Monitored Network correctly matches the agent's network. The local Internet provider or Network Administrator should be able to provide you with the public network network address range your Endpoint Agents will use. If you are unsure what network your agent will be operating from, set a catch all Monitored Network e.g. 0.0.0.0/0 (see Figure 3.6 and Figure 5.1). Check that you have applied the correct domains in the Monitored Domain set. Hostnames are valid Monitored Domains.
5.4. Recordings can take up to 15 minutes to appear as agent check-ins run every 10 minutes and within agent application itself processes configuration updates run at 5 minute intervals.
5.4. If after following the previous steps the automatic Browser Sessions recordings are working then revert back to where you left off on the Endpoint Agent troubleshooting flow diagram for the next troubleshooting step. If the automatic Browser Sessions recordings are still not working, contact the Customer Engineering Team for assistance.
This section is work in progress and will be added to shortly. If you spot any anomalies in the test data, contact the Customer Engineering Team for further assistance.
Figure-1.6
Figure-3.1
Figure-3.2
Figure-3.3
Figure-3.4
Figure-3.5
Figure-3.5.1
Figure-3.5.2
Figure-3.6
Figure-3.7
Figure-4.1
Figure-4.2
Figure-4.3
Figure-4.4
Figure-4.5
Figure-4.6
Figure-4.7
Figure-4.8
Figure-5.1
