Excluding ThousandEyes Agents from Google Analytics
Last updated
Last updated
Users may wish to exclude the web requests generated by ThousandEyes Cloud or Enterprise Agents to sites which use Google Analytics for traffic reporting and analysis. The primary way to exclude these requests in Google Analytics is via the User-Agent string. User-Agent strings are part of an HTTP request header, and identify the type of browser that is contacting the server. Alternatively, the "ISP domain" or IP addresses of the Agents can be used in Google Analytics to exclude traffic from the Agents. Information about exclusion by User-Agent, domain and IP address and configuration of ThousandEyes tests is provided in the first sections of this article.
Google Analytics provides two methods for excluding request data in reporting: segments or filters. Since the User-Agent string is present in the header of every HTTP request, the queries generated in Google Analytics with both filters and segments should normally produce the same results. However, you should consider any potential differences in your data that could produce different results when choosing which method to use. Information about configuring segments and filters is provided in the section "Google Analytics Configuration" below the "Exclusion by..." sections.
The last section of this article describes another approach to excluding requests from ThousandEyes Agents, based on modification of the target web pages.
ThousandEyes Web Layer tests use different User-Agent strings, by default. The HTTP Server test type has a default User-Agent string based on the cURL program which will look similar to the following, depending on the current software version of the Agent:
The Page Load and Transaction test types (BrowserBot-based tests) have a default User-Agent string based on the Chrome browser, which will look similar to the following, depending on the current software version of the Agent:
Using the BrowserBot User-Agent string as an example, Google Analytics will extract the following information about the sending system:
Browser: Chrome Browser Version: 49.0.2623.87 OS: Linux Architecture: x86_64
Any User-Agent string whose Browser field is a known browser such as Chrome or Safari will result in the request being categorized as from that browser. In order to uniquely identify requests from ThousandEyes Cloud or Enterprise Agents, the default User-Agent strings for HTTP Server and BrowserBot tests can be changed to a custom string by selecting the Advanced Settings tab on a Web Layer test configuration page, and setting the User Agent selector to "Custom", then entering a unique User-Agent string in the text field below the selector:
A suitable User-Agent string that will identify requests from ThousandEyes Cloud or Enterprise Agents is the following:
Add this custom User-Agent string to each ThousandEyes test configuration that requires exclusion from Google Analytics data. You may verify that your test is sending the custom User-Agent string by checking the Request Headers in the test results page. Instructions for viewing the Request Headers can be found in Identifying Traffic From ThousandEyes Agents.
Excluding traffic by Google Analytics' "ISP domain" option is an option in the filter settings (also called "Service Provider" when viewing your Audience data from the Audience > Technology > Network view). This option best is reserved for excluding ThousandEyes Cloud Agents. Google Analytics determines ISP domain by performing a "reverse lookup" in the Domain Name System (DNS) on the source IP address of an incoming HTTP request. In DNS terminology, this is a resolution of a PTR or "pointer" record, which maps an IP address to a domain name. For example, the following IP addresses map via PTR records to the following domain names:
IP Address | Domain Name |
202.46.179.6 | vm-esxi-wlg.agents.thousandeyes.com |
192.34.85.187 | a2-docker4-bos.ag1.thousandeyes.com |
198.89.98.68 | a3-docker1-sjc.ag1.thousandeyes.com |
ThousandEyes Cloud Agents normally have PTR records in DNS (most IP addresses do not), but there may be reasons why a particular Agent location does not have PTR records for the Agent IP addresses. Contact support@thousandeyes.com for questions regarding PTR record availability.
Excluding traffic by IP address may be useful if the number of ThousandEyes Agents is small. This method is best suited for excluding Enterprise Agents with fixed IP addresses, although Cloud Agent IP addresses are relatively unchanging. The IP address filter method will exclude all requests from any ThousandEyes Web Layer tests from the Agents with the specified IP addresses.
To obtain the IP addresses of Cloud or Enterprise Agents that you need to exclude, see Obtaining a List of ThousandEyes Agent IP Addresses.
To exclude ThousandEyes traffic by User-Agent, create either a segment or a filter in Google Analytics for the page data. Creating segments and filters is done by logging into your Google Analytics account at https://analytics.google.com and selecting the web site context to which the filter or segment will apply. Then follow the instructions below for the method you've chosen.
For some of the methods, supplying the matching criterion requires entering a regular expression that will match the User-Agent, domain name, or IP addresses. Suggested regular expressions are provided in the instructions, below. For additional information on creating or testing regular expressions used in Google Analytics, see the Google Analytics Help article About regular expressions.
If you use other segments to view your data, and can exclude traffic by custom User-Agent string, then this would likely be the preferred method to exclude HTTP requests from ThousandEyes Agents. This method works on a per-test basis, so applies to both Cloud and Enterprise Agents. The segment method can only be used to exclude traffic by User-Agent. To exclude traffic by IP address or ISP domain name, use a filter, per the instructions in the Filter method section, below.
To create a new segment to exclude page visits based on User-Agent string, follow these steps:
Click on the Reporting tab
Click + Add Segment
Click + New Segment
Enter a name in the Segment Name field, such as “ThousandEyes Agents User-Agent”
Click Technology
In the Browser setting, click the pull-down menu and select “does not exactly match”
In the text field, enter the following string:
Click the Save button
The new segment should appear similar to the following:
The filter method allows for three approaches: one based on the User-Agent, one based on Agent domain name ("ISP domain") and one based on Agent IP address.
To create a new filter to exclude page visits based on User-Agent string, follow these steps:
Click on the Admin tab
Click on Filters in the View column
Click + Add Filter
Enter a name in the Filter Name field, such as “ThousandEyes Agents User-Agent”
Click the Custom button
Select the Exclude radio button for the Filter Type
in the Filter Field setting, select “Browser”
in the Filter Pattern field, enter the following string to match the custom User-Agent string:
Click the Save button
The new filter should appear similar to the following:
To create a new filter to exclude page visits based on Agent domain name, follow these steps:
Click on the Admin tab
Click on Filters in the View column
Click + Add Filter
Enter a name in the Filter Name field, such as “ThousandEyes Agents domain name”
Click the Predefined button for the Filter Type
Select "Exclude" in the first selector
Select “traffic from the ISP Domain” in the second selector
Select "that end with" in the third selector
in the ISP Domain field, enter the text:
Click the Save button
The new filter should appear similar to the following:
To create a new filter to exclude page visits based on Agent IP address, follow these steps:
Click on the Admin tab
Click on Filters in the View column
Click + Add Filter
Enter a name in the Filter Name field, such as “ThousandEyes Agents IP addresses”
Click the Custom button for the Filter Type
Select the Exclude radio button
in the Filter Field setting, select “IP Address”
in the Filter Pattern field, enter the list of ThousandEyes Agent IP addresses you obtained from the UI or API (do not use verbatim the example below) using a regular expression similar to the following:
Click the Save button
The new filter should appear similar to the following:
Note: The Predefined filter type does offer an IP address-based option but only for a single IP address. Therefore, one filter per Agent would need to be created. Unless you are filtering a small number of Enterprise Agents, this method is less convenient than the Custom filter type.
The previously mentioned approaches were based on configuring Google Analytics queries to exclude the unwanted data. However, another approach is to modify the code of web pages requested by ThousandEyes Agents to prevent the unwanted data from being collected by Google Analytics. This approach eliminates the need to configure Google Analytics to exclude the ThousandEyes Agent request data. If users have the ability to perform or request code changes, we recommend this method if you plan to monitor your site with ThousandEyes for the foreseeable future.
The modification to the targeted web pages is straightforward; the Google Analytics code is placed inside a Javascript conditional execution code block such as an "if" statement, and only executed if the User-Agent string is not a custom ThousandEyes User-Agent string such as the one used in the previous example configurations.
The following pseudo-code illustrates the approach:
if (!navigator.userAgent || !navigator.userAgent.match("TE/1\.0'?$")) { // perform analytics here }
The analytics code is performed if the User-Agent is empty or is not the custom ThousandEyes User-Agent string. Users can modify this to suit their needs.
