Using the node-fetch Module
The node-fetch module is vitally useful in your transaction script for creating HTTP requests. Transaction test scripts can use node-fetch to make requests against one or more HTTP API endpoints, and to chain data from one call to the next.
node-fetch is maintained and distributed by npm at https://www.npmjs.com/package/node-fetch. As such, you should always look there for authoritative information on usage and implementation.
This article describes some basic uses of node-fetch in ThousandEyes transaction scripts.

Importing node-fetch

To use the node-fetch module, make sure to first import it within your transaction script:
import fetch from 'node-fetch';

Making a Request

To make a request, use the fetch function:
1
await fetch('<url>');
2
<options>
Copied!
The fetch function takes the following parameters:
  • url
    A string representing the URL that will receive the request.
  • options
    A JavaScript object that contains request parameters. For information on defaults, see the npm documentation. This parameter can be used to specify HTTP method, custom headers, timeout, and other settings.

Example POST Request

In the following example, you make a POST request and include a custom header:
1
await fetch('https://example.com',
2
{
3
method: 'POST',
4
headers: {'X-Custom-Header': 'Value-Here'}
5
}
6
)
Copied!

Arbitrary HTTP Request

You can use fetch() to make arbitrary HTTP requests in transaction scripts. Here is an example using fetch() to pull a webpage that requires Basic authentication (username and password are both "admin").
1
import { By, Key, until } from 'selenium-webdriver';
2
import { driver, markers, credentials, downloads, transaction, authentication } from 'thousandeyes';
3
import fetch from 'node-fetch';
4
runScript();
5
6
async function runScript() {
7
8
var b = new Buffer("admin:admin");
9
var s = b.toString("base64");
10
11
var requestBody = {
12
method: 'GET',
13
headers: {
14
'Authorization': 'Basic '+s,
15
}
16
}
17
markers.start('FetchTime');
18
const response = await fetch('http://the-internet.herokuapp.com/basic_auth', requestBody);
19
const pageText = await response.text();
20
if (!response.ok) {
21
throw new Error('non-200 response');
22
}
23
markers.stop('FetchTime');
24
25
};
Copied!
You can also pass different headers or use OAuth, although that is not shown in the example above.

fetch, net, and tls on Enterprise Agents

Cloud Agents have no known problems running the above script for arbitrary HTTP request, but if you run it on an Enterprise Agent, you might see that the transaction is timing out. In the event of a transaction timeout, you can find the following line in /var/log/te-sandboxd/te-sandboxd.log:
WARNING: IPv4 forwarding is disabled. Networking to destinations outside of the agent will not work.
Just because you see this message, doesn't mean you need to perform the following procedure. The following is intended to enable the use of node-fetch, net, and tls. If you are not using any of these, this warning may still appear, but it has no known impact.
In order to run this example script properly, ThousandEyes Enterprise Agents need to be configured to allow IPv4 forwarding. For TEVAs and Ubuntu packages, you'll need to edit /etc/sysctl.conf as follows:
This procedure involves system-level changes that require sudo.
sudo nano /etc/sysctl.conf
Uncomment the existing line, or add a line at the end that says:
net.ipv4.ip_forward=1
Save the changes and then tell sysctl to reload its settings:
sudo sysctl -p
That's it! Your tests should fetch() properly now.
If you follow Enterprise Agent Deployment Using Docker, you don’t need to explicitly enable IPv4 forwarding as described above, for Enterprise Agents running in Docker containers.

Working with a Returned Promise

The fetch function returns a JavaScript promise.
To access the details of this response, use the await keyword before the fetch call, so that the promise can reach the fulfilled state:
let resp = await fetch('https://example.com');
The fulfilled promise is a Response instance, which has attributes and methods to read the response details:
1
console.log(resp.status); // HTTP Status Code
2
console.log(resp.headers.raw()); // HTTP Headers
3
console.log(await resp.text()); // HTTP Response Body
Copied!

Example GET Request for a Promise

In the following example, notice that first the fetch function from the node-fetch module is imported. Then, when fetch is called, it is preceded by await so that the promise can reach the fulfilled state:
1
// fetch tx2.0
2
import fetch from 'node-fetch';
3
import assert from 'assert';
4
5
runScript();
6
7
async function runScript() {
8
let target = 'http://beta.the-acme-corporation.com';
9
10
let resp = await fetch(target, {method:'GET', headers: {'User-Agent': 'thousandeyes-transaction'}});
11
12
let resp_text = await resp.text()
13
assert(resp_text.includes('Welcome'), 'Did not find welcome text in response');
14
if(resp.status != 200){
15
throw Error('Non-200 response: ' + resp.status)
16
}
17
18
};
Copied!

Advanced Examples

Example: GET Request with Authentication

In the following example, notice that the script first defines requestOptions for basic authentication; then fetches the token from the response:
1
import {markers, credentials} from 'thousandeyes';
2
import fetch from 'node-fetch';
3
runScript();
4
5
async function runScript() {
6
const username = credentials.get('Peter Parker Username');
7
const token = credentials.get('Peter Parker Password');
8
var buffer = new Buffer(username+':'+token);
9
var apiToken = buffer.toString("base64");
10
11
var requestOptions = {
12
method: 'GET',
13
headers: {
14
'Authorization': 'Basic '+apiToken
15
}
16
}
17
18
markers.start('FetchTime');
19
const response = await fetch('https://api.thousandeyes.com/v6/agents.json?agentTypes=CLOUD', requestOptions);
20
const responseText = await response.text();
21
if (!response.ok) {
22
await console.log(responseText);
23
throw new Error('non-200 response');
24
}
25
markers.stop('FetchTime');
26
};
Copied!

Example: Multiple GET Requests

In the following example, the first fetch request fetches a list of users; the second request fetches the activity log:
1
markers.start('Fetch User List');
2
let response = await fetch('https://api.thousandeyes.com/v6/users.json', requestBody);
3
let responseText = await response.json();
4
if (!response.ok) {
5
await console.log(responseText);
6
throw new Error('non-200 response while fetching User List');
7
}
8
markers.stop('Fetch User List');
9
10
markers.start('Fetch Activity Log');
11
response = await fetch('https://api.thousandeyes.com/v6/audit/user-events/search.json', requestBody);
12
responseText = await response.json();
13
if (!response.ok) {
14
await console.log(responseText);
15
throw new Error('non-200 response while fetching Activity Log');
16
}
17
markers.stop('Fetch Activity Log');
Copied!

Example: Chained GET Requests with Authentication

In the following example, you find the list of agents assigned to a particular test; then extract the details for these agents:
1
let response = await fetch(`https://api.thousandeyes.com/v6/tests/${TEST_ID}.json`, requestBody);
2
let responseText = await response.json();
3
if (!response.ok) {
4
throw new Error('non-200 response while fetching Test details');
5
}
6
7
// Extract list of agents assigned to the given test
8
let test_agents = responseText['test'][0]['agents'];
9
let agent_details_promises = []
10
11
// Get details for individual Agents
12
for(var agent of test_agents){
13
markers.start('Agent Details ' + agent['agentId']);
14
agent_details_promises.push(fetch(`https://api.thousandeyes.com/v6/agents/${agent['agentId']}.json`, requestBody).then((value) => {
15
markers.stop('Agent Details ' + agent['agentId']);
16
return value;
17
}));
18
}
19
20
// Wait for all agent detail requests to complete
21
let agent_details = await Promise.all(agent_details_promises);
22
23
// Handle the response
24
for(var agent_detail_response of agent_details){
25
let agent_json = await agent_detail_response.json();
26
agent_json = agent_json['agents'][0]
27
}
Copied!

Proxy Support for API Monitoring

You can make API requests through a proxy by using fetch() and an HTTP proxy agent. This approach supports:
  • Configuring proxy settings for fetch() calls to auto-detect and apply transaction test settings
  • Trusting arbitrary certification chains
  • Tunneling connections through an arbitrary static proxy that might be using a self-signed certificate (e.g., for a decrypting TLS proxy)
  • Tunneling connections through a proxy determined by a PAC script
Supporting custom certificate chains is needed in instances where you need to run a fetch() command targeting a host for which a certificate was manually installed. For a custom certificate chain, copy the X509 of the certificate into your transaction script body.
The following examples are from the API transaction scripts folder of the ThousandEyes transaction scripting repository.
Read the script comments carefully. These script examples use fake endpoints and fake certificates. Be sure to replace the placeholder values with real ones.

Example: Proxies with Custom Configuration

This ThousandEyes transaction script uses proxies with a custom configuration to call the endpoint with a self-signed certificate:
1
//
2
// ThousandEyes Transaction script which uses proxies with custom configuration
3
//
4
import {markers, fetchAgent } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
8
runScript();
9
10
async function runScript() {
11
// HTTP proxy
12
markers.start('HTTP Proxy');
13
// custom proxy settings
14
const httpProxySettings = {
15
host: 'proxy-host.com',
16
port: 3333,
17
proxyAuth: {
18
username: 'username',
19
password: 'password'
20
}
21
};
22
// get a HTTP proxy agent
23
const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings)
24
// set the agent in the request options
25
const httpProxyRequestOptions = {
26
agent: httpProxyAgent,
27
};
28
// call the endpoint with a self signed certificate
29
// NOTE: this is a fake endpoint make sure to replace with a real one
30
const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
31
// verify that response status is 200
32
assert.equal(200, response1.status);
33
markers.end('HTTP Proxy');
34
35
// HTTPS proxy
36
markers.start('HTTPS Proxy');
37
// custom proxy settings
38
const httpsProxySettings = {
39
host: 'ssl-proxy-host.com',
40
port: 3333,
41
proxyAuth: {
42
username: 'username',
43
password: 'password'
44
}
45
};
46
// get a HTTPS proxy agent
47
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings)
48
// set the agent in the request options
49
const httpsProxyRequestOptions = {
50
agent: httpsProxyAgent,
51
};
52
// call the endpoint with a self signed certificate
53
// NOTE: this is a fake endpoint make sure to replace with a real one
54
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
55
// verify that response status is 200
56
assert.equal(200, response2.status);
57
markers.end('HTTPS Proxy');
58
59
// PAC proxy
60
markers.start('PAC Proxy');
61
// custom proxy settings
62
const pacProxySettings = {
63
pacScriptUrl: 'http://pac-script-location.com',
64
proxyAuth: {
65
username: 'username',
66
password: 'password'
67
}
68
};
69
// get a PAC proxy agent
70
const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings)
71
// set the agent in the request options
72
const pacProxyRequestOptions = {
73
agent: pacProxyAgent,
74
};
75
// call the endpoint with a self signed certificate
76
// NOTE: this is a fake endpoint make sure to replace with a real one
77
const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
78
// verify that response status is 200
79
assert.equal(200, response3.status);
80
markers.end('PAC Proxy');
81
};
Copied!

Example: Trusting Arbitrary Certificates

This ThousandEyes Transaction script calls the endpoint with a self signed certificate:
1
//
2
// ThousandEyes Transaction script which trusts arbitrary certificates
3
//
4
import fetch from 'node-fetch';
5
import { fetchAgent } from 'thousandeyes';
6
import assert from 'assert';
7
8
runCode();
9
10
async function runCode() {
11
// X509 of the certificate
12
// NOTE: these are fake certificates make sure to replace with real ones
13
const cert1 = `-----BEGIN CERTIFICATE-----
14
self signed certificate
15
-----END CERTIFICATE-----
16
`;
17
18
const cert2 = `-----BEGIN CERTIFICATE-----
19
self signed certificate
20
-----END CERTIFICATE-----
21
`;
22
// SSL options with custom certificates
23
const sslOptions = { customCA: [cert1, cert2] };
24
// get a fetch agent with custom certificates
25
const agent = fetchAgent.getHttpsAgent(sslOptions);
26
27
// set the agent in the request options
28
const requestOptions = {
29
agent: agent,
30
};
31
32
// call the endpoint with a self signed certificate
33
// NOTE: this is a fake endpoint make sure to replace with a real one
34
const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);
35
36
//verify that response status is 200
37
assert.equal(200, response.status);
38
}
Copied!

Example: Disable SSL Verification

This example uses a custom HTTPS agent with SSL verification disabled to call the endpoint with a self-signed certificate:
1
//
2
// ThousandEyes Transaction script which disables ssl verification
3
//
4
import fetch from 'node-fetch';
5
import { fetchAgent } from 'thousandeyes';
6
import assert from 'assert';
7
8
runCode();
9
10
async function runCode() {
11
12
// SSL options with verification disabled certificates
13
const sslOptions = { verifySSLCertificate: false };
14
// get a HTTPS agent with ssl verification disabled
15
const agent = fetchAgent.getHttpsAgent(sslOptions);
16
17
// set the agent in the request options
18
const requestOptions = {
19
agent: agent,
20
};
21
22
// call the endpoint with a self signed certificate
23
// NOTE: this is a fake endpoint make sure to replace with a real one
24
const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);
25
26
//verify that response status is 200
27
assert.equal(200, response.status);
28
}
Copied!

Example: Proxy With Custom SSL Certificate

This example uses a proxy and a custom SSL certificate.
1
//
2
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
3
//
4
import { fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
8
runScript();
9
10
async function runScript() {
11
// X509 of the certificate
12
// NOTE: this is a fake certificate make sure to replace with a real one
13
const cert = `-----BEGIN CERTIFICATE-----
14
self signed certificate
15
-----END CERTIFICATE-----
16
`;
17
// SSL options with custom certificates
18
const sslOptions = { customCA: [cert] };
19
// test proxy settings
20
const httpsProxySettings = test.getSettings().proxy;
21
22
// get a HTTPS proxy agent
23
// NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
24
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
25
// set the agent in the request options
26
const httpsProxyRequestOptions = {
27
agent: httpsProxyAgent,
28
};
29
// call the endpoint with a self signed certificate
30
// NOTE: this is a fake endpoint make sure to replace with a real one
31
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
32
// verify that response status is 200
33
assert.equal(200, response2.status);
34
};
Copied!

Example: Proxy With SSL Disabled

This example uses a proxy with SSL disabled:
1
//
2
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
3
//
4
import { fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
8
runScript();
9
10
async function runScript() {
11
// SSL options with verification disabled certificates
12
const sslOptions = { verifySSLCertificate: false };
13
// test proxy settings
14
const httpsProxySettings = test.getSettings().proxy;
15
16
// get a HTTPS proxy agent
17
// NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
18
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
19
// set the agent in the request options
20
const httpsProxyRequestOptions = {
21
agent: httpsProxyAgent,
22
};
23
// call the endpoint with a self signed certificate
24
// NOTE: this is a fake endpoint make sure to replace with a real one
25
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
26
// verify that response status is 200
27
assert.equal(200, response2.status);
28
};
Copied!

Example: Proxy With Test Configuration

This example uses a proxy from the ThousandEyes test configuration:
1
//
2
// ThousandEyes Transaction script which uses proxies with configuration from the test config
3
//
4
import {markers, fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
8
runScript();
9
10
async function runScript() {
11
// HTTP proxy
12
markers.start('HTTP Proxy');
13
// test proxy settings
14
const httpProxySettings = test.getSettings().proxy;
15
// get a HTTP proxy agent
16
const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings);
17
// set the agent in the request options
18
const httpProxyRequestOptions = {
19
agent: httpProxyAgent,
20
};
21
// call the endpoint with a self signed certificate
22
// NOTE: this is a fake endpoint make sure to replace with a real one
23
const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
24
// verify that response status is 200
25
assert.equal(200, response1.status);
26
markers.end('HTTP Proxy');
27
28
// HTTPS proxy
29
markers.start('HTTPS Proxy');
30
// test proxy settings
31
const httpsProxySettings = test.getSettings().proxy;
32
// get an HTTPS proxy agent
33
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings);
34
// set the agent in the request options
35
const httpsProxyRequestOptions = {
36
agent: httpsProxyAgent,
37
};
38
// call the endpoint with a self signed certificate
39
// NOTE: this is a fake endpoint; make sure to replace with a real one
40
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
41
// verify that response status is 200
42
assert.equal(200, response2.status);
43
markers.end('HTTPS Proxy');
44
45
// PAC proxy
46
markers.start('PAC Proxy');
47
// test proxy settings
48
const pacProxySettings = test.getSettings().proxy;
49
// get a PAC proxy agent
50
const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings);
51
// set the agent in the request options
52
const pacProxyRequestOptions = {
53
agent: pacProxyAgent,
54
};
55
// call the endpoint with a self signed certificate
56
// NOTE: this is a fake endpoint; make sure to replace with a real one
57
const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
58
// verify that response status is 200
59
assert.equal(200, response3.status);
60
markers.end('PAC Proxy');
61
};
Copied!