Using the node-fetch Module

The node-fetch module is vitally useful in your transaction script for creating HTTP requests. Transaction test scripts can use node-fetch to make requests against one or more HTTP API endpoints, and to chain data from one call to the next.
node-fetch is maintained and distributed by npm at https://www.npmjs.com/package/node-fetch. As such, you should always look there for authoritative information on usage and implementation.
This article describes some basic uses of node-fetch in ThousandEyes transaction scripts.
Importing node-fetch
To use the node-fetch module, make sure to first import it within your transaction script:
import fetch from 'node-fetch';
Making a Request
To make a request, use the fetch function:
1
await fetch('<url>');
2
    <options>
Copied!
The fetch function takes the following parameters:
url
A string representing the URL that will receive the request.
options
A JavaScript object that contains request parameters. For information on defaults, see the npm documentation. This parameter can be used to specify HTTP method, custom headers, timeout, and other settings.
Example POST Request
In the following example, you make a POST request and include a custom header:
1
await fetch('https://example.com', 
2
    {
3
    method: 'POST', 
4
    headers: {'X-Custom-Header': 'Value-Here'}
5
    }
6
)
Copied!
Arbitrary HTTP Request
You can use fetch() to make arbitrary HTTP requests in transaction scripts. Here is an example using fetch() to pull a webpage that requires Basic authentication (username and password are both "admin").
1
import { By, Key, until } from 'selenium-webdriver';
2
import { driver, markers, credentials, downloads, transaction, authentication } from 'thousandeyes';
3
import fetch from 'node-fetch';
4
runScript();
5
 
6
async function runScript() {
7
 
8
    var b = new Buffer("admin:admin");
9
    var s = b.toString("base64");
10
     
11
    var requestBody = {
12
        method: 'GET',
13
        headers: {
14
            'Authorization': 'Basic '+s,
15
        }
16
    }
17
    markers.start('FetchTime');
18
    const response = await fetch('http://the-internet.herokuapp.com/basic_auth', requestBody);     
19
    const pageText = await response.text();
20
    if (!response.ok) {
21
        throw new Error('non-200 response');
22
    }
23
    markers.stop('FetchTime');
24
 
25
};
Copied!
You can also pass different headers or use OAuth, although that is not shown in the example above.
fetch, net, and tls on Enterprise Agents
Cloud Agents have no known problems running the above script for arbitrary HTTP request, but if you run it on an Enterprise Agent, you might see that the transaction is timing out. In the event of a transaction timeout, you can find the following line in /var/log/te-sandboxd/te-sandboxd.log:
WARNING: IPv4 forwarding is disabled. Networking to destinations outside of the agent will not work.
Just because you see this message, doesn't mean you need to perform the following procedure. The following is intended to enable the use of node-fetch, net, and tls. If you are not using any of these, this warning may still appear, but it has no known impact.
In order to run this example script properly, ThousandEyes Enterprise Agents need to be configured to allow IPv4 forwarding. For TEVAs and Ubuntu packages, you'll need to edit /etc/sysctl.conf as follows:
This procedure involves system-level changes that require sudo.
sudo nano /etc/sysctl.conf
Uncomment the existing line, or add a line at the end that says:
net.ipv4.ip_forward=1
Save the changes and then tell sysctl to reload its settings:
sudo sysctl -p
That's it! Your tests should fetch() properly now.
If you follow Enterprise Agent Deployment Using Docker, you don’t need to explicitly enable IPv4 forwarding as described above, for Enterprise Agents running in Docker containers.
Working with a Returned Promise
The fetch function returns a JavaScript promise.
To access the details of this response, use the await keyword before the fetch call, so that the promise can reach the fulfilled state:
let resp = await fetch('https://example.com');
The fulfilled promise is a Response instance, which has attributes and methods to read the response details:
1
console.log(resp.status);  // HTTP Status Code
2
console.log(resp.headers.raw());  // HTTP Headers
3
console.log(await resp.text());	// HTTP Response Body
Copied!
Example GET Request for a Promise
In the following example, notice that first the fetch function from the node-fetch module is imported. Then, when fetch is called, it is preceded by await so that the promise can reach the fulfilled state:
1
// fetch tx2.0
2
import fetch from 'node-fetch';
3
import assert from 'assert';
4
​
5
runScript();
6
​
7
async function runScript() {
8
   let target = 'http://beta.the-acme-corporation.com';
9
​
10
   let resp = await fetch(target, {method:'GET', headers: {'User-Agent': 'thousandeyes-transaction'}});
11
​
12
   let resp_text = await resp.text()
13
   assert(resp_text.includes('Welcome'), 'Did not find welcome text in response');
14
   if(resp.status != 200){
15
       throw Error('Non-200 response: ' + resp.status)
16
   }
17
  
18
};
Copied!
Advanced Examples
For additional examples, see the public repository of ThousandEyes transaction scripts.
Example: GET Request with Authentication
In the following example, notice that the script first defines requestOptions for basic authentication; then fetches the token from the response:
1
import {markers, credentials} from 'thousandeyes';
2
import fetch from 'node-fetch';
3
runScript();
4
​
5
async function runScript() {
6
   const username = credentials.get('Peter Parker Username');
7
   const token = credentials.get('Peter Parker Password');
8
   var buffer = new Buffer(username+':'+token);
9
   var apiToken = buffer.toString("base64");
10
​
11
   var requestOptions = {
12
      method: 'GET',
13
      headers: {
14
          'Authorization': 'Basic '+apiToken
15
      }
16
   }
17
​
18
   markers.start('FetchTime');
19
   const response = await fetch('https://api.thousandeyes.com/v6/agents.json?agentTypes=CLOUD', requestOptions);
20
   const responseText = await response.text();
21
   if (!response.ok) {
22
      await console.log(responseText);
23
      throw new Error('non-200 response');
24
   }
25
   markers.stop('FetchTime');
26
};
Copied!
Example: Multiple GET Requests
In the following example, the first fetch request fetches a list of users; the second request fetches the activity log:
1
markers.start('Fetch User List');
2
let response = await fetch('https://api.thousandeyes.com/v6/users.json', requestBody);
3
let responseText = await response.json();
4
if (!response.ok) {
5
    await console.log(responseText);
6
    throw new Error('non-200 response while fetching User List');
7
}
8
markers.stop('Fetch User List');
9
​
10
markers.start('Fetch Activity Log');
11
response = await fetch('https://api.thousandeyes.com/v6/audit/user-events/search.json', requestBody);
12
responseText = await response.json();
13
if (!response.ok) {
14
    await console.log(responseText);
15
    throw new Error('non-200 response while fetching Activity Log');
16
}
17
markers.stop('Fetch Activity Log');
Copied!
Example: Chained GET Requests with Authentication
In the following example, you find the list of agents assigned to a particular test; then extract the details for these agents:
1
let response = await fetch(`https://api.thousandeyes.com/v6/tests/${TEST_ID}.json`, requestBody);
2
let responseText = await response.json();
3
if (!response.ok) {
4
    throw new Error('non-200 response while fetching Test details');
5
}
6
​
7
// Extract list of agents assigned to the given test
8
let test_agents = responseText['test'][0]['agents'];
9
let agent_details_promises = []
10
​
11
// Get details for individual Agents
12
for(var agent of test_agents){
13
    markers.start('Agent Details ' + agent['agentId']);
14
    agent_details_promises.push(fetch(`https://api.thousandeyes.com/v6/agents/${agent['agentId']}.json`, requestBody).then((value) => {
15
        markers.stop('Agent Details ' + agent['agentId']);
16
        return value;
17
    }));
18
}
19
​
20
// Wait for all agent detail requests to complete
21
let agent_details = await Promise.all(agent_details_promises);
22
​
23
// Handle the response
24
for(var agent_detail_response of agent_details){
25
    let agent_json = await agent_detail_response.json();
26
    agent_json = agent_json['agents'][0]
27
}
Copied!
Proxy Support for API Monitoring
You can make API requests through a proxy by using fetch() and an HTTP proxy agent. This approach supports:
Configuring proxy settings for fetch() calls to auto-detect and apply transaction test settings
Trusting arbitrary certification chains
Tunneling connections through an arbitrary static proxy that might be using a self-signed certificate (e.g., for a decrypting TLS proxy)
Tunneling connections through a proxy determined by a PAC script
Supporting custom certificate chains is needed in instances where you need to run a fetch() command targeting a host for which a certificate was manually installed. For a custom certificate chain, copy the X509 of the certificate into your transaction script body.
The following examples are from the API transaction scripts folder of the ThousandEyes transaction scripting repository.
Read the script comments carefully. These script examples use fake endpoints and fake certificates. Be sure to replace the placeholder values with real ones.
Example: Proxies with Custom Configuration
This ThousandEyes transaction script uses proxies with a custom configuration to call the endpoint with a self-signed certificate:
1
//
2
// ThousandEyes Transaction script which uses proxies with custom configuration
3
//
4
import {markers, fetchAgent } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
​
8
runScript();
9
​
10
async function runScript() {
11
    // HTTP proxy
12
    markers.start('HTTP Proxy');
13
    // custom proxy settings
14
    const httpProxySettings = {
15
        host: 'proxy-host.com',
16
        port: 3333,
17
        proxyAuth: {
18
            username: 'username',
19
            password: 'password'
20
        }
21
    };
22
    // get a HTTP proxy agent
23
    const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings)
24
    // set the agent in the request options
25
    const httpProxyRequestOptions = {
26
        agent: httpProxyAgent,
27
    };
28
    // call the endpoint with a self signed certificate 
29
    // NOTE: this is a fake endpoint make sure to replace with a real one
30
    const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
31
    // verify that response status is 200
32
    assert.equal(200, response1.status);
33
    markers.end('HTTP Proxy');
34
​
35
    // HTTPS proxy
36
    markers.start('HTTPS Proxy');
37
    // custom proxy settings
38
    const httpsProxySettings = {
39
        host: 'ssl-proxy-host.com',
40
        port: 3333,
41
        proxyAuth: {
42
            username: 'username',
43
            password: 'password'
44
        }
45
    };
46
    // get a HTTPS proxy agent
47
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings)
48
    // set the agent in the request options
49
    const httpsProxyRequestOptions = {
50
        agent: httpsProxyAgent,
51
    };
52
    // call the endpoint with a self signed certificate 
53
    // NOTE: this is a fake endpoint make sure to replace with a real one
54
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
55
    // verify that response status is 200
56
    assert.equal(200, response2.status);
57
    markers.end('HTTPS Proxy');
58
​
59
    // PAC proxy
60
    markers.start('PAC Proxy');
61
    // custom proxy settings
62
    const pacProxySettings = {
63
        pacScriptUrl: 'http://pac-script-location.com',
64
        proxyAuth: {
65
            username: 'username',
66
            password: 'password'
67
        }
68
    };
69
    // get a PAC proxy agent
70
    const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings)
71
    // set the agent in the request options
72
    const pacProxyRequestOptions = {
73
        agent: pacProxyAgent,
74
    };
75
    // call the endpoint with a self signed certificate 
76
    // NOTE: this is a fake endpoint make sure to replace with a real one
77
    const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
78
    // verify that response status is 200
79
    assert.equal(200, response3.status);
80
    markers.end('PAC Proxy');
81
};
Copied!
Example: Trusting Arbitrary Certificates
This ThousandEyes Transaction script calls the endpoint with a self signed certificate:
1
//
2
// ThousandEyes Transaction script which trusts arbitrary certificates
3
//
4
import fetch from 'node-fetch';
5
import { fetchAgent } from 'thousandeyes';
6
import assert from 'assert';
7
​
8
runCode();
9
​
10
async function runCode() {
11
    // X509 of the certificate
12
    // NOTE: these are fake certificates make sure to replace with real ones
13
    const cert1 = `-----BEGIN CERTIFICATE-----
14
self signed certificate
15
-----END CERTIFICATE-----
16
`;
17
​
18
    const cert2 = `-----BEGIN CERTIFICATE-----
19
self signed certificate
20
-----END CERTIFICATE-----
21
`;
22
    // SSL options with custom certificates
23
    const sslOptions = { customCA: [cert1, cert2] };
24
    // get a fetch agent with custom certificates
25
    const agent = fetchAgent.getHttpsAgent(sslOptions);
26
​
27
    // set the agent in the request options
28
    const requestOptions = {
29
        agent: agent,
30
    };
31
​
32
    // call the endpoint with a self signed certificate 
33
    // NOTE: this is a fake endpoint make sure to replace with a real one
34
    const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);
35
​
36
    //verify that response status is 200
37
    assert.equal(200, response.status);
38
}
Copied!
Example: Disable SSL Verification
This example uses a custom HTTPS agent with SSL verification disabled to call the endpoint with a self-signed certificate:
1
//
2
// ThousandEyes Transaction script which disables ssl verification
3
//
4
import fetch from 'node-fetch';
5
import { fetchAgent } from 'thousandeyes';
6
import assert from 'assert';
7
​
8
runCode();
9
​
10
async function runCode() {
11
​
12
    // SSL options with verification disabled certificates
13
    const sslOptions = { verifySSLCertificate: false };
14
    // get a HTTPS agent with ssl verification disabled
15
    const agent = fetchAgent.getHttpsAgent(sslOptions);
16
​
17
    // set the agent in the request options
18
    const requestOptions = {
19
        agent: agent,
20
    };
21
​
22
    // call the endpoint with a self signed certificate 
23
    // NOTE: this is a fake endpoint make sure to replace with a real one
24
    const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);
25
​
26
    //verify that response status is 200
27
    assert.equal(200, response.status);
28
}
Copied!
Example: Proxy With Custom SSL Certificate
This example uses a proxy and a custom SSL certificate.
1
//
2
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
3
//
4
import { fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
​
8
runScript();
9
​
10
async function runScript() {
11
    // X509 of the certificate
12
    // NOTE: this is a fake certificate make sure to replace with a real one
13
    const cert = `-----BEGIN CERTIFICATE-----
14
self signed certificate
15
-----END CERTIFICATE-----
16
`;
17
    // SSL options with custom certificates
18
    const sslOptions = { customCA: [cert] };
19
    // test proxy settings
20
    const httpsProxySettings = test.getSettings().proxy;
21
​
22
    // get a HTTPS proxy agent
23
    // NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
24
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
25
    // set the agent in the request options
26
    const httpsProxyRequestOptions = {
27
        agent: httpsProxyAgent,
28
    };
29
    // call the endpoint with a self signed certificate 
30
    // NOTE: this is a fake endpoint make sure to replace with a real one
31
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
32
    // verify that response status is 200
33
    assert.equal(200, response2.status);
34
};
Copied!
Example: Proxy With SSL Disabled
This example uses a proxy with SSL disabled:
1
//
2
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
3
//
4
import { fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
​
8
runScript();
9
​
10
async function runScript() {
11
    // SSL options with verification disabled certificates
12
    const sslOptions = { verifySSLCertificate: false };
13
    // test proxy settings
14
    const httpsProxySettings = test.getSettings().proxy;
15
​
16
    // get a HTTPS proxy agent
17
    // NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
18
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
19
    // set the agent in the request options
20
    const httpsProxyRequestOptions = {
21
        agent: httpsProxyAgent,
22
    };
23
    // call the endpoint with a self signed certificate 
24
    // NOTE: this is a fake endpoint make sure to replace with a real one
25
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
26
    // verify that response status is 200
27
    assert.equal(200, response2.status);
28
};
Copied!
Example: Proxy With Test Configuration
This example uses a proxy from the ThousandEyes test configuration:
1
//
2
// ThousandEyes Transaction script which uses proxies with configuration from the test config
3
//
4
import {markers, fetchAgent, test } from 'thousandeyes';
5
import fetch from 'node-fetch';
6
import assert from 'assert';
7
​
8
runScript();
9
​
10
async function runScript() {
11
    // HTTP proxy
12
    markers.start('HTTP Proxy');
13
    // test proxy settings
14
    const httpProxySettings = test.getSettings().proxy;
15
    // get a HTTP proxy agent
16
    const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings);
17
    // set the agent in the request options
18
    const httpProxyRequestOptions = {
19
        agent: httpProxyAgent,
20
    };
21
    // call the endpoint with a self signed certificate 
22
    // NOTE: this is a fake endpoint make sure to replace with a real one
23
    const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
24
    // verify that response status is 200
25
    assert.equal(200, response1.status);
26
    markers.end('HTTP Proxy');
27
​
28
    // HTTPS proxy
29
    markers.start('HTTPS Proxy');
30
    // test proxy settings
31
    const httpsProxySettings = test.getSettings().proxy;
32
    // get an HTTPS proxy agent
33
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings);
34
    // set the agent in the request options
35
    const httpsProxyRequestOptions = {
36
        agent: httpsProxyAgent,
37
    };
38
    // call the endpoint with a self signed certificate 
39
    // NOTE: this is a fake endpoint; make sure to replace with a real one
40
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
41
    // verify that response status is 200
42
    assert.equal(200, response2.status);
43
    markers.end('HTTPS Proxy');
44
​
45
    // PAC proxy
46
    markers.start('PAC Proxy');
47
    // test proxy settings
48
    const pacProxySettings = test.getSettings().proxy;
49
    // get a PAC proxy agent
50
    const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings);
51
    // set the agent in the request options
52
    const pacProxyRequestOptions = {
53
        agent: pacProxyAgent,
54
    };
55
    // call the endpoint with a self signed certificate 
56
    // NOTE: this is a fake endpoint; make sure to replace with a real one
57
    const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
58
    // verify that response status is 200
59
    assert.equal(200, response3.status);
60
    markers.end('PAC Proxy');
61
};
Copied!