Using the node-fetch module

The node-fetch module is vitally useful in your transaction script for creating HTTP requests. Transaction test scripts can use node-fetch to make requests against one or more HTTP API endpoints, and to chain data from one call to the next.

node-fetch is maintained and distributed by npm at https://www.npmjs.com/package/node-fetch. As such, you should always look there for authoritative information on usage and implementation.

This article describes some basic uses of node-fetch in ThousandEyes transaction scripts.

If you need to run a test that solely consists of API calls, and don’t need to emulate a user journey in a browser, use the API Test Type.

Importing node-fetch

To use the node-fetch module, make sure to first import it within your transaction script:

import fetch from 'node-fetch';

Making a Request

To make a request, use the fetch function:

await fetch('<url>');
    <options>

The fetch function takes the following parameters:

• url

  A string representing the URL that will receive the request.

• options

  A JavaScript object that contains request parameters. For information on defaults, see the npm documentation. This parameter can be used to specify HTTP method, custom headers, timeout, and other settings.

Example POST Request

In the following example, you make a POST request and include a custom header:

await fetch('https://example.com', 
    {
    method: 'POST', 
    headers: {'X-Custom-Header': 'Value-Here'}
    }
)

Arbitrary HTTP Request

You can use fetch() to make arbitrary HTTP requests in transaction scripts. Here is an example using fetch() to pull a webpage that requires Basic authentication (username and password are both "admin").

import { By, Key, until } from 'selenium-webdriver';
import { driver, markers, credentials, downloads, transaction, authentication } from 'thousandeyes';
import fetch from 'node-fetch';
runScript();
 
async function runScript() {
 
    var b = new Buffer("admin:admin");
    var s = b.toString("base64");
     
    var requestBody = {
        method: 'GET',
        headers: {
            'Authorization': 'Basic '+s,
        }
    }
    markers.start('FetchTime');
    const response = await fetch('http://the-internet.herokuapp.com/basic_auth', requestBody);     
    const pageText = await response.text();
    if (!response.ok) {
        throw new Error('non-200 response');
    }
    markers.stop('FetchTime');
 
};

You can also pass different headers or use OAuth, although that is not shown in the example above.

fetch, net, and tls on Enterprise Agents

Cloud Agents have no known problems running the above script for arbitrary HTTP request, but if you run it on an Enterprise Agent, you might see that the transaction is timing out. In the event of a transaction timeout, you can find the following line in /var/log/te-sandboxd/te-sandboxd.log:

WARNING: IPv4 forwarding is disabled. Networking to destinations outside of the agent will not work.

Just because you see this message, doesn't mean you need to perform the following procedure. The following is intended to enable the use of node-fetch, net, and tls. If you are not using any of these, this warning may still appear, but it has no known impact.

In order to run this example script properly, ThousandEyes Enterprise Agents need to be configured to allow IPv4 forwarding. For TEVAs and Ubuntu packages, you'll need to edit /etc/sysctl.conf as follows:

This procedure involves system-level changes that require sudo.

sudo nano /etc/sysctl.conf

Uncomment the existing line, or add a line at the end that says:

net.ipv4.ip_forward=1

Save the changes and then tell sysctl to reload its settings:

sudo sysctl -p

That's it! Your tests should fetch() properly now.

If you follow Enterprise Agent Deployment Using Docker, you don’t need to explicitly enable IPv4 forwarding as described above, for Enterprise Agents running in Docker containers.

Working with a Returned Promise

The fetch function returns a JavaScript promise.

To access the details of this response, use the await keyword before the fetch call, so that the promise can reach the fulfilled state:

let resp = await fetch('https://example.com');

The fulfilled promise is a Response instance, which has attributes and methods to read the response details:

console.log(resp.status);  // HTTP Status Code
console.log(resp.headers.raw());  // HTTP Headers
console.log(await resp.text());	// HTTP Response Body

Example GET Request for a Promise

In the following example, notice that first the fetch function from the node-fetch module is imported. Then, when fetch is called, it is preceded by await so that the promise can reach the fulfilled state:

// fetch tx2.0
import fetch from 'node-fetch';
import assert from 'assert';

runScript();

async function runScript() {
   let target = 'http://beta.the-acme-corporation.com';

   let resp = await fetch(target, {method:'GET', headers: {'User-Agent': 'thousandeyes-transaction'}});

   let resp_text = await resp.text()
   assert(resp_text.includes('Welcome'), 'Did not find welcome text in response');
   if(resp.status != 200){
       throw Error('Non-200 response: ' + resp.status)
   }
  
};

Advanced Examples

For additional examples, see the public repository of ThousandEyes transaction scripts.

Example: GET Request with Authentication

In the following example, notice that the script first defines requestOptions for basic authentication; then fetches the token from the response:

import {markers, credentials} from 'thousandeyes';
import fetch from 'node-fetch';
runScript();

async function runScript() {
   const username = credentials.get('Peter Parker Username');
   const token = credentials.get('Peter Parker Password');
   var buffer = new Buffer(username+':'+token);
   var apiToken = buffer.toString("base64");

   var requestOptions = {
      method: 'GET',
      headers: {
          'Authorization': 'Basic '+apiToken
      }
   }

   markers.start('FetchTime');
   const response = await fetch('https://api.thousandeyes.com/v6/agents.json?agentTypes=CLOUD', requestOptions);
   const responseText = await response.text();
   if (!response.ok) {
      await console.log(responseText);
      throw new Error('non-200 response');
   }
   markers.stop('FetchTime');
};

Example: Multiple GET Requests

In the following example, the first fetch request fetches a list of users; the second request fetches the activity log:

markers.start('Fetch User List');
let response = await fetch('https://api.thousandeyes.com/v6/users.json', requestBody);
let responseText = await response.json();
if (!response.ok) {
    await console.log(responseText);
    throw new Error('non-200 response while fetching User List');
}
markers.stop('Fetch User List');

markers.start('Fetch Activity Log');
response = await fetch('https://api.thousandeyes.com/v6/audit/user-events/search.json', requestBody);
responseText = await response.json();
if (!response.ok) {
    await console.log(responseText);
    throw new Error('non-200 response while fetching Activity Log');
}
markers.stop('Fetch Activity Log');

Example: Chained GET Requests with Authentication

In the following example, you find the list of agents assigned to a particular test; then extract the details for these agents:

let response = await fetch(`https://api.thousandeyes.com/v6/tests/${TEST_ID}.json`, requestBody);
let responseText = await response.json();
if (!response.ok) {
    throw new Error('non-200 response while fetching Test details');
}

// Extract list of agents assigned to the given test
let test_agents = responseText['test'][0]['agents'];
let agent_details_promises = []

// Get details for individual Agents
for(var agent of test_agents){
    markers.start('Agent Details ' + agent['agentId']);
    agent_details_promises.push(fetch(`https://api.thousandeyes.com/v6/agents/${agent['agentId']}.json`, requestBody).then((value) => {
        markers.stop('Agent Details ' + agent['agentId']);
        return value;
    }));
}

// Wait for all agent detail requests to complete
let agent_details = await Promise.all(agent_details_promises);

// Handle the response
for(var agent_detail_response of agent_details){
    let agent_json = await agent_detail_response.json();
    agent_json = agent_json['agents'][0]
}

Proxy Support for API Monitoring

You can make API requests through a proxy by using fetch() and an HTTP proxy agent. This approach supports:

• Configuring proxy settings for fetch() calls to auto-detect and apply transaction test settings

• Trusting arbitrary certification chains

• Tunneling connections through an arbitrary static proxy that might be using a self-signed certificate (e.g., for a decrypting TLS proxy)

• Tunneling connections through a proxy determined by a PAC script

Supporting custom certificate chains is needed in instances where you need to run a fetch() command targeting a host for which a certificate was manually installed. For a custom certificate chain, copy the X509 of the certificate into your transaction script body.

The following examples are from the API transaction scripts folder of the ThousandEyes transaction scripting repository.

Read the script comments carefully. These script examples use fake endpoints and fake certificates. Be sure to replace the placeholder values with real ones.

Example: Proxies with Custom Configuration

This ThousandEyes transaction script uses proxies with a custom configuration to call the endpoint with a self-signed certificate:

//
// ThousandEyes Transaction script which uses proxies with custom configuration
//
import {markers, fetchAgent } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';

runScript();

async function runScript() {
    // HTTP proxy
    markers.start('HTTP Proxy');
    // custom proxy settings
    const httpProxySettings = {
        host: 'proxy-host.com',
        port: 3333,
        proxyAuth: {
            username: 'username',
            password: 'password'
        }
    };
    // get a HTTP proxy agent
    const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings)
    // set the agent in the request options
    const httpProxyRequestOptions = {
        agent: httpProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response1.status);
    markers.end('HTTP Proxy');

    // HTTPS proxy
    markers.start('HTTPS Proxy');
    // custom proxy settings
    const httpsProxySettings = {
        host: 'ssl-proxy-host.com',
        port: 3333,
        proxyAuth: {
            username: 'username',
            password: 'password'
        }
    };
    // get a HTTPS proxy agent
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings)
    // set the agent in the request options
    const httpsProxyRequestOptions = {
        agent: httpsProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response2.status);
    markers.end('HTTPS Proxy');

    // PAC proxy
    markers.start('PAC Proxy');
    // custom proxy settings
    const pacProxySettings = {
        pacScriptUrl: 'http://pac-script-location.com',
        proxyAuth: {
            username: 'username',
            password: 'password'
        }
    };
    // get a PAC proxy agent
    const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings)
    // set the agent in the request options
    const pacProxyRequestOptions = {
        agent: pacProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response3.status);
    markers.end('PAC Proxy');
};

Example: Trusting Arbitrary Certificates

This ThousandEyes Transaction script calls the endpoint with a self signed certificate:

//
// ThousandEyes Transaction script which trusts arbitrary certificates
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';

runCode();

async function runCode() {
    // X509 of the certificate
    // NOTE: these are fake certificates make sure to replace with real ones
    const cert1 = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;

    const cert2 = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
    // SSL options with custom certificates
    const sslOptions = { customCA: [cert1, cert2] };
    // get a fetch agent with custom certificates
    const agent = fetchAgent.getHttpsAgent(sslOptions);

    // set the agent in the request options
    const requestOptions = {
        agent: agent,
    };

    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response = await fetch('https://some-website-with-a-self-signed-cert.com', requestOptions);

    //verify that response status is 200
    assert.equal(200, response.status);
}

Example: Disable SSL Verification

This example uses a custom HTTPS agent with SSL verification disabled to call the endpoint with a self-signed certificate:

//
// ThousandEyes Transaction script which disables ssl verification
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';

runCode();

async function runCode() {

    // SSL options with verification disabled certificates
    const sslOptions = { verifySSLCertificate: false };
    // get a HTTPS agent with ssl verification disabled
    const agent = fetchAgent.getHttpsAgent(sslOptions);

    // set the agent in the request options
    const requestOptions = {
        agent: agent,
    };

    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);

    //verify that response status is 200
    assert.equal(200, response.status);
}

Example: Proxy With Custom SSL Certificate

This example uses a proxy and a custom SSL certificate.

//
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
//
import { fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';

runScript();

async function runScript() {
    // X509 of the certificate
    // NOTE: this is a fake certificate make sure to replace with a real one
    const cert = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
    // SSL options with custom certificates
    const sslOptions = { customCA: [cert] };
    // test proxy settings
    const httpsProxySettings = test.getSettings().proxy;

    // get a HTTPS proxy agent
    // NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
    // set the agent in the request options
    const httpsProxyRequestOptions = {
        agent: httpsProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response2.status);
};

Example: Proxy With SSL Disabled

This example uses a proxy with SSL disabled:

//
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
//
import { fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';

runScript();

async function runScript() {
    // SSL options with verification disabled certificates
    const sslOptions = { verifySSLCertificate: false };
    // test proxy settings
    const httpsProxySettings = test.getSettings().proxy;

    // get a HTTPS proxy agent
    // NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
    // set the agent in the request options
    const httpsProxyRequestOptions = {
        agent: httpsProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response2.status);
};

Example: Proxy With Test Configuration

This example uses a proxy from the ThousandEyes test configuration:

//
// ThousandEyes Transaction script which uses proxies with configuration from the test config
//
import {markers, fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';

runScript();

async function runScript() {
    // HTTP proxy
    markers.start('HTTP Proxy');
    // test proxy settings
    const httpProxySettings = test.getSettings().proxy;
    // get a HTTP proxy agent
    const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings);
    // set the agent in the request options
    const httpProxyRequestOptions = {
        agent: httpProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response1.status);
    markers.end('HTTP Proxy');

    // HTTPS proxy
    markers.start('HTTPS Proxy');
    // test proxy settings
    const httpsProxySettings = test.getSettings().proxy;
    // get an HTTPS proxy agent
    const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings);
    // set the agent in the request options
    const httpsProxyRequestOptions = {
        agent: httpsProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint; make sure to replace with a real one
    const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response2.status);
    markers.end('HTTPS Proxy');

    // PAC proxy
    markers.start('PAC Proxy');
    // test proxy settings
    const pacProxySettings = test.getSettings().proxy;
    // get a PAC proxy agent
    const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings);
    // set the agent in the request options
    const pacProxyRequestOptions = {
        agent: pacProxyAgent,
    };
    // call the endpoint with a self signed certificate 
    // NOTE: this is a fake endpoint; make sure to replace with a real one
    const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
    // verify that response status is 200
    assert.equal(200, response3.status);
    markers.end('PAC Proxy');
};

Example: Using Client Certificates

This example shows a transaction test script that uses client certificates:

//
// ThousandEyes Transaction script which uses client certificates authentication
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';

runCode();

async function runCode() {
    // X509 of the certificate
    // NOTE: these are fake certificates make sure to replace with real ones
    const key = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;

    const clientCert = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
    const passphrase = '';

    // SSL options with client certificate, key and passphrase
    const sslOptions = { key: key, cert: clientCert, passphrase: passphrase };
    // get a fetch agent
    const agent = fetchAgent.getHttpsAgent(sslOptions);

    // set the agent in the request options
    const requestOptions = {
        agent: agent,
    };

    // call the endpoint
    // NOTE: this is a fake endpoint make sure to replace with a real one
    const response = await fetch('https://some-website.com', requestOptions);

    //verify that response status is 200
    assert.equal(200, response.status);
}