The node-fetch module is vitally useful in your transaction script for creating HTTP requests. Transaction test scripts can use node-fetch to make requests against one or more HTTP API endpoints, and to chain data from one call to the next.
node-fetch is maintained and distributed by npm at https://www.npmjs.com/package/node-fetch. As such, you should always look there for authoritative information on usage and implementation.
This article describes some basic uses of node-fetch in ThousandEyes transaction scripts.
If you need to run a test that solely consists of API calls, and don’t need to emulate a user journey in a browser, use the API Test Type.
Importing node-fetch
To use the node-fetch module, make sure to first import it within your transaction script:
import fetch from 'node-fetch';
Making a Request
To make a request, use the fetch function:
await fetch('<url>');
<options>
The fetch function takes the following parameters:
url
A string representing the URL that will receive the request.
options
A JavaScript object that contains request parameters. For information on defaults, see the npm documentation. This parameter can be used to specify HTTP method, custom headers, timeout, and other settings.
Example POST Request
In the following example, you make a POST request and include a custom header:
You can use fetch() to make arbitrary HTTP requests in transaction scripts. Here is an example using fetch() to pull a webpage that requires Basic authentication (username and password are both "admin").
import { By, Key, until } from 'selenium-webdriver';
import { driver, markers, credentials, downloads, transaction, authentication } from 'thousandeyes';
import fetch from 'node-fetch';
runScript();
async function runScript() {
var b = new Buffer("admin:admin");
var s = b.toString("base64");
var requestBody = {
method: 'GET',
headers: {
'Authorization': 'Basic '+s,
}
}
markers.start('FetchTime');
const response = await fetch('http://the-internet.herokuapp.com/basic_auth', requestBody);
const pageText = await response.text();
if (!response.ok) {
throw new Error('non-200 response');
}
markers.stop('FetchTime');
};
You can also pass different headers or use OAuth, although that is not shown in the example above.
fetch, net, and tls on Enterprise Agents
Cloud Agents have no known problems running the above script for arbitrary HTTP request, but if you run it on an Enterprise Agent, you might see that the transaction is timing out. In the event of a transaction timeout, you can find the following line in /var/log/te-sandboxd/te-sandboxd.log:
WARNING: IPv4 forwarding is disabled. Networking to destinations outside of the agent will not work.
Just because you see this message, doesn't mean you need to perform the following procedure. The following is intended to enable the use of node-fetch, net, and tls. If you are not using any of these, this warning may still appear, but it has no known impact.
In order to run this example script properly, ThousandEyes Enterprise Agents need to be configured to allow IPv4 forwarding. For TEVAs and Ubuntu packages, you'll need to edit /etc/sysctl.conf as follows:
This procedure involves system-level changes that require sudo.
sudo nano /etc/sysctl.conf
Uncomment the existing line, or add a line at the end that says:
net.ipv4.ip_forward=1
Save the changes and then tell sysctl to reload its settings:
sudo sysctl -p
That's it! Your tests should fetch() properly now.
If you follow Enterprise Agent Deployment Using Docker, you don’t need to explicitly enable IPv4 forwarding as described above, for Enterprise Agents running in Docker containers.
To access the details of this response, use the await keyword before the fetch call, so that the promise can reach the fulfilled state:
let resp = await fetch('https://example.com');
The fulfilled promise is a Response instance, which has attributes and methods to read the response details:
console.log(resp.status); // HTTP Status Code
console.log(resp.headers.raw()); // HTTP Headers
console.log(await resp.text()); // HTTP Response Body
Example GET Request for a Promise
In the following example, notice that first the fetch function from the node-fetch module is imported. Then, when fetch is called, it is preceded by await so that the promise can reach the fulfilled state:
// fetch tx2.0
import fetch from 'node-fetch';
import assert from 'assert';
runScript();
async function runScript() {
let target = 'http://beta.the-acme-corporation.com';
let resp = await fetch(target, {method:'GET', headers: {'User-Agent': 'thousandeyes-transaction'}});
let resp_text = await resp.text()
assert(resp_text.includes('Welcome'), 'Did not find welcome text in response');
if(resp.status != 200){
throw Error('Non-200 response: ' + resp.status)
}
};
In the following example, notice that the script first defines requestOptions for basic authentication; then fetches the token from the response:
import {markers, credentials} from 'thousandeyes';
import fetch from 'node-fetch';
runScript();
async function runScript() {
const username = credentials.get('Peter Parker Username');
const token = credentials.get('Peter Parker Password');
var buffer = new Buffer(username+':'+token);
var apiToken = buffer.toString("base64");
var requestOptions = {
method: 'GET',
headers: {
'Authorization': 'Basic '+apiToken
}
}
markers.start('FetchTime');
const response = await fetch('https://api.thousandeyes.com/v6/agents.json?agentTypes=CLOUD', requestOptions);
const responseText = await response.text();
if (!response.ok) {
await console.log(responseText);
throw new Error('non-200 response');
}
markers.stop('FetchTime');
};
Example: Multiple GET Requests
In the following example, the first fetch request fetches a list of users; the second request fetches the activity log:
markers.start('Fetch User List');
let response = await fetch('https://api.thousandeyes.com/v6/users.json', requestBody);
let responseText = await response.json();
if (!response.ok) {
await console.log(responseText);
throw new Error('non-200 response while fetching User List');
}
markers.stop('Fetch User List');
markers.start('Fetch Activity Log');
response = await fetch('https://api.thousandeyes.com/v6/audit/user-events/search.json', requestBody);
responseText = await response.json();
if (!response.ok) {
await console.log(responseText);
throw new Error('non-200 response while fetching Activity Log');
}
markers.stop('Fetch Activity Log');
Example: Chained GET Requests with Authentication
In the following example, you find the list of agents assigned to a particular test; then extract the details for these agents:
let response = await fetch(`https://api.thousandeyes.com/v6/tests/${TEST_ID}.json`, requestBody);
let responseText = await response.json();
if (!response.ok) {
throw new Error('non-200 response while fetching Test details');
}
// Extract list of agents assigned to the given test
let test_agents = responseText['test'][0]['agents'];
let agent_details_promises = []
// Get details for individual Agents
for(var agent of test_agents){
markers.start('Agent Details ' + agent['agentId']);
agent_details_promises.push(fetch(`https://api.thousandeyes.com/v6/agents/${agent['agentId']}.json`, requestBody).then((value) => {
markers.stop('Agent Details ' + agent['agentId']);
return value;
}));
}
// Wait for all agent detail requests to complete
let agent_details = await Promise.all(agent_details_promises);
// Handle the response
for(var agent_detail_response of agent_details){
let agent_json = await agent_detail_response.json();
agent_json = agent_json['agents'][0]
}
Proxy Support for API Monitoring
You can make API requests through a proxy by using fetch() and an HTTP proxy agent. This approach supports:
Configuring proxy settings for fetch() calls to auto-detect and apply transaction test settings
Trusting arbitrary certification chains
Tunneling connections through an arbitrary static proxy that might be using a self-signed certificate (e.g., for a decrypting TLS proxy)
Tunneling connections through a proxy determined by a PAC script
Supporting custom certificate chains is needed in instances where you need to run a fetch() command targeting a host for which a certificate was manually installed. For a custom certificate chain, copy the X509 of the certificate into your transaction script body.
Read the script comments carefully. These script examples use fake endpoints and fake certificates. Be sure to replace the placeholder values with real ones.
Example: Proxies with Custom Configuration
This ThousandEyes transaction script uses proxies with a custom configuration to call the endpoint with a self-signed certificate:
//
// ThousandEyes Transaction script which uses proxies with custom configuration
//
import {markers, fetchAgent } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';
runScript();
async function runScript() {
// HTTP proxy
markers.start('HTTP Proxy');
// custom proxy settings
const httpProxySettings = {
host: 'proxy-host.com',
port: 3333,
proxyAuth: {
username: 'username',
password: 'password'
}
};
// get a HTTP proxy agent
const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings)
// set the agent in the request options
const httpProxyRequestOptions = {
agent: httpProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response1.status);
markers.end('HTTP Proxy');
// HTTPS proxy
markers.start('HTTPS Proxy');
// custom proxy settings
const httpsProxySettings = {
host: 'ssl-proxy-host.com',
port: 3333,
proxyAuth: {
username: 'username',
password: 'password'
}
};
// get a HTTPS proxy agent
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings)
// set the agent in the request options
const httpsProxyRequestOptions = {
agent: httpsProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response2.status);
markers.end('HTTPS Proxy');
// PAC proxy
markers.start('PAC Proxy');
// custom proxy settings
const pacProxySettings = {
pacScriptUrl: 'http://pac-script-location.com',
proxyAuth: {
username: 'username',
password: 'password'
}
};
// get a PAC proxy agent
const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings)
// set the agent in the request options
const pacProxyRequestOptions = {
agent: pacProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response3.status);
markers.end('PAC Proxy');
};
Example: Trusting Arbitrary Certificates
This ThousandEyes Transaction script calls the endpoint with a self signed certificate:
//
// ThousandEyes Transaction script which trusts arbitrary certificates
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';
runCode();
async function runCode() {
// X509 of the certificate
// NOTE: these are fake certificates make sure to replace with real ones
const cert1 = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
const cert2 = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
// SSL options with custom certificates
const sslOptions = { customCA: [cert1, cert2] };
// get a fetch agent with custom certificates
const agent = fetchAgent.getHttpsAgent(sslOptions);
// set the agent in the request options
const requestOptions = {
agent: agent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response = await fetch('https://some-website-with-a-self-signed-cert.com', requestOptions);
//verify that response status is 200
assert.equal(200, response.status);
}
Example: Disable SSL Verification
This example uses a custom HTTPS agent with SSL verification disabled to call the endpoint with a self-signed certificate:
//
// ThousandEyes Transaction script which disables ssl verification
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';
runCode();
async function runCode() {
// SSL options with verification disabled certificates
const sslOptions = { verifySSLCertificate: false };
// get a HTTPS agent with ssl verification disabled
const agent = fetchAgent.getHttpsAgent(sslOptions);
// set the agent in the request options
const requestOptions = {
agent: agent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response = await fetch('https://some-website-with-a-self-signde-cert.com', requestOptions);
//verify that response status is 200
assert.equal(200, response.status);
}
Example: Proxy With Custom SSL Certificate
This example uses a proxy and a custom SSL certificate.
//
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
//
import { fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';
runScript();
async function runScript() {
// X509 of the certificate
// NOTE: this is a fake certificate make sure to replace with a real one
const cert = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
// SSL options with custom certificates
const sslOptions = { customCA: [cert] };
// test proxy settings
const httpsProxySettings = test.getSettings().proxy;
// get a HTTPS proxy agent
// NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
// set the agent in the request options
const httpsProxyRequestOptions = {
agent: httpsProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response2.status);
};
Example: Proxy With SSL Disabled
This example uses a proxy with SSL disabled:
//
// ThousandEyes Transaction script which uses a proxy and a custom SSL certificate.
//
import { fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';
runScript();
async function runScript() {
// SSL options with verification disabled certificates
const sslOptions = { verifySSLCertificate: false };
// test proxy settings
const httpsProxySettings = test.getSettings().proxy;
// get a HTTPS proxy agent
// NOTE: this works with getHttpProxyAgent and getPACProxyAgent too.
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings, sslOptions);
// set the agent in the request options
const httpsProxyRequestOptions = {
agent: httpsProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response2.status);
};
Example: Proxy With Test Configuration
This example uses a proxy from the ThousandEyes test configuration:
//
// ThousandEyes Transaction script which uses proxies with configuration from the test config
//
import {markers, fetchAgent, test } from 'thousandeyes';
import fetch from 'node-fetch';
import assert from 'assert';
runScript();
async function runScript() {
// HTTP proxy
markers.start('HTTP Proxy');
// test proxy settings
const httpProxySettings = test.getSettings().proxy;
// get a HTTP proxy agent
const httpProxyAgent = fetchAgent.getHttpProxyAgent(httpProxySettings);
// set the agent in the request options
const httpProxyRequestOptions = {
agent: httpProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint make sure to replace with a real one
const response1 = await fetch('http://some-website.com', httpProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response1.status);
markers.end('HTTP Proxy');
// HTTPS proxy
markers.start('HTTPS Proxy');
// test proxy settings
const httpsProxySettings = test.getSettings().proxy;
// get an HTTPS proxy agent
const httpsProxyAgent = fetchAgent.getHttpsProxyAgent(httpsProxySettings);
// set the agent in the request options
const httpsProxyRequestOptions = {
agent: httpsProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint; make sure to replace with a real one
const response2 = await fetch('https://some-website.com', httpsProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response2.status);
markers.end('HTTPS Proxy');
// PAC proxy
markers.start('PAC Proxy');
// test proxy settings
const pacProxySettings = test.getSettings().proxy;
// get a PAC proxy agent
const pacProxyAgent = fetchAgent.getPACProxyAgent(pacProxySettings);
// set the agent in the request options
const pacProxyRequestOptions = {
agent: pacProxyAgent,
};
// call the endpoint with a self signed certificate
// NOTE: this is a fake endpoint; make sure to replace with a real one
const response3 = await fetch('https://some-website.com', pacProxyRequestOptions);
// verify that response status is 200
assert.equal(200, response3.status);
markers.end('PAC Proxy');
};
Example: Using Client Certificates
This example shows a transaction test script that uses client certificates:
//
// ThousandEyes Transaction script which uses client certificates authentication
//
import fetch from 'node-fetch';
import { fetchAgent } from 'thousandeyes';
import assert from 'assert';
runCode();
async function runCode() {
// X509 of the certificate
// NOTE: these are fake certificates make sure to replace with real ones
const key = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
const clientCert = `-----BEGIN CERTIFICATE-----
self signed certificate
-----END CERTIFICATE-----
`;
const passphrase = '';
// SSL options with client certificate, key and passphrase
const sslOptions = { key: key, cert: clientCert, passphrase: passphrase };
// get a fetch agent
const agent = fetchAgent.getHttpsAgent(sslOptions);
// set the agent in the request options
const requestOptions = {
agent: agent,
};
// call the endpoint
// NOTE: this is a fake endpoint make sure to replace with a real one
const response = await fetch('https://some-website.com', requestOptions);
//verify that response status is 200
assert.equal(200, response.status);
}
