Installing CA Certificates on Enterprise Agents

Enterprise Agents which perform Web Layer tests to targets protected by SSL/TLS (a target URL beginning with https://, ftps://, or ftp:// when using implicit-mode SSL/TLS) must validate the SSL server digital certificate and any intermediate certificates returned by the server or fetched by the client. Additionally, Enterprise Agents perform configuration download and data upload to ThousandEyes servers using TLS. Certificate validation requires that the sequence of certificates “chain” via digital signature back to the trusted root certification authority (CA) certificate that signed the previous certificate in the chain. As with most browsers and operating systems, ThousandEyes Enterprise Agents are pre-loaded with the standard X.509 CA certificate store of root CA certificates, which is provided by the Mozilla NSS project.

In some environments, the certificate(s) returned by the server do not chain back to a CA certificate in the standard certificate store, causing Web Layer tests to produce certificate errors and/or the administrative communication to ThousandEyes to fail. To avoid the errors, customers can add any needed certificates to an Enterprise Agent’s certificate store. The steps to add certificates vary, depending on the type of Enterprise Agent.

Adding root CA certificates cannot be performed on Cloud Agents, due to the shared nature of Cloud Agents. Endpoint Agents are installed on standard operating systems which the customer controls, including control of the certificate stores.

When to add certificates to an Enterprise Agent

When any Web Layer test (HTTP Server test, Page Load test, Transaction test or FTP Server test) or administrative communication from the Agent to ThousandEyes produces a certificate error, review the following scenarios to determine whether a root CA certificate must be added to the Enterprise Agent’s certificate store.

• Certificates are issued by a private root CA certificate (internal PKI)

Organizations which run their own public key infrastructure (PKI) will issue their own SSL server certificates, intermediate certificates (if any), and root CA certificate(s) which will not be included in the standard Mozilla root certificate store. The root CA certificate that issued certificates on servers that are the targets of ThousandEyes tests must be added to the Enterprise Agent.

• Decrypting proxy server

If an Enterprise Agent is explicitly configured to use a proxy or if an Agent's web traffic is captured by a transparent proxy (a proxy that does not require configuration of the client) and the proxy performs SSL/TLS decryption, then the proxy's signing certificate(s) must be added to the Enterprise Agent. Proxy servers which decrypt and inspect data carried by SSL/TLS use a signing certificate to rewrite server certificates. Typically, the proxy signing certificate is not issued by one of the certificates in the standard certificate store. Often, the proxy software generates this certificate. The decrypting proxy scenario affects both ThousandEyes Web Layer tests and Agent communication to ThousandEyes.

• Self-signed SSL server certificate

  A self-signed SSL server certificate will not chain back to a root CA certificate in the Enterprise Agent's standard certificate store. If correctly created, a self-signed certificate can be added to the Enterprise Agent's certificate store to eliminate certificate errors when the Agent performs tests to the server with the self-signed certificate.

  Additionally, if the target of the test does have certificates issued by a Certificate Authority whose root certificate is in the Agent's certificate store, but the target server does not return all needed intermediate certificates, and the customer cannot add the missing certificates on the server, then the intermediate certificate(s) can be added to the Enterprise Agent’s certificate store to create the “trust anchor” other than the root CA certificate. However, the customer should take great care in ensuring the veracity of any intermediate certificates, and understand and accept the impact the new trust anchor could have on other tests or aspects of the Agent operation, either in the present or in the future. ThousandEyes does not recommend adding intermediate certificates to an Enterprise Agent's certificate store.

Converting certificates into PEM format

To add a CA certificate to an Enterprise Agent, the certificate file must be in PEM format. This format can easily be recognized by viewing the file:

-----BEGIN CERTIFICATE-----
...
... (certificate content, base64 encoded)
...
-----END CERTIFICATE-----

If your CA certificate is in a format other than PEM format, either use one of the free online certificate converters (link #1, link #2 or link #3), or use the commands below to convert your CA certificate into PEM format with the openssl command line utility.

Convert a DER file (.crt, .cer or .der) to PEM format:

openssl x509 -inform der -in infile -out outfile.pem

Convert a PKCS#12 file (.pfx or .p12) to PEM format:

openssl pkcs12 -in infile -out outfile.pem -nodes

Installing on Virtual Appliances

Log into the Virtual Appliance's web management console, and click on the Network tab:

In the CA Certificate section, either paste the CA certificate into the Add CA Certificate field or browse to your PEM-formatted certificate file:

If pasting, ensure that whole certificate is pasted into the field, including the "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" markers at the beginning and end of the certificate.

Multiple CA certificates may be installed by copying and pasting each certificate, concatenating the certificates in the Add CA Certificate field:

Click the Save button at the bottom of the page to complete the operation.

Installing on supported Linux Distributions

When installing CA certificates on supported Linux distributions with the Enterprise Agent, CA certificates must be installed in two locations:

1. The system's CA certificate store

2. The NSSDB certificate store in the .pki/nssdb sub-directory of BrowserBot's home directory /var/lib/te-browserbot

   All commands should be executed as root. If logging into the system as a non-privileged user, begin each command with sudo.

The instructions below are provided in two sections for the two sets of supported Linux distributions: 1) Ubuntu and 2) Red Hat Enterprise Linux, CentOS and Oracle Linux. These instructions use the example certfile filename MY-CA-CERT.pem.

Ubuntu

Install the required packages for managing CA certificates:

apt-get install ca-certificates libnss3-tools

Copy the certfile file (in PEM format; see above for conversion information) into the /usr/share/ca-certificates directory of the Enterprise Agent:

cp MY-CA-CERT.pem /usr/share/ca-certificates/

Open the /etc/ca-certificates.conf file in a text editor:

nano /etc/ca-certificates.conf

Append a line containing only the certfile filename to the end of the file:

#...
#...(existing certificates here)
#...
MY-CA-CERT.pem

Multiple certificate files can be added using multiple lines.

Update the system certificate store using the update-ca-certificates command, as shown below, with successful output:

root@my-ubuntu-agent:~# update-ca-certificates
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:MY-CA-CERT.pem

done.
done.
root@my-ubuntu-agent:~#

The system CA certificate store now contains your CA certificate(s).

For BrowserBot, first create the directory that will contain the certificate store:

mkdir -p /var/lib/te-browserbot/.pki/nssdb

Initialize the certificate store in directory created above:

certutil -N --empty-password -d sql:/var/lib/te-browserbot/.pki/nssdb

Add the CA certificate into the newly created certificate store:

certutil \
  -d sql:/var/lib/te-browserbot/.pki/nssdb/ \
  -A \
  -t "C" \
  -n "MY-CA-CERT" \
  -i /usr/share/ca-certificates/MY-CA-CERT.pem

Change both the owner and the group of all newly created files and directories to browserbot:

chown -R browserbot.browserbot /var/lib/te-browserbot/.pki

Restart the te-browserbot service:

systemctl restart te-browserbot   # Ubuntu 16.04
restart te-browserbot             # Ubuntu 14.04
service te-browserbot restart     # Ubuntu 16.04 and/or 14.04

Red Hat Enterprise Linux, CentOS and Oracle Linux

Install the required packages for managing CA certificates:

yum install ca-certificates nss-tools

Copy the certfile file (in PEM format; see above for conversion information) into the /etc/pki/ca-trust/source/anchors/ directory of the Enterprise Agent:

cp MY-CA-CERT.pem /etc/pki/ca-trust/source/anchors/

Update the system certificate store using the update-ca-trust command:

update-ca-trust extract

The system CA certificate store now contains your the CA certificate.

For BrowserBot, first create the directory that will contain the certificate store:

mkdir -p /var/lib/te-browserbot/.pki/nssdb

Initialize the certificate store in the directory:

certutil -N --empty-password -d sql:/var/lib/te-browserbot/.pki/nssdb

Add the CA certificate to the newly created certificate store:

certutil \
  -d sql:/var/lib/te-browserbot/.pki/nssdb/ \
  -A \
  -t "C" \
  -n "certificate name" \
  -i /etc/pki/ca-trust/source/anchors/MY-CA-CERT.pem

On RHEL/CentOS 7 only, change both the owner and the group of all newly created files and directories to browserbot. On RHEL/CentOS 6 this step must be skipped:

chown -R browserbot.browserbot /var/lib/te-browserbot/.pki   # RHEL / CentOS 7 only!

Restart the te-browserbot service:

systemctl restart te-browserbot   # RHEL / CentOS 7
restart te-browserbot             # RHEL / CentOS 6

Installing on Docker

The Enterprise Agent Docker image has built-in support for CA certificates for both the Enterprise Agent and BrowserBot NSSDB store. On container startup, the supplied CA certificates will be installed in the container to be used for all tests, as well as communications with the ThousandEyes platform.

To use CA certificates in your Docker Enterprise Agent, create a directory on the Docker host that contains your CA certificate files. Add that directory to your Docker container run script as a volume mount pointing to /var/lib/te-agent/ca-certificates:

-v '/host/path/to/ca/certificates/':/var/lib/te-agent/ca-certificates

Restart the container for the certificate loading to occur.

Certificates are only processed on container startup. If any modifications are made to your certificate store, the container must be restarted.

Certificates in the volume mount must be regular files in PEG format and not in a subdirectory.

To verify certificate installation, check the Docker logs for your container:

Refreshing certificate store...
Installed certificate: /usr/local/share/ca-certificates/MY-CA-CERT.pem

Installing on Cisco Docker Devices

Starting with Cisco Application Hosting 5.0, Cisco Enterprise Agents will use apk instead of apt for auto-updates

To install a CA certificate on a Cisco Docker Device:

1. Ensure your Docker configuration includes the SSL decrypting proxy location before running the Docker:

   Device(config)# app-hosting appid example
   Device(config-app-hosting)# app-resource docker
   Device(config-app-hosting-docker)# prepend-pkg-opts
   Device(config-app-hosting-docker)# run-opts 1 "-e TEAGENT_ACCOUNT_TOKEN"
   Device(config-app-hosting-docker)# run-opts 2 "--hostname DESIRED_AGENT_HOSTNAME"
   Device(config-app-hosting-docker)# run-opts 3 "-e TEAGENT_PROXY_TYPE=STATIC"
   Device(config-app-hosting-docker)# run-opts 4 "-e TEAGENT_PROXY_LOCATION=<proxy-ssldecrypting.com>:3128"
   Device(config-app-hosting-docker)# exit
   Device(config-app-hosting)# exit
   Device(config)# exit

   If the proxy configuration is missing, stop and deactivate the application. Edit the configuration as needed, then activate and restart the application. For more information on the steps to stop/start the application, see the Lifecycle of an Application section of the Cisco documentation.
2. Ensure the container is in the RUNNING state:

   Device#sh app-hosting list
   
   App id                               	      State
   ---------------------------------------------------------
   example                                  	RUNNING

3. Connect to the container shell:

   Device#app-hosting connect appid example session

4. Confirm that the agent registration fails due to the proxy requiring a CA certificate in the agent logs:

   root@example:/home# tail /var/log/agent/te-agent.log
   2022-06-30 03:35:52.211 INFO  [9997ff00] [te.agent.ClusterMasterAdapter] {} Attempting to get agent id from https://sc1.thousandeyes.com
   2022-06-30 03:35:52.242 ERROR [9997ff00] [te.agent.status] {} Error calling createAgent: Curl error - Peer certificate cannot be authenticated with given CA certificates

   You can also use the curl command, replacing the proxy and port variables:

   # curl -v https://sc1.thousandeyes.com -x <proxy>:<port>

   If there is no certificate, you will see output similar to the example below:

   * TLSv1.3 (IN), TLS handshake, Server hello (2):
   * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
   * TLSv1.3 (IN), TLS handshake, Certificate (11):
   * TLSv1.3 (OUT), TLS alert, unknown CA (560):
   * SSL certificate problem: unable to get local issuer certificate

   The expected response is “404 Not Found”, validating that while the agent still needs a CA certificate, it does have access to the Internet through proxy.

5. Create a CA certificate file in /usr/share/ca-certificates:

   1. Option One: Manually create the file, and add the PEM content:

      1. Create a new PEM file in the following location:

         #vi /usr/share/ca-certificates/MY-CA-CERT.pem

      2. Copy and paste your PEM file content, and save the changes.

   2. Transfer the PEM certificate:

      1. Copy the PEM file to the switch’s flash directory using SCP, FTP, TFTP, or USB storage:

         Device#copy ftp://10.100.21.239/MY-CA-CERT.pem flash:

      2. Transfer the file to the application:

         Device#app-hosting data appid example copy bootflash:MY-CA-CERT.pem MY-CA-CERT.pem
         
         Successfully copied file /flash/MY-CA-CERT.pem to cat9kproxy as MY-CA-CERT.pem

      3. Files transferred from the switch to the application are stored in the /iox_data/appdata directory. This folder is created inside the container once the application is running. You will need to move the file to the correct location after it is transferred:

         Device#app-hosting connect appid example session bash
         example# mv /iox_data/appdata/MY-CA-CERT.pem /usr/share/ca-certificates/MY-CA-CERT.pem

         Note: Files can only be transferred from the switch to the application and not vice versa.

         Warning: While files stored in /ios_data/appdata persist, if the container is deactivated or reloaded, any certificates will be removed, and the step above will need to be repeated, along with the rest of the instructions below.

6. Append the name of the created CA certificates file in /etc/ca-certificates.conf:

   # vi /etc/ca-certificates.conf
   … 
   thousandeyes/Microsoft_Root_Certificate_Authority_2011.crt
   MY-CA-CERT.pem
   :wq

7. Execute the command update-ca-certificates and wait for a successful output:

   # update-ca-certificates
   Updating certificates in /etc/ssl/certs…
   1 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d…
   done.

8. Uninstall the cisco-core-trsb service. This step is only required for Cisco Agents:

   # apk del --purge cisco-core-trsb

9. Restart the te-agent service:

   # sv restart te-agent
   # sv status te-agent
   
   Run: te-agent: 1s, normally up, want up

10. Check the agent log and make sure the previous curl error from step 4 disappears. If it no longer appears, the CA certificate has been installed correctly.

The CA certificate is now installed. If the agent uses browserbot, continue to the next section to import the certificate into the browserbot store.

Import the Certificate into the BrowserBot Store

The following steps should be completed after the previous instructions (Installing on Cisco Docker Devices), to import the certificate to the browserbot store:

1. Install the Certutil tool:

   example# apt-get install ca-certificates libnss3-tools

2. Create the directory that will contain the certificate store:

   example# mkdir -p /var/lib/te-browserbot/.pki/nssdb

3. Initialize the certificate store in the new directory:

   example# certutil -N --empty-password -d sql:/var/lib/te-browserbot/.pki/nssdb

4. Add the CA certificate into the newly created certificate store:

   certutil \
    -d sql:/var/lib/te-browserbot/.pki/nssdb/ \
    -A \
    -t "C" \
    -n "MY-CA-CERT" \
    -i /usr/share/ca-certificates/MY-CA-CERT.pem

5. Change both the owner and group of all newly created files and directories to browserbot:

chown -R browserbot.browserbot /var/lib/te-browserbot/.pki

1. Restart the te-browserbot service:

sv restart te-browserbot