AWS for Test Recommendations

The ThousandEyes integration with Amazon Web Services (AWS) allows you to monitor the critical AWS resources you host within Amazon's API Gateway, CloudFront and Global Accelerator services. ThousandEyes provides a quick and easy way to securely connect to your AWS account, discover your API endpoints (for API Gateway), domain names (for CloudFront), and load balancers (for Global Accelerator), and generate the appropriate test recommendations to help monitor these resources in ThousandEyes.

This article describes how to configure the ThousandEyes integration with AWS for test recommendations, set up the resulting recommended tests, and view their results. The basic steps are:

  1. Check the prerequisites.

  2. In ThousandEyes and AWS, create the integration.

  3. Create recommended tests to monitor the services you rely on.

  4. View the results of your tests.

  5. View the dashboard (for API Gateway tests only) that provides an overview of your API performance.

  6. View the alerts associated with your tests and set up notifications.

This integration and its resulting recommendations, tests and template are only accessible and visible in the account group within which it is created.

Prerequisites

You must have the following account requirements and permissions to successfully set up your AWS integration.

Account Requirements

To use the features described in this document, you must have both of the following:

AWS Permissions

In AWS, you need permissions to create roles and policies. How you attain these permissions depends on which interface you use (e.g., console or command line). Make sure you have the correct permissions according to your interface preferences; see Permissions required to access IAM resources

ThousandEyes Permissions

To create the integration, you must have an Organization Admin or Account Admin role in the ThousandEyes platform. To enable or activate test recommendations, you must have an Organization Admin role. For more information about ThousandEyes roles, see Role-Based Access Control, Explained.

Creating the AWS Integration

To create your ThousandEyes integration with AWS for test recommendations, it is easiest to be logged into both services at the same time, as you will need to provide information within both services that is dynamically generated within the other.

View a simplified flow of steps between ThousandEyes and the AWS IAM Console below.

  1. In ThousandEyes, go to Integrations from the left menu.

  2. Click + New Integration in the top right.

  3. Select Amazon Web Services from the available options. An Add AWS Integration side panel opens.

  4. Enter the values for your AWS instance as follows:

  • Name: Give your integration a unique name. Duplicate names are not permitted.

  • ThousandEyes Supported Services: Make sure Test Recommendations is selected.

At this point, you will need to switch to your AWS account to create an IAM Role for ThousandEyes. You can choose to do this in the AWS Console or in the Command Line Interface.

Note: You must be logged in as an IAM Administrator to complete the steps below.

For the best and most up-to-date information regarding steps to take in AWS, view AWS’s integration documentation through the links provided.

AWS Console

  1. If using the AWS Console, go to https://console.aws.amazon.com/iam/.

  2. Create the permission policy first, using the steps outlined in Creating policies using the JSON editor.

  • Where AWS prompts you to paste in a “JSON policy document”, paste in the permission policy generated in the ThousandEyes Add AWS Integration panel. There is a “copy” icon on the Permission Policy header in ThousandEyes for this purpose. The permission policy covers permissions for API Gateway, CloudFront and Global Accelerator by default.

  • Note: you do not need to follow any of AWS’s optional steps.

  1. Next, follow the steps outlined in Creating an IAM role using a custom trust policy (console). These steps include creating the trust policy and the role for ThousandEyes.

  • Where AWS prompts you to paste in a “custom trust policy”, paste in the trust policy generated in the ThousandEyes Add AWS Integration panel. There is a “copy” icon on the Trust Policy header in ThousandEyes for this purpose.

  • Where AWS prompts you to Add permissions, locate in the generated list the permission policy you created in Step 2 and select the checkbox next to its name to assign it to the role.

  • Note: you do not need to follow any of AWS’s optional steps.

  1. Now that you have created an IAM Role for ThousandEyes, you need to collect the Account Resource Name (ARN) from AWS to paste it into the ThousandEyes Add AWS Integration panel.

  • In the AWS Console at https://console.aws.amazon.com/iam/, click on Roles.

  • In the list of roles that appear, scroll or search for the role that you just created, and click on it.

  • At the top of the resulting page, you will find your ARN in the Summary box.

  • Use the “copy” symbol next to the ARN to copy it.

  1. Return to the Add AWS Integration panel on the ThousandEyes platform.

  • Paste into the Account Resource Name (ARN) field the ARN that you just copied. It will look something like arn:aws:iam::123456789098:username.

  • When all fields in ThousandEyes and AWS are completed, click Test.

  • A callout box at the top of the panel tells you whether testing succeeded or failed. If failed, see Failed Testing for next steps.

  • If testing succeeded, click Save.

The side panel closes back to the Integration screen and two messages appear in the bottom right of your screen. The first confirms success of the saved integration. The second explains that the service discovery is taking place. While this is happening, your integration shows a status of Pending. Note that you cannot set up your recommended tests until service discovery is complete, resulting in a status of Connected.

AWS Command Line Interface (CLI)

Unlike with the AWS Console, in the CLI, you create the role and trust policy first, then add the permission policy after. You can find additional instructions and explanation of how to set up an IAM Role using the CLI from Configuring and using a role.

The following assumes you are using AWS CLI version 2. However, the same steps will work with version 1.

  1. Open your preferred terminal.

  2. In ThousandEyes, go to Integrations from the left menu.

  3. Click + New Integration in the top right.

  4. Select Amazon Web Services from the available options. An Add AWS Integration side panel opens.

  5. Select the services you would like to integrate with within ThousandEyes using the ThousandEyes Supported Services dropdown.

  • The permission policy updates dynamically based on the services you choose in this field.

  1. Open and copy the commands within the Trust Policy field. There is a “copy” icon on the Trust Policy header for this purpose.

  1. Paste the commands within the terminal.

  2. Enter your preferred --role-name and --description as required.

  3. Hit Enter.

  4. Check that the role was created successfully: for successful create-role commands, the whole role will be returned in the response body.

  • If role creation fails, you will receive an error message explaining what error has occurred.

  1. Within the response body is the new role’s Account Resource Name "ARN". Copy the ARN.

  2. Go back to the Add AWS Integration dialog on ThousandEyes.

  3. Paste into the Account Resource Name (ARN) field the ARN that you just copied.

With the role created, you now assign the permission policy to the role.

  1. In the ThousandEyes Add AWS Integration panel, copy the commands within the Permission policy field. There is a “copy” icon on the Permission policy header for this purpose.

  1. Paste the commands within the CLI.

  2. Enter your preferred --role-name and --policy-name as required.

  3. Hit Enter.

  4. Check that the permission was assigned successfully. For the put-role-policy command, no success response is returned. However:

  • If permission assignment fails, you will receive an error message explaining what error has occurred.

  1. Go back to the Add AWS Integration dialog on ThousandEyes.

  2. Click Test.

  3. If Testing was successful, click Save.

Successful Integration

View your completed integration on the Integrations screen. Note its status in the Status column, and the services you selected in the Supported Services column.

If the Status column shows Connected, your integration is ready and you can start using ThousandEyes test recommendations. Note that for AWS integrations, you can only have a status of Pending, Connected or Failed.

A status of Connected means that the ThousandEyes platform was able to generate at least one recommendation. If you do not see all the recommendations that you expect in the Recommendations dialog, or if your status is Pending and does not resolve to Connected or Failed after 30 minutes (note, you will need to refresh the page to see the status change), you may have encountered a backend error. You will need to raise a ticket with ThousandEyes Support.

Failed Integration

If your integration didn’t pass saving, your integration status will read Failed. Failure may result from, for example, trying to create a duplicate AWS integration in the same account group, or not having the right permissions. Click on your integration to open the side panel and to check your settings in the dialog. Ensure that you have entered valid AWS credentials and that you are an IAM Administrator.

If the integration continues to fail, contact ThousandEyes Support.

Creating Recommended Tests

Now that you have set up the integration, put it to work: the connection between AWS and ThousandEyes brings information into ThousandEyes about AWS services such as API Gateway, CloudFront and Global Accelerator in your AWS account. With this information, the ThousandEyes platform recommends tests, test configurations, and templates including dashboards (for API Gateway only), labels, and default alerts, to bring you quick, full visibility of those services. For more information about how to deploy templates, see Templates.

To set up recommended tests, do the following:

  1. Go to Cloud & Enterprise Agents > Test Settings. At the Add New Test button, click the dropdown arrow and select Add From Recommendations.

  • Alternatively, find your integration on the Integrations screen, open the ellipsis dropdown menu, and click View Test Recommendations.

  • The Recommendations dialog appears. It lists all the integration instances for test recommendations you have set up, including those for the AWS.

  • API Gateway instances are listed as separate items. All CloudFront and Global Accelerator resources are grouped into a single instance, as displayed in the image below.

By default, the ThousandEyes platform names each API Gatweway instance with the API endpoint name from the API Gateway followed by its staging environment. Any APIs that belong to separate regions are also given separate instances. For example, if on the API Gateway you named your API endpoint “Fred” and it belonged to both the “Astaire” and “Mercury” environments in US-West-1 and US-East-1, ThousandEyes would create four instances in the test recommendations dialog; two called “Fred-Astaire” – one for each region – and two called “Fred-Mercury” – one for each region. You can change the instance name if you wish during your test template setup. You will find your Integration Name on the second line below the instance name. You can search for your integration instance via the search bar or **Integration Instance** filter.

  1. In the Step 1 of 4 – Select Instance pane, select the resources you wish to configure.

  • Note: you can select a maximum of 50 resources at any given time.

  1. In Step 2 of 4 – Select Targets (for API Gateway) or Step 2 of 4 – Select Locations (for CloudFront and Global Accelerator), select the desired targets/locations and methods (for API Gateway only) among the list that you wish to configure tests for.

  • For CloudFront and Global Accelerator tests, a banner appears above the location table indicating how many resources (absolute and percentage) are currently monitored by ThousandEyes tests. Those that are already currently monitored are greyed out and cannot be selected for deployment as part of the template.

  1. Click Configure Tests.

Step 3 of 4 – Configure tests shows the Global Settings pane.

Your settings in the Global Settings pane will apply to all the tests you are creating for all the resources you just selected, which are listed on the left of the pane.

  1. Select the agents that should run the tests you are creating.

  • A quick way to select appropriate agents is to use the built-in labels in the agent selection drop-down.

  • For backend services, we recommend that you use Enterprise Agents for monitoring. Note: Enterprise Agents must have BrowserBot installed to run API tests (see What is BroswerBot for more information). For frontend services, we recommend that you use Cloud Agents for monitoring. For more information on agent types and their differences, see Global Vantage Points.

  1. Set the frequency at which the tests should run.

  2. Give your instance test template a new name if you wish (see note in Step 1, above).

  3. [Optional] If you want to override the global settings for individual tests, you can do this now by selecting each test from the list under Global Settings. If not, the configuration in Global Settings will apply to all the tests you are creating. You can also adjust your tests’ settings at any point in the future.

  1. Click Review.

  2. In Step 4 of 4 – Review template, review the selections you’ve made and the tools – tests, dashboards (for API Gateway only), and labels (given the same name as your instance name) - that will be created.

  1. Click Deploy Now.

  2. Finally, view the tests (or dashboards if configuring API Gateway) you just created from the confirmation screen of the Recommendations dialog.

  • For test settings, click Go to Test Settings. Click the arrow next to your new tests to see their configuration details. For detailed information about the Test Settings screen, see Tests.

  • For dashboards (for API Gateway only), click Go to Dashboards. This takes you directly to the dashboard you just created from the test recommendations dialog. Note that some widgets will take time to load and show data as testing will have only just begun. For detailed information about viewing dashboards, see Dashboards.

  1. To finish without viewing your test settings or dashboards, click Done.

Configuring API Gateway Tests Post-Deployment

Some API Gateway tests require configuration beyond what is possible in the recommendations dialog. While this does not affect successful completion of the dialog, the tests that fall into this category will not run until you complete their configuration in Cloud & Enterprise Agents > Test Settings. You will know whether any of your tests fall into this category after step 5, above, from Creating Recommended Tests. On the Step 3 of 4 – Configure tests pane, a warning message will appear to inform you that at least one of the tests you are setting up will require further configuration after deployment.

Generally, any test targeting API Gateway that requires specific headers, parameters or request paths will need post-deployment configuration.

There are a number of ways to access the test settings screen to finish your configuration:

  1. On the final screen of the recommendations dialog, you will find a link to the test settings screen.

  1. If you have already clicked away from the recommendations dialog, go to Cloud & Enterprise Agents > Test Settings.

  2. In the Notifications dropdown at the top of the screen, find a link to the test settings screen in the notification about your unfinished test configurations.

From here, follow these steps to finish your test configuration.

  1. When you arrive at Test Settings, you will see another warning message. Click the Filter Tests button to view only those tests that need further configuration. Each partially configured test displays a yellow warning symbol next to the Type.

  1. Click on the test row to open its settings pane.

  • Another warning message lists the variables that require configuration to complete test setup.

  1. Click the 1-Step API button in the Target API field.

  • This opens a separate dialog where you find further instructions to help you complete configuration.

  1. Click the Variables button under Environment Setup for a summary of the required variables.

  1. Type the values required into the “TODO” fields.

  2. Follow any other on-screen messages until no further warning messages are displayed.

  3. Click Save API Configuration. This closes the API Configuration dialog.

  4. Click Save and Enable. This closes the test setting dialog.

  • The warning symbol next to Type disappears automatically, the test is no longer grayed out, and the Enabled box is checked.

  • Note: clicking Save instead of Save and Enable saves your new configurations but does not make the test run.

  1. Complete these steps for all partially configured tests you wish to run.

Viewing your Tests

To view and manage the tests that you have just created:

  1. Go to Cloud & Enterprise Agents > Test Settings.

  2. Scroll or search for your AWS test and click to open the settings pane.

  • The pane opens up to the Basic Configuration tab.

  1. Edit your test settings using the Configuring Test Settings guide.

Viewing Your Test Data

Now that you have created the recommended tests, view the test results.

With your AWS integration, you have access to Views 2.0 (as well as the legacy functionality in Views 1.0).

  1. Go to Cloud & Enterprise Agents > Views.

  • For information on the elements of the Views screen, see Views 2.0.

  1. In the Test dropdown field, find and open your AWS test.

  • Your views pane will immediately update to show your AWS test.

To learn how to view and understand your AWS test data, see Reading the API Test Views.

Viewing Your API Gateway Dashboard

When you set up your recommended tests - for API Gateway instances only - ThousandEyes automatically creates a dashboard that includes an overview of the health of the AWS APIs you maintain there and the status of the tests you have deployed.

To access your dashboard:

  1. On the main menu, go to Dashboards.

  2. In the dashboard ribbon, open the dropdown within the Dashboard button.

  3. Scroll or search for the name of the instance test template you just created (tip: they are listed in order of most recently created).

  4. Click on your instance test template name.

Note: you cannot add a dashboard widget relating to your API tests to another dashboard.

Viewing Your Alerts and Notifications

When you set up your recommended tests, ThousandEyes automatically assigns alerts to your tests to ensure you are made aware when any of your AWS services underperform.

How Alerts Are Assigned

Alerts are assigned to your test according to its test type and default status. If the blue check is selected within the Default column of the Alerts > Alert List screen, and relate to your test type or nested test type, it will be assigned to your test.

While an alert may have “Default” in its title (e.g., the Default SSL Certificate Expiry Rule 2.0 alert in the image below), it is not assigned to the test unless the blue check in the Default column is selected.

For example, in the case of an API test type, the test includes the API test type in the Web layer, and nested beneath it the Agent to Server and Path Trace test types in the Network layer, and the BGP test type in the Routing layer. Therefore, any alerts with the default check selected and showing those test types in the Type column, will be assigned to your API test.

For more information about test layering, see Tests. For more information about how to view alerts, see Alerts.

Viewing Your Assigned Alerts

To see which alerts have been assigned to your tests:

  1. Go to Cloud & Enterprise Agents > Test Settings.

  2. Scroll or search for your API test and click to open the settings pane.

  3. Scroll to the Alerts section.

  4. Open the Alert Rules field and click Selected under the search bar.

  1. View the full list of alerts assigned to your test.

To view any alerts assigned to your tests that may be currently active:

  1. Go to Alerts > Alert List.

  2. Search by Test Name for your tests.

  • Any alerts that are assigned to your tests appear.

  1. Click any alert to open a side panel with more details and links to view the data for each monitored object (e.g. agent or BGP monitor).

Setting Up Alert Notifications

If your email address is already listed on the Notifications tab of the default alerts for tests within your account group, then the alerts assigned to the tests you just created will automatically notify you when your API test alerts fire. If your default alerts also have webhooks or integrations set up for you, then you will also be automatically notified through those other media when your API alerts fire.

If you are not on the email list of notifications for default alerts in your account group, or the person who needs to be notified is not on the email list:

  1. Follow the guide to setting up alert notifications on Creating and Editing Alert Rules.

  • You will need to set up a new notification for each alert rule assigned to your new test(s).

  1. [Optional] If you would like to set up reception of notifications by other methods, follow the steps for setting up notifications via SMS, custom webhooks, or our custom-built integrations.

PreviousAppDynamics for Test RecommendationsNextAWS for Cloud Insights

Last updated