AWS for Cloud Insights
Limited Preview: Any information provided in this document regarding future functionalities is for informational purposes only and is subject to change including ceasing any further development of such functionality. Many of these future functionalities remain in varying stages of development and will be offered on a when-and-if available basis, and Cisco makes no commitment as to the final delivery of any of such future functionalities. Cisco will have no liability for Cisco's failure to deliver any or all future functionalities and any such failure would not in any way imply the right to return any previously purchased Cisco products.
Cloud Insights allows you to discover your cloud infrastructure and monitor the flow of traffic within or across your virtual private clouds. The discovered cloud infrastructure and traffic flow is time-correlated with your Cloud and Enterprise Agent views to combine performance measurements captured from your Cloud or Enterprise Agents with cloud infrastructure configuration changes and traffic flow. To learn more about how Cloud Insights can work for you, see Cloud Insights.
In this article, we’ll explain how to create the integrations you need with Amazon Web Services (AWS) to get your ThousandEyes Cloud Insights up and running. AWS for Cloud Insights consists of two separate integrations per each AWS account you want to monitor. Inventory Monitoring is mandatory and allows you to monitor your AWS account assets, topology and config changes. Flow Logs Monitoring is an optional add-on and allows you to monitor your AWS account traffic using the AWS VPC (Virtual Private Cloud) flow logs.
Prerequisites
You must have the following AWS account and ThousandEyes organization requirements and permissions to successfully set up your Cloud Insights integrations.
Account/Organization Requirements
To use the features described in this document, you must have both of the following:
An active account for Amazon Web Services.
An active organization for ThousandEyes.
AWS Permissions
For the inventory monitoring integration, you need AWS permissions to create roles and policies. How you attain these permissions depends on which interface you use (e.g., console or command line). Make sure you have the correct permissions according to your interface preferences; see Permissions required to access IAM resources.
For the flow logs monitoring integration, you need the above, plus (if not already permitted):
VPC permissions to enable/configure VPC flow logs.
S3 permissions to create the S3 (Simple Storage Service) flow logs bucket and enable event notification to the SNS topic.
SNS permissions to create an SNS (Simple Notification Service) topic to which the S3 events will be sent and to create the role for ThousandEyes to register with.
ThousandEyes Permissions
To create the integrations, you must have an Organization Admin or Account Admin role in the ThousandEyes platform. For more information about ThousandEyes roles, see Role-Based Access Control, Explained.
Overview of the Cloud Insights Integration Architecture
What ThousandEyes Collects from AWS
To take full advantage of Cloud Insights, you need to create two distinct integrations which capture two types of data and serve different purposes. These are:
Inventory monitoring is used to collect inventory and configuration information from your AWS accounts over time. We use this data in Cloud Insights to:
Display your AWS network assets, including their types and locations within Cloud Insights > Inventory.
Track asset configuration changes over time in the form of events, viewable in both Cloud Insights > Inventory and Cloud Insights > Views.
Display within Cloud and Enterprise Agents > Views your cloud network topology and events.
Enrich the flow log data with resource information once the flow logs integration is set up.
Flow logs monitoring is used to track real traffic flows in your AWS network. We use this data, for example, to:
Display outbound/inbound throughput for each entity in Cloud Insights > Views and Cloud and Enterprise Agents > Views.
Highlight rejected traffic, visible in Cloud Insights > Views.
Cloud Insights cannot work without at least the inventory monitoring integration, though full capability is only available with both integrations per AWS account monitored; flow logs monitoring cannot surface Cloud Insights data as an integration on its own.
How ThousandEyes Manages the Integrations
In ThousandEyes, each organization is divided into account groups. The Cloud Insights integrations are account group-specific. This means that an integration created in one account group is not shared with another account group. Each account group can set up integrations to monitor AWS accounts in your AWS network.
The monitored AWS accounts in your network can span one or more regions. The flow-logs used for Cloud Insights are fetched per AWS location, which is a combination of the AWS account and region, for example: 351945360856, us-west-1.
In the image below, three different account groups within a ThousandEyes organization each monitor one or more AWS accounts. Each monitored AWS account spans one or more regions. Account Group 1 monitors three AWS locations (two AWS accounts, the first in two regions and the second in one region), Account Group 2 monitors one location (a single AWS account in one region), and Account Group 3 monitors two locations (a single AWS account in two regions).
Currently, each monitored AWS account requires a separate inventory monitoring integration in ThousandEyes. If you wish to monitor two AWS accounts within the same account group, you must set up two separate inventory monitoring integrations in that account group, as the integrations each require a separate AWS IAM (Identity and Access Management) role for ThousandEyes.
Likewise with flow logs monitoring. Currently, for each AWS account that stores flow logs in buckets you want to fetch data from, you need to set up separate integrations within the same account group. If an account group wants to monitor flow logs in buckets from two AWS accounts, the account group must set up two separate flow logs monitoring integrations – one per AWS account where flow logs are stored.
How the Integrations Work Together
ThousandEyes uses inventory monitoring integrations to access the AWS accounts that you want to monitor with Cloud Insights. These monitored accounts may also contain VPCs that generate traffic flow information via AWS flow logs. The logs from the VPCs are published to S3 buckets, which can reside in the same or different AWS accounts. You configure event notifications for these buckets to be sent to SNS topics, which we subscribe to to be alerted whenever a new flow log files is available.
The purpose of the flow logs integrations is to access the log files in these buckets for ingestion by ThousandEyes. You should aim to surface only traffic from accounts and VPCs explicitly selected for monitoring through inventory integrations. Other traffic will be filtered out or marked as external traffic.
Therefore, for Cloud Insights to work optimally, you need to identify the AWS accounts you want to monitor and create inventory monitoring integrations for them. You then locate the S3 buckets where the corresponding VPC flow logs are stored (or create them) and set up the flow logs monitoring integrations accordingly.
Flow logs are typically published from VPCs in a given region to an S3 bucket within the same region to avoid extra AWS fees associated with cross-region data transfer. Customers with multiple accounts across various regions may choose to create a local bucket for each account in each region, as in the image above. In this example, because separate integrations are needed per account containing flow logs, the customer would need to set up two flow logs monitoring integrations.
Alternatively, customers can send flow logs from monitored accounts to a separate account with buckets in each region, potentially reducing the number of accounts and integrations that need to be managed, as in the image below. In this example, because all the flow logs are contained within one account, the customer only needs to set up one flow logs monitoring integration.
Flow Logs Monitoring Overview
Flow logs monitoring itself is done in two steps. These steps are:
ThousandEyes receives notification of a new flow log via subscription to your SNS topic.
In your AWS network, event notifications are published from the monitored flow logs bucket to an SNS topic in the same region.
We subscribe to your SNS topic to get a notification whenever a new flow log is created.
ThousandEyes downloads the flow log files from your S3 bucket.
You create an IAM Role that we can assume and use to fetch the flow log files from your S3 bucket.
This means that for each monitored AWS account, we need access to both the S3 buckets that contain the flow logs and the SNS topics that alert us to their existence.
The image below shows the flow logs:
Originating in your VPC.
Getting sent to the S3 bucket for storage.
Triggering an event notification into an SNS topic.
ThousandEyes receiving the event notification through a subscription to that topic.
ThousandEyes then retrieving the flow logs from the S3 bucket.
Each integration connects us to a single account where your flow log buckets are stored. For each integration:
We need an SNS topic per region (in the same account) to get notifications of new flow logs.
We need an IAM Role to enable read-only access to the buckets.
Creating the Inventory Monitoring Integration
The integration for AWS inventory monitoring gives ThousandEyes secure access to your AWS account information and data.
To set up inventoring monitoring:
Go to the Integrations screen.
Click + New Integration in the top right.
In the Add New Integration side panel that opens, select Amazon Web Services.
The resulting < Add AWS Integration screen defaults to showing the fields required for the “Test Recommendations” service.
Select “Inventory Monitoring” from the ThousandEyes Supported Services dropdown.
Name your integration.
Give your integration a unique name. Duplicate names are not permitted.
You must then use the generated trust and permission policies to create the IAM Role that will give ThousandEyes access to your AWS account.
You do this the same way as described in AWS for Test Recommendations.
Follow steps 1-5 under AWS Console, except don’t yet test the integration.
At the end of this process, you should have pasted the ARN for the IAM Role into the corresponding field in the ThousandEyes < Add AWS Integration screen.
Note: The trust policy for the inventory monitoring integration will look the same as the trust policy for the test recommendations integration. However, the permission policies differ. Do not paste a permission policy from a test recommendation integration into an IAM Role intended for an inventory monitoring integration. Selecting the “Inventory Monitoring” service automatically updates the code (or commands, depending on which interface you choose to create the integration) in the required permission policy.
Click Test.
Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.
If testing was successful, click Save.
See Checking Your Integrations Are Working for information about what happens after you save your integration.
Creating the Flow Logs Monitoring Integration
Steps within AWS
Before you begin to create the integration, you first need to have several items set up within your AWS account for the integration to be a success. Follow the links in each step to find instructions within AWS.
Create a bucket to collect flow logs in each region that your account covers in the S3 console.
Set your VPCs to publish flow logs to this bucket in the VPC console.
Under the Flow log settings, set:
Filter to “All” to see accepted and rejected traffic flows.
Maximum aggregation interval to either 10 minutes or 1 minute, depending on the granularity of data you need. Flow logs with a maximum aggregation interval of 1 minute produce a higher volume of flow log records than flow logs with a maximum aggregation interval of 10 minutes.
Destination to “Send to an Amazon S3 bucket”.
S3 bucket ARN to the bucket where you’d like the flow logs to be published from this VPC – you retrieve this from step 1, above.
Log record format to custom (because not all the fields use the default setting), and containing the following fields as mandatory:
<account-id><action><bytes><dstaddr><dstport><end><flow-direction><interface-id><log-status><packets><protocol><srcaddr><srcport><start><tcp-flags><traffic-path><version>
All other fields are optional, while users can include any Standard attributes.
Repeat this step for every VPC you want to monitor in this region.
Note: It is typically best practice to configure all your VPCs in a particular region to publish flow logs to the same regional S3 bucket to avoid cross-region traffic costs.
Create an SNS topic in each AWS region where you are monitoring a bucket in the SNS console.
Select “Standard” under Type of topic.
Configure event notifications for each regional SNS topic in the S3 console.
Skip Step 1: Create an Amazon SQS queue; it is not required for this integration.
In the Event types section choose “All object create events (s3:ObjectCreated:*)”.
In the Destination section, choose “SNS topic” and enter or select the SNS topic ARN where you want to send the notifications, retrieved from step 3, above.
You will come back to some of these steps as you create the integration in the next section, so it’s best to have both accounts open during the following steps.
Steps within ThousandEyes
Follow steps 1-6 in Creating the Inventory Monitoring Integration, except choose "Flow Logs Integration" from the ThousandEyes Supported Services dropdown.
This not only updates the permission policy for the flow logs integration and creates the IAM role that will give ThousandEyes access to your S3 bucket resources, but reveals the additional fields required to successfully implement this integration.
Update the access policy of your SNS topics to allow ThousandEyes to subscribe to these topics.
In your AWS account, find the SNS topics you created in step 3 of Steps within AWS for each region in the S3 console.
In each one, click Edit and open the Access policy dropdown to reveal configuration options.
Under the Advanced option, paste into the JSON editor the Access Policy generated within the ThousandEyes < Add AWS Integration screen.
Replace
<TOPIC-ARN>with the ARN of your specific SNS topic.
Add all relevant SNS topic ARNs to the fields at the bottom of the < Add AWS Integration screen. You can add as many as necessary for each account.
Click + Add SNS Topic ARN to add a topic.
Click the “minus” sign to the right of each topic to remove it.
Click Test.
Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.
If testing was successful, click Save.
Checking Your Integrations Are Working
After saving your integrations, ThousandEyes will start monitoring the resources specified in their policies. During this process, each integration status displays as “Pending”.
Inventory and flow logs are monitored every five minutes so you can expect to see the "Pending" state updated within five to ten minutes. Any flow log errors will be detected once we receive flow log files. If we encounter any issue that causes the monitoring to be unsuccessful, we will surface the error on the integration screen and move the status to “Failed”. In most cases, you need to fix the permissions and click Save to restart the monitoring.
Troubleshooting Cloud Insights Integrations
Inventory Monitoring Errors
ThousandEyes unable to assume a role for your monitored account:
If ThousandEyes cannot assume the IAM role for your monitored AWS account, you will encounter the following error if you try to either test or save the integration.
Either a role with this ARN doesn't exist, or the role's trust policy is not configured to allow ThousandEyes to assume the role. In the latter case, you must re-paste the trust policy into AWS, as outlined in step 3 under AWS Console.
Note: if you disable the trust policy in AWS at some point, but do not delete the integration within ThousandEyes, the integration will transition out of the status of Connected, and will not resolve back to Connected until you re-instate the trust policy.
Cannot duplicate the integration:
• Each inventory monitoring integration is associated with a unique role ARN, and the same role ARN cannot be used across multiple inventory integrations per account group (note that role ARNs can be used for corresponding inventory monitoring and flow logs monitoring integrations). If you try to duplicate an inventory monitoring integration using the same role ARN as an existing inventory monitoring integration, you will encounter the following error and be unable to save the new integration:
Cannot access certain AWS resources:
The permission policy lists a number of API types to which you grant ThousandEyes read access. You define in AWS which specific APIs fall into each API type. If you decide to limit ThousandEyes’ access to a subset of APIs within these types, your integration will still work, but you will encounter error messages in the sidebar describing which APIs we have been denied access to (note, each error message is specific to the AWS resource being restricted – the message below is an example):
Important: When you deny ThousandEyes access to some AWS resources (but not all), the integration status shows as Connected since all other resources granted read access permission will continue to work as part of the integration. You must click open the integration's editing sidebar to see the error messages listing out which APIs you have denied us access to. Due to space, only the first few errors reported are displayed in the sidebar.
You can restrict read access to other resources in AWS, such as regions. If you disallow read access to ThousandEyes for any regions, ThousandEyes will have access to all other regions, but you will receive an error message explaining that a region(s) was restricted:
Flow Logs Monitoring Errors
Common errors you may encounter while setting up the flow logs integration in ThousandEyes are explained below. For common errors when configuring your AWS VPCs, S3 buckets, and SNS topics or events, visit the relevant topics at https://docs.aws.amazon.com/.
Integration Creation Errors
Cannot duplicate the integration:
Each flow logs monitoring integration is associated with a unique role ARN, and the same role ARN cannot be used across multiple flow logs integrations per account group (note that role ARNs can be used for corresponding inventory monitoring and flow logs monitoring integrations). If you try to duplicate a flow logs monitoring integration using the same role ARN as an existing flow logs integration, you will encounter the following error and will not be able to save the new integration:
SNS topic ARNs belong to a different account than the role ARN:
Each flow logs monitoring integration is associated with a single AWS account, which corresponds to the account of the role ARN. The AWS account of the SNS topics specified in the integration must match the account of the role ARN. If you attempt to create an integration where any of the topics belong to a different account than the role ARN, you will encounter the following error and will not be able to save the integration. In this example, the second SNS topic ARN belongs to a different account than the role ARN.
SNS topic ARN appears in more than one integration:
The same SNS topic ARN cannot appear in more than one integration per account group or the integration will fail and the following error message will appear. The message includes the ARN of the topic(s) which is already defined in another integration.
ThousandEyes is unable to assume a role for accessing your S3 buckets:
If ThousandEyes cannot assume the IAM role provided in the integration, you will encounter the following error if you try to either test or save the integration.
Either a role with this ARN doesn't exist, or the role's trust policy is not configured to allow ThousandEyes to assume the role. In the latter case, you must re-paste the trust policy into AWS, as outlined in step 3 within AWS Console.
SNS Topic Subscription Errors
ThousandEyes is unable to subscribe to SNS topics:
Once an integration is configured and saved, ThousandEyes will attempt to subscribe to the topics listed in the integration. The integration will remain in the “Pending” state until ThousandEyes successfully subscribes to all of the topics. If one or more of the subscription attempts fail, the integration status will change to "Failed" and the following error message will be displayed in the integration side panel with the list of topics for which the subscription failed.
The common causes of subscription failures include:
A topic with the specified ARN doesn’t exist.
The topic’s access policy is missing permissions to allow ThousandEyes to subscribe to the topic.
To resolve this error, update the access policy of the role as outlined in step 2 of Steps within ThousandEyes.
Flow Logs Processing Errors
When the integration state changes to “Connected”, it indicates that ThousandEyes is able to assume the IAM role and that it has successfully subscribed to all of the topics in the integration.
ThousandEyes is now receiving messages from these topics. This section outlines potential errors related to the retrieval and processing of incoming flow logs messages.
If you do not see flow logs traffic from your monitored accounts, consider checking for the errors listed below.
There is no associated monitored account:
ThousandEyes will only consume and process flow logs from accounts that have an associated inventory monitoring integration. Ensure that you create inventory monitoring integrations for all accounts you wish to monitor using Cloud Insights. See How the Integrations Work Together for more information.
Mandatory log format fields are missing:
Step 2 of Steps within AWS lists the mandatory fields that must be contained within the log format. All but the following three fields are part of the default format:
$(flow-direction)$(tcp-flags)$(traffic-path)
You must choose the custom setting for the log record format to add the additional fields. If any of the default or additional fields are missing, Cloud Insights will not process the logs.
You can verify the current log format by checking your AWS flow logs table for your VPCs.
Note: It is not possible to add or remove fields from an existing flow log configuration. To make changes, you must create a new flow log configuration with the required fields and then remove the old one.
Flow logs aren’t downloading from the user’s S3 buckets:
When ThousandEyes receives a notification about a new flow log, it will attempt to download the file from the S3 bucket using the IAM role specified in the integration. The most common reason for download failures is improper configuration of the role’s permission policy. To resolve this, update the role’s permission policy as outlined in step 2 within AWS Console.
Now that you have completed your Cloud Insights integrations, see Cloud Insights to learn how to view and manage your Cloud Insights data.
Last updated
