# AWS for Cloud Insights Cloud Insights discovers your AWS infrastructure and correlates inventory, configuration changes, and traffic flows with **Network & App Synthetics**. Use it to see how your AWS environment affects real user experience. This guide shows how to set up integrations in the AWS Management Console. For CLI steps, see [Set up Cloud Insights with the AWS CLI](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/aws-for-cloud-insights/aws-for-cloud-insights-using-cli). To learn more about Cloud Insights features, see [Cloud Insights overview](https://docs.thousandeyes.com/product-documentation/cloud-insights). To use Cloud Insights with AWS, create two integrations for each AWS account: 1. **Inventory Monitoring** — Start here. Discovers AWS assets, topology, and configuration changes. Required for Cloud Insights to show AWS data. 2. **Flow Logs Monitoring** — Optional. Adds visibility into traffic by ingesting VPC flow logs and Transit Gateway flow logs. Displays throughput and rejected traffic. For details, see: * [VPC flow logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) * [Transit Gateway flow logs](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-flow-logs.html) For a hands-on walkthrough, see: * [Cloud Insights Inventory Integration Tutorial for AWS](https://demo.thousandeyes.com/player/?demoId=a5ba6861-d7b7-460d-87a9-414a14001c67\&showGuide=true\&showGuidesToolbar=true\&showHotspots=true\&source=app) ## Prerequisites for AWS Integration with Cloud Insights Before you set up the AWS integration, make sure you have: * An active ThousandEyes organization * A ThousandEyes user with **Organization Admin** or **Account Admin** access * An active AWS account * **AWS IAM permissions** to create roles and policies. See [AWS permissions for Cloud Insights](#aws-permissions-for-cloud-insights). * **Flow Log Format**: Your VPC or Transit Gateway flow logs must be configured to output in either text or Parquet format. For additional permissions required by **Flow Logs Monitoring**, see [AWS permissions for Cloud Insights](#aws-permissions-for-cloud-insights). ### AWS permissions for Cloud Insights Integration When you create an **Inventory Monitoring** integration, ThousandEyes generates a permission policy for you. Review this template to see the read-only actions that Cloud Insights requires. For details, see [Creating the Inventory Monitoring Integration](#creating-the-inventory-monitoring-integration). For general IAM reference, see [Actions defined by AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/list_awsidentityandaccessmanagementiam.html). If you also plan to set up **Flow Logs Monitoring**, make sure your AWS account includes the following permissions: * **VPC** — enable and configure VPC flow logs. See [VPC permissions](https://docs.aws.amazon.com/vpc/latest/userguide/security-iam.html). * **Transit Gateway (TGW)** — enable and configure TGW flow logs. See [TGW permissions](https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-authentication-access-control.html). * **S3** — create S3 buckets for flow logs and configure event notifications to SNS. See [S3 permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). * **SNS** — create SNS topics, configure access policies, and allow ThousandEyes to subscribe. See [SNS permissions](https://docs.aws.amazon.com/sns/latest/dg/sns-setting-up.html). ### Cloud Insights Onboarding Constraints for AWS ThousandEyes Cloud Insights currently supports onboarding up to 200 AWS accounts, with up to 17 regions per account (the AWS default). If you need to onboard more accounts or regions, contact the support team to open a ticket. For more information on contacting ThousandEyes support, see [Contacting Support](https://docs.thousandeyes.com/product-documentation/getting-started/getting-support-from-thousandeyes#contacting-support). ### ThousandEyes Permissions for Cloud Insights To set up integrations, use a ThousandEyes account with the *Organization Admin* or *Account Admin* role. For details about roles and permissions, see [Role-based access control explained](https://docs.thousandeyes.com/product-documentation/user-management/rbac/role-based-access-control-explained). ## Overview of the Cloud Insights Integration Architecture ### What Cloud Insights Collects From AWS Cloud Insights works through two types of integrations, each capturing a different view of your AWS environment. To get the full picture, set up both for every account you monitor. **Inventory Monitoring** collects inventory and configuration information from your AWS accounts over time. Cloud Insights uses this data to: * Show your AWS network assets, including types and locations, in **Cloud Insights > Inventory** * Track configuration changes as events, visible in **Inventory** and **Views** * Display your cloud network topology in **Network & App Synthetics > Views** * Enrich flow log data with resource information (when Flow Logs Monitoring is also enabled) For more information, see [Create the AWS Inventory Monitoring Integration](##create-the-aws-inventory-monitoring-integration-for-cloud-insights). **Flow Logs Monitoring** tracks real traffic flows in your AWS network. Cloud Insights uses this data to: * Display inbound and outbound throughput for each entity in **Views** * Highlight rejected traffic in **Cloud Insights > Views** For more information, see [Creating the AWS Flow Logs Monitoring integration](#creating-the-aws-flow-logs-monitoring-integration-for-cloud-insights). {% hint style="info" %} Cloud Insights requires \*\*Inventory Monitoring\*\*. \*\*Flow Logs Monitoring\*\* adds traffic visibility but cannot run on its own. {% endhint %} ### How Cloud Insights Manages AWS Integrations In ThousandEyes, each organization is divided into account groups. Cloud Insights integrations belong to the account group where you create them. Integrations are not shared across groups. Within an account group, you can connect one or more AWS accounts. Each AWS account may span multiple regions. Cloud Insights treats each unique **account + region** pair as an AWS location (for example: `351945360856, us-west-1`). Flow logs are fetched per location. The diagram below shows an example: * **Account Group 1** monitors two AWS accounts, with one account spanning two regions. * **Account Group 2** monitors one AWS account in a single region. * **Account Group 3** monitors one AWS account spanning two regions. ![Example of an organization with three account groups, each monitoring different AWS accounts and regions. Account groups are independent: each one manages its own integrations for the AWS accounts it monitors](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-49e60aa1cc88262eaaf526df1eedc4fa20439a1f%2Fproduct-documentation_integration-guides_custom-built-integrations_aws-for-cloud-insights_3.png?alt=media\&token=deead1d1-0c14-4acf-b240-9f27fe76518d) Each monitored AWS account requires its own integration. * For **Inventory Monitoring**, create one integration per AWS account in an account group. Each integration needs a separate IAM role. * For **Flow Logs Monitoring**, create one integration per AWS account that stores flow logs. ### How Cloud Insights AWS Inventory and Flow Logs Monitoring Work Together **Inventory Monitoring** discovers the AWS resources you want to observe, such as VPCs and Transit Gateways. Once these resources are identified, their corresponding flow logs provide the traffic data needed for analysis. For Cloud Insights to collect this data, the flow logs are published to S3 buckets, either in the same account or in a different one. Event notifications from these buckets are sent to SNS topics, and ThousandEyes subscribes to those topics to know when new log files are available. This is where the **Flow Logs Monitoring** integration takes over. **Flog Logs Monitoring** retrieves the log files and processes the data, using the inventory context provided by **Inventory Monitoring** to surface traffic only for the accounts you’ve set up. This crucial step ensures that traffic from any source not included in your inventory is either filtered out or correctly marked as external, providing a clean and focused view of your network. To learn more about how **Flow Logs Monitoring** works, see [AWS Flow Logs Monitoring](#aws-flow-logs-monitoring-overview). ## Create the AWS Inventory Monitoring Integration for Cloud Insights Setting up the AWS **Inventory Monitoring** integration in ThousandEyes involves three main tasks: 1. [Create an IAM Role for ThousandEyes in AWS](#create-an-iam-role-for-thousandeyes-in-aws) — gives ThousandEyes read-only access to your AWS account. 2. [Look Up the Role ARN in AWS](#look-up-the-role-arn-in-aws) — provides the identifier needed to connect the role with ThousandEyes. 3. [Create a New Integration in ThousandEyes](#create-a-new-integration-in-thousandeyes) — links your AWS account to ThousandEyes using the IAM role. ### Create an IAM role for ThousandEyes in AWS The IAM role gives ThousandEyes read-only access to your AWS account. Setting it up involves two parts: 1. [Create the permission policy in AWS](#create-the-permission-policy-in-aws) 2. [Create the trust policy in AWS](#create-the-trust-policy-in-aws) ThousandEyes generates both policies for you, so you can copy them directly into AWS. {% hint style="info" %} The trust policy for \*\*Inventory Monitoring\*\* looks the same as other AWS integrations, but the permission policy is different. Always use the policy generated for \*IInventory Monitoring\*\* when creating the role. {% endhint %} The steps below use the AWS Management Console. For CLI instructions, see [Set up Inventory Monitoring with the AWS CLI](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/aws-for-cloud-insights/aws-for-cloud-insights-using-cli). #### Create the Permission Policy in AWS Follow these steps to create the permission policy in the AWS Management Console. For the most up-to-date guidance, see [Creating IAM policies (AWS documentation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-json-editor). 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/iam/) and open the IAM console. 2. In the navigation pane, choose **Policies**, then choose **Create policy**. 3. In **Policy editor**, select **JSON**. 4. Paste the permission policy JSON generated by ThousandEyes. * For an example, see [AWS Permission Policy for Console](#aws-permission-policy-for-console). 5. Resolve any validation warnings or errors, then choose **Next** until you reach the **Review and create** page. 6. On the **Review and create** page, enter a **Policy name** and an optional **Description**. 7. Review the permissions granted by the policy. 8. Choose **Create policy** to save your new policy. #### AWS Permission Policy for Console This example shows the permission policy ThousandEyes generates to allow read-only access to your AWS resources. {% hint style="info" %} This version is for reference only and may not be up to date. Always copy the latest version from the ThousandEyes UI. {% endhint %} To get the latest policy text: 1. In ThousandEyes, go to **Manage > Integrations**. 2. Click **+ New Integration**. 3. In the **Add New Integration** panel, select **Amazon Web Services**. 4. Under **IAM Role**, expand **Permission Policy** and copy the text. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowTECloudInsightsApiGatewayReadAccess", "Effect": "Allow", "Action": [ "apigateway:Get" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsCloudFrontReadAccess", "Effect": "Allow", "Action": [ "cloudfront:ListDistributions", "cloudfront:GetDistribution", "cloudfront:GetDistributionConfig" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsDirectConnectReadAccess", "Effect": "Allow", "Action": [ "directconnect:describeDirectConnectGateways", "directconnect:describeVirtualInterfaces", "directconnect:describeDirectConnectGatewayAssociations", "directconnect:describeDirectConnectGatewayAttachments", "directconnect:DescribeConnections", "directconnect:DescribeLags" ], "Resource": "*" }, { "Sid": "AllowCloudInsightsEc2ReadAccess", "Effect": "Allow", "Action": [ "iam:ListAccountAliases", "autoscaling:DescribeAutoScalingGroups", "ec2:DescribeRegions", "ec2:DescribeInstances", "ec2:DescribeNatGateways", "ec2:DescribeTransitGateways", "ec2:DescribeTransitGatewayRouteTables", "ec2:GetTransitGatewayRouteTableAssociations", "ec2:GetTransitGatewayRouteTablePropagations", "ec2:SearchTransitGatewayRoutes", "ec2:DescribeTransitGatewayAttachments", "ec2:DescribeTransitGatewayVpcAttachments", "ec2:DescribeTransitGatewayPeeringAttachments", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:DescribeVpcEndpoints", "ec2:DescribeNetworkInterfaces", "ec2:DescribeAddresses", "ec2:DescribeAvailabilityZones", "ec2:DescribeNetworkAcls", "ec2:DescribeSecurityGroups", "ec2:DescribeRouteTables", "ec2:DescribeVpnGateways", "ec2:DescribeInternetGateways", "ec2:DescribeManagedPrefixLists", "ec2:GetManagedPrefixListEntries", "ec2:DescribeVpcPeeringConnections", "ec2:DescribeVpnConnections", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeRules", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetGroupAttributes", "elasticloadbalancing:DescribeTargetHealth" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsEcsEksReadAccess", "Effect": "Allow", "Action": [ "ecs:ListClusters", "ecs:DescribeClusters", "ecs:DescribeServices", "ecs:DescribeTaskDefinition", "eks:DescribeNodeGroup", "eks:ListNodeGroups", "eks:ListClusters", "eks:DescribeCluster" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsGlobalAcceleratorReadAccess", "Effect": "Allow", "Action": [ "globalaccelerator:ListAccelerators", "globalaccelerator:ListListeners", "globalaccelerator:ListEndpointGroups", "globalaccelerator:ListTagsForResource" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsCloudTrailReadAccess", "Effect": "Allow", "Action": [ "cloudtrail:LookupEvents" ], "Resource": "*" }, { "Sid": "AllowTECloudInsightsS3ReadAccess", "Action": [ "s3:GetBucketPolicy", "s3:GetBucketTagging", "s3:ListAllMyBuckets", "s3express:ListAllMyDirectoryBuckets" ], "Effect": "Allow", "Resource": "*" }, { "Sid": "AllowTECloudInsightsNetworkFirewallReadAccess", "Action": [ "network-firewall:ListFirewalls", "network-firewall:DescribeFirewall", "network-firewall:ListFirewallPolicies", "network-firewall:DescribeFirewallPolicy", "network-firewall:ListRuleGroups", "network-firewall:DescribeRuleGroup" ], "Effect": "Allow", "Resource": "*" } ] } ``` ### Create the Trust Policy in AWS The trust policy allows ThousandEyes to assume the IAM role with the permissions you created in [Create the Permission Policy in AWS](#create-the-permission-policy-in-aws). Follow these steps in the AWS Management Console. For the most up-to-date guidance, see [Creating a role with a custom trust policy (AWS documentation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html). 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/iam/) and open the IAM console. 2. In the navigation pane, choose **Roles**, then choose **Create role**. 3. Select **Custom trust policy** as the role type. 4. In **Custom trust policy**, paste the trust policy JSON generated by ThousandEyes. * For a reference example, see [AWS trust policy for console](#aws-trust-policy-for-console). 5. Resolve any validation warnings or errors, then choose **Next**. 6. Enter a unique **Role name**. * Role names are case-insensitive, must be unique in your account, and cannot be changed after creation. 7. Review the role, then choose **Create role**. #### AWS Trust Policy for Console This example shows the trust policy ThousandEyes generates to allow read-only access to your AWS resources. {% hint style="info" %} This version is for reference only and may not be up to date. Always copy the latest version from the ThousandEyes UI. {% endhint %} To get the latest policy text: 1. In ThousandEyes, go to **Manage > Integrations**. 2. Click **+ New Integration**. 3. In the **Add New Integration** panel, select **Amazon Web Services**. 4. Under **IAM Role**, expand **Trust Policy** and copy the text. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::874690651150:user/thousandeyes-integrations-user" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "75fbb467732cb3fe17ce05a3ef106c1c3ae2de3f" } } } ] } ``` ### Look Up the Role ARN in AWS You’ll need the role’s Amazon Resource Name (ARN) to complete the integration in ThousandEyes. 1. In the [AWS Management Console](https://console.aws.amazon.com/iam/), open the **IAM** service. 2. In the navigation pane, choose **Roles**. 3. Find the role you created, then select it. 4. In the **Summary** box at the top, copy the **ARN** (for example: `arn:aws:iam::123456789098:role/ThousandEyesInventoryRole`). * Use the **copy** icon next to the ARN for accuracy. {% hint style="info" %} Keep the ARN handy. You’ll paste it into ThousandEyes in the \[Create the Integration in ThousandEyes]\(#create-the-integration-in-thousandeyes) step. {% endhint %} ### Create the integration in ThousandEyes The AWS **Inventory Monitoring** integration gives ThousandEyes secure, read-only access to your AWS account. Before you start, make sure you’ve: * Created the IAM role in AWS * Copied the role ARN (see [Look up the role ARN in AWS](#look-up-the-role-arn-in-aws)) To create the integration: 1. In ThousandEyes, go to **Manage > Integrations**. 2. Click **+ New Integration** in the top-right corner. 3. In the **Add New Integration** panel, select **Amazon Web Services**. 4. Enter a unique name for your integration. * Duplicate names are not allowed. 5. From **ThousandEyes Supported Services**, select **Inventory Monitoring**. 6. Paste the IAM role ARN into **Account Resource Name (ARN)**. 7. Click **Test** to validate the trust policy between AWS and ThousandEyes. * **Note**: The **Test** function only validates the trust relationship. It does not check the permission policy. 8. If the test succeeds, click **Save**. * If it fails, see [Troubleshooting AWS Integration for Cloud Insights](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/cloud-insights/aws-for-cloud-insights/troubleshooting-aws-integration-for-cloud-insights). 9. After saving, the integration appears in the list with a status of `Pending`. The status changes to `Connected` once service discovery is complete. See [Checking your integrations are working](#checking-your-integrations-are-working) for details on integration states. ## Creating the AWS Flow Logs Monitoring Integration for Cloud Insights ### AWS Flow Logs Monitoring Overview **Flow Logs Monitoring** works alongside **Inventory Monitoring** to add traffic visibility. Once your AWS accounts are connected through **Inventory Monitoring**, **Flow Logs Monitoring** ingests VPC and Transit Gateway flow logs by following two steps: 1. **Receive notifications** — Your flow logs are stored in an S3 bucket. Event notifications from that bucket are published to an SNS topic in the same region. ThousandEyes subscribes to the topic to learn when new log files are available. 2. **Retrieve log files** — Using an IAM role that you create, ThousandEyes downloads the log files from your S3 bucket for processing. To complete this process, each **Flow Logs Monitoring** integration needs: * An SNS topic for each region you want to monitor (in the same account as the bucket) * An IAM role with read-only access to the S3 buckets containing your flow logs ![VPC flow logs are stored in an S3 bucket, which triggers event notifications to an SNS topic. ThousandEyes subscribes to the topic and then retrieves the flow logs from the S3 bucket.](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-02e7214a8b56fece97fb63ed2964d43025e83b60%2Fproduct-documentation_integration-guides_custom-built-integrations_aws-for-cloud-insights_5.png?alt=media\&token=ddf18c4c-751d-428a-9f23-b347f907c8ce) To create the AWS **Flow Logs Monitoring** integration: 1. [Create an S3 Bucket in AWS](#create-an-s3-bucket-in-aws) 2. [Configure VPCs to publish flow logs in AWS](#configure-vpcs-to-publish-flow-logs-in-aws) 3. [Create an SNS topic in AWS](#create-an-sns-topic-in-aws) 4. [Create a ThousandEyes IAM Role for S3 bucket access in AWS](#create-a-thousandeyes-iam-role-for-s3-bucket-access-in-aws) 5. [Create the Flow Logs Monitoring Integration in ThousandEyes](#create-the-flow-logs-monitoring-integration-in-thousandeyes) {% hint style="info" %} You may incur some marginal costs to your AWS account to create, store, and transfer flow logs to ThousandEyes. {% endhint %} #### Create an S3 Bucket in AWS You’ll need at least one S3 bucket in each region where you want to collect flow logs. There are two common ways to organize buckets for ThousandEyes flow logs integration: 1. **Buckets in each monitored account** — Each account publishes flow logs to a bucket in the same region. * Minimizes cross-region data transfer fees * Requires one **Flow Logs Monitoring** integration per account ![Each monitored AWS account publishes flow logs to an S3 bucket in the same account and region. ThousandEyes needs one Flow Logs Monitoring integration per account.](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-ae78218d1129774233256517f715721b9ad96f14%2Fproduct-documentation_integration-guides_custom-built-integrations_aws-for-cloud-insights_1.png?alt=media) 2. **Centralized buckets** — All monitored accounts publish flow logs to a separate account that hosts buckets in each region. * Reduces the number of **Flow Logs Monitoring** integrations * Requires a central account with permissions to receive logs from other accounts ![All monitored AWS accounts publish flow logs to S3 buckets in a single, central account. ThousandEyes only needs one Flow Logs Monitoring integration for that account.](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-c67641fc43a353afbd8da719ae502029c696139e%2Fproduct-documentation_integration-guides_custom-built-integrations_aws-for-cloud-insights_2.png?alt=media\&token=94120957-97dd-4ebf-a6dc-9d786c87245d) {% hint style="info" %} Choose centralized buckets if you have many accounts and want fewer integrations; choose per-account buckets if you want strict account isolation and localized data paths. {% endhint %} Follow these steps to create a bucket in the AWS Management Console. For the most up-to-date instructions, see [Creating a bucket (AWS documentation)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/s3/) and open the **Amazon S3** console. 2. Choose **Create bucket**. 3. On the **Create bucket** page, either: * Accept the default settings, or * Customize the settings as described in the [AWS bucket creation guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). 4. Choose **Create bucket** to finish. #### Configure VPCs to Publish Flow Logs in AWS Use the [VPC console](https://console.aws.amazon.com/vpc/) to send VPC flow logs to your S3 bucket. 1. Open the **Amazon VPC** console. 2. For each VPC you want to monitor in this region, configure the following **Flow log settings**: * **Filter** → `All` (to include both accepted and rejected traffic) * **Maximum aggregation interval** → `1 minute` or `10 minutes`, depending on how detailed you want your data * *1 minute produces more detailed data but creates more records.* * **Destination** → `Send to an Amazon S3 bucket` * **S3 bucket ARN** → the ARN of the bucket you created earlier * **Log file format** → `Text` or `Parquet` * Note: Parquet is a columnar format that can be more efficient and cost-effective for processing large volumes of flow logs. * **Log record format** → `Custom` Include these mandatory fields: ``` ``` You can include any other standard attributes as needed. {% hint style="info" %} To minimize cross-region data transfer costs, configure all VPCs in the same region to publish flow logs to the same regional S3 bucket. {% endhint %} For instructions on publishing flow logs from Transit Gateways, see [Configure Transit Gateways to publish flow logs in AWS](#configure-transit-gateways-to-publish-flow-logs-in-aws). For more background on flow logs, see [AWS VPC Flow Logs documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html). #### Configure Transit Gateways to Publish Flow Logs in AWS You can also publish [Transit Gateway (TGW) flow logs](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-s3.html) to the same S3 bucket you use for VPC flow logs. {% hint style="info" %} Transit Gateway flow logs are optional. Configure them if your AWS network uses Transit Gateways to route traffic between VPCs or accounts. Doing so provides a more complete view of network performance and traffic patterns. {% endhint %} 1. In the [VPC console](https://console.aws.amazon.com/vpc/), open **Transit Gateways**. 2. Under **Flow log settings**, configure the following: * **Log file format** → `Text` or `Parquet` * Note: Parquet is a columnar format that can be more efficient and cost-effective for processing large volumes of flow logs. * **Destination** → `Send to an Amazon S3 bucket` * **S3 bucket ARN** → the ARN of the S3 bucket you created earlier * **Log record format** → include at least all fields from the **Default** format {% hint style="info" %} Using the same regional S3 bucket for both VPC and TGW flow logs helps keep your configuration simple and avoids cross-region data transfer costs. {% endhint %} For more information, see [AWS Transit Gateway Flow Logs documentation](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-s3.html). #### Create an SNS topic in AWS You’ll need an SNS topic in each region where you’re collecting flow logs. ThousandEyes uses these topics to receive notifications when new flow log files are available. {% hint style="info" %} Keep the SNS topic ARNs handy — you’ll need them to complete the role configuration in ThousandEyes. {% endhint %} 1. In the [SNS console](https://console.aws.amazon.com/sns/), create a new topic. * Choose **Standard** for **Type**. * For step-by-step instructions, see [Creating an Amazon SNS topic (AWS documentation)](https://docs.aws.amazon.com/sns/latest/dg/sns-create-topic.html). 2. In the [S3 console](https://console.aws.amazon.com/s3/), configure **event notifications** for each flow log bucket in the region to send to this SNS topic. * Follow the steps in [Enabling Amazon SNS notifications using the Amazon S3 console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html). * Under **Event types**, choose **All object create events (s3:ObjectCreated:\*)**. * Under **Destination**, select **SNS topic**, then enter or select the topic ARN you just created. 3. Update the SNS topic access policy to allow the S3 bucket to publish notifications to the topic. For more details, see [Grant Destinations Permissions to S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html). * You’ll also update this policy later in [Update the SNS Topic Access Policy](#update-the-sns-topic-access-policy). {% hint style="info" %} Create one SNS topic per region. This keeps notifications organized and helps avoid cross-region latency or permission issues. {% endhint %} {% hint style="info" %} Cloud Insights also supports S3 event notifications through \[AWS EventBridge]\(), which can provide additional options for managing and routing events. {% endhint %} #### Create the ThousandEyes IAM role for S3 bucket access in AWS This IAM role gives ThousandEyes read access to your S3 buckets that store flow logs. Use the **Trust** and **Permissions** policies generated in the ThousandEyes UI to configure the role. 1. In AWS, create a new IAM role. * For AWS instructions, see [Creating IAM roles (AWS documentation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html). 2. In the **Permissions policy**, replace the placeholder `` with the ARNs of your flow log buckets. * Include both of the following ARNs for each bucket: ``` arn:aws:s3::: arn:aws:s3:::/* ``` 3. In the **Trust policy**, paste the version generated by ThousandEyes. This allows ThousandEyes to assume the role securely. {% hint style="info" %} You’ll also need the SNS topic information you created earlier. {% endhint %} {% hint style="info" %} Keep the \*\*SNS topic ARNs\*\* from \[Create an SNS topic in AWS]\(#create-an-sns-topic-in-aws) handy — you’ll need them to complete the role configuration in ThousandEyes. {% endhint %} #### Update the SNS Topic Access Policy After creating the IAM role, update each SNS topic’s access policy to allow ThousandEyes to subscribe and receive flow log notifications. For the **Access Policy** generated by ThousandEyes, see the section [AWS SNS Access Policy for Console](#aws-sns-access-policy-for-console). To create the SNS topic in AWS: 1. In the [SNS console](https://console.aws.amazon.com/sns/), locate the topics you created in [Create an SNS topic in AWS](#create-an-sns-topic-in-aws). 2. Choose a topic, then select **Edit** > **Access policy** > **Advanced**. 3. Paste the **Access Policy** generated by ThousandEyes into the JSON editor. For more information, see the section [AWS SNS Access Policy for Console](#aws-sns-access-policy-for-console). 4. In the policy JSON, replace the following placeholders: * ``: the ARN of your SNS topic * ``: the AWS account ID for your flow log bucket * ``: the ARN of your flow log bucket This policy allows S3 event notifications to be sent to your SNS topic and grants ThousandEyes permission to subscribe to it. {% hint style="info" %} Configure one SNS topic per region for better scalability and simpler troubleshooting. {% endhint %} **AWS SNS Access Policy for Console** This example shows the access policy ThousandEyes generates to allow read-only access to your AWS SNS topics. {% hint style="info" %} This version is for reference only and may not be up to date. Always copy the latest version from the ThousandEyes UI. {% endhint %} To get the latest policy text: 1. In ThousandEyes, go to **Manage > Integrations**. 2. Click **+ New Integration**. 3. In the **Add New Integration** panel, select **Amazon Web Services**. 4. Under **SNS Topics**, expand **Access Policy** and copy the text. ```json { "Version": "2008-10-17", "Statement": [ { "Sid": "AllowTECloudInsightsSubscribe", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::874690651150:user/thousandeyes-integrations-user" }, "Action": "SNS:Subscribe", "Resource": "" }, { "Sid": "AllowS3EventNotifications", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SNS:Publish", "Resource": "", "Condition": { "StringEquals": { "aws:SourceAccount": "" }, "ArnLike": { "aws:SourceArn": "" } } } ] } ``` #### Create the Flow Logs Monitoring Integration in ThousandEyes Before you start, make sure you’ve: * Created the IAM role in AWS * Copied the role ARN (see [Look up the Role ARN in AWS](#look-up-the-role-arn-in-aws)) To create the integration: 1. In ThousandEyes, go to **Manage > Integrations**. 2. Click **+ New Integration** in the top-right corner. 3. In the **Add New Integration** panel, select **Amazon Web Services**. 4. Enter a unique name for your integration. * Duplicate names are not allowed. 5. From **ThousandEyes Supported Services**, select **Flow Logs Monitoring**. 6. Paste the IAM role ARN into **Account Resource Name (ARN)**. 7. Under **SNS Topics**, add the SNS topic ARNs. * Click **+ Add SNS Topic ARN** to add more. Use the **minus** icon to remove one. 8. Click **Test** to validate the trust policy between AWS and ThousandEyes. * **Note**: The **Test** function only validates the trust relationship. It does not check the permission policy or SNS subscriptions. 9. If the test succeeds, click **Save**. * If it fails, see [Troubleshooting AWS Integration for Cloud Insights](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/cloud-insights/aws-for-cloud-insights/troubleshooting-aws-integration-for-cloud-insights). 10. After saving, the integration appears with a status of `Pending`. Once ThousandEyes validates subscriptions and begins processing logs, the status changes to `Connected` (or `Partially Connected` if some topics or files fail). ## Verify Your AWS Cloud Insights Integrations After you save an integration, ThousandEyes begins monitoring the AWS resources defined in its policy. * To check the status, go to **Manage > Integrations**. * To review detailed logs, go to **Cloud Insights > Settings > Integration Logs**. When you first save, the integration status shows as `Pending`. This state usually updates within 5–10 minutes (inventory and flow logs are polled every five minutes). Refresh the page to see the latest status. * **Connected** — Monitoring is working. * **Failed** — An issue prevents monitoring. Click the integration to view error details. In most cases, adjust permissions and click **Save** to retry. * **Pending** — Temporary state until ThousandEyes validates the connection. Flow log errors appear once flow log files are received. Any issues are shown on the integration screen and in **Integration Logs**. For more information about status meanings and next steps, see the separate troubleshooting guide: [Troubleshooting AWS for Cloud Insights](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/cloud-insights/aws-for-cloud-insights/troubleshooting-aws-integration-for-cloud-insights). ## Additional resources * View and manage Cloud Insights data - [Cloud Insights](https://docs.thousandeyes.com/product-documentation/cloud-insights/views) * Manage integrations - [Integrations](https://docs.thousandeyes.com/product-documentation/integration-guides) * Review integration logs in Cloud Insights settings - [Cloud Insights Settings: Integration Logs](https://docs.thousandeyes.com/product-documentation/cloud-insights/settings#integration-logs)