AWS for Cloud Insights Using CLI

The Amazon Web Services (AWS) for Cloud Insights integrations can be created and managed via two methods: the AWS Console and the AWS Command Line Interface (CLI). For steps within the AWS Console, and to understand more of the context around these integrations, see AWS for Cloud Insights. For steps to create the integrations by the CLI only, see below.

Overview of CLI Steps

For inventory monitoring:

• Create IAM (Identity and Access Management) policies and roles.

For flow logs monitoring:

• Create IAM policies and roles.

• Create an S3 (Simple Storage Service) bucket from which to send traffic flows.

• Configure VPC (Virtual Private Cloud) flow logs.

• Create an SNS (Simple Notification Service) topic and apply a policy.

• Configure S3 bucket notifications to send events to the SNS topic.

Prerequisites

Before running the commands in this guide, ensure you meet the following prerequisites.

AWS CLI Installation

1. Have the AWS CLI installed on your system (see how to download and install it).

2. Your AWS CLI must be configured with appropriate IAM credentials.

   • You can configure the credentials using the aws configure command:

     aws configure

   • You will be prompted to enter your AWS Access Key ID, Secret Access Key, default region, and output format. The AWS keys must correspond to an IAM user that has the necessary permissions to perform the actions in this guide. For inventory monitoring permissions, see step 6 of IAM Configuration for Inventory Monitoring. For flow logs monitoring, see step 3 of IAM Configuration for Flow Logs Monitoring.

Environment Variables

Ensure the following environment variables are set for the flow logs monitoring integration (see Configuring environment variables for the AWS CLI for more information about setting environment variables) for more information about setting environment variables.

• EXTERNAL_ID – A unique ID for external integrations which you can retrieve from the ThousandEyes-generated Trust Policy (see step 5 of IAM Configuration for Flow Logs Monitoring).

• ACCOUNT_ID – Can be obtained from the AWS console or from STS (Security Token Service): type aws sts get-caller-identity --query "Account" --output text into the CLI.

• S3_BUCKET_NAME – The name of the S3 bucket to be created (see bucket naming rules).

• AWS_REGION – The AWS region where the resources will be created.

• VPC_ID – The ID of the VPC from which flow logs will be created.

• SNS_TOPIC_NAME – The name of the SNS topic to be created.

CLI Steps for Inventory Monitoring

The following steps create:

• An IAM role with a trust policy that allows ThousandEyes to assume the role to execute read actions and build an inventory.

• An IAM permission policy that allows ThousandEyes read access to your AWS resources.

Note: You will need to be logged into your ThousandEyes account to retrieve certain information while you enter commands into the CLI.

IAM Configuration for Inventory Monitoring

1. In the ThousandEyes platform, go to Manage > Integrations.

2. Click + New Integration in the top right.

3. In the Add New Integration side panel that opens, select Amazon Web Services.

   • The resulting < Add AWS Integration screen defaults to showing the fields required for the “Test Recommendations” service.

4. Select “Inventory Monitoring” from the ThousandEyes Supported Services dropdown.

5. Give your integration a unique name. Duplicate names are not permitted.

6. Next to IAM Role, select generate for Command Line.

7. Open and copy the script within the Trust Policy dropdown and paste the script into your terminal.

8. Type in your preferred --role-name and --description, and hit Enter.

9. Copy the role ARN from the response for use in step 12.

   {
        "Role": {
             "Path": "/",
             "RoleName": "TECloudInsightsRole",
             "RoleId": "AROA123456789ABCDFG",
             "Arn": "arn:aws:iam::123456789456:role/TECloudInsightsRole",
             ...

10. Back in the ThousandEyes platform, open and copy the script in the Permission Policy dropdown and paste the script into your terminal.

11. Type in the same --role-name as in your trust policy and your preferred --policy-name, and hit Enter.

12. In the ThousandEyes platform, paste the IAM role ARN from step 9 into the Account Resource Name (ARN) field.

    

    Account Resource Name (ARN) field on inventory monitoring integration screen
13. Click Test.

    • Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.

14. If testing was successful, click Save.

CLI Steps for Flow Logs Monitoring

The following steps create:

• An S3 bucket in your specified region, if not already done so.

• An IAM permission policy to allow ThousandEyes read access to your AWS S3 resources.

• An IAM role with a trust policy that allows ThousandEyes to assume the role to execute read actions and build an inventory.

• An IAM policy to attach traffic flow permissions to the IAM role.

• VPC flow logs for the specified VPC ID, with logs stored in a central S3 bucket, if not already done so.

• An SNS topic, and a policy applied to it, if not already done so.

• Configuration of the S3 bucket to send notifications to the SNS topic when objects are created.

Creating the S3 Bucket

1. If not already created, in your terminal type the following command to create an S3 bucket in your specified AWS region. You will need the S3_BUCKET_NAME environment variable, and potentially the AWS_REGION environment variable, which you created in Environment Variables.

   • Use the LocationConstraint parameter only if your preferred region is not the default us-east-1 region. The command errors if you use the LocationConstraint parameter and your region is the default; the command also errors if the LocationConstraint parameter is missing and your region is not the default.

     aws s3api create-bucket \
       --bucket $S3_BUCKET_NAME \
       --create-bucket-configuration LocationConstraint=$AWS_REGION

IAM Configuration for Flow Logs Monitoring

1. Follow steps 1-6 of IAM Configuration for Inventory Monitoring, substituting "Flow Logs Monitoring" for "Inventory Monitoring" in step 4.

2. Copy the text in the Permission Policy dropdown and paste it into a text editor.

3. In the editor, replace all bucket names in the policy, then save it as "te_cloud_insights_allow-s3-bucket_flow_logs.json". For example:

   "Resource": [
     "arn:s3:::<BUCKET-NAME>",
     "arn:s3:::<BUCKET-NAME>/*"  

4. Back in the ThousandEyes platform, open the Trust Policy dropdown. Copy the trust policy and save it as "iam_role_trust_policy.json".

5. In your terminal, type the following command to create the IAM role with the trust policy attached.

   aws iam create-role \
     --role-name TECloudInsightsFlowLogsRole \
     --assume-role-policy-document file://iam_role_trust_policy.json 

   • Copy or make a note of the role ARN from within the output for use within step 9.

6. Create the S3 read access policy for flow logs, using the permission policy saved in step 4.

   aws iam create-policy \
     --policy-name TECloudInsightsReadAccessToS3 \
     --policy-document file://te_cloud_insights_allow-s3-bucket_flow_logs.json

7. Attach the S3 read access policy to the IAM role created in step 6.

   aws iam attach-role-policy \
     --role-name TECloudInsightsFlowLogsRole \
     --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/TECloudInsightsReadAccessToS3

8. Back in the ThousandEyes platform, paste or type the role ARN you copied/noted in step 6 into the Account Resource Name (ARN) field.

VPC Flow Log Configuration

1. If not already created, in your terminal type the following command to create VPC flow logs for the specified VPC ID (see environment variables). The logs are stored in an S3 bucket, and various log parameters are specified, including the log format and aggregation interval. See create-flow-logs for more information.

     RESPONSE=$(aws ec2 create-flow-logs \
       --resource-type VPC \
       --region $AWS_REGION \
       --resource-ids $VPC_ID \
       --traffic-type ALL \
       --log-destination-type s3 \
       --log-destination arn:aws:s3:::$S3_BUCKET_NAME \
       --log-format '${account-id} ${srcaddr} ${srcport} ${dstaddr} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${flow-direction} ${traffic-path} ${region}' \
       --max-aggregation-interval 600 \
       --destination-options PerHourPartition=true \
       --tag-specifications 'ResourceType=vpc-flow-log,Tags=[{Key=Name,Value=CloudInsightsFlow}]')

Creating SNS Topics

1. If not already created, type the following command to create an SNS topic. You will need the SNS_TOPIC_NAME and AWS_REGION environment variables you created in Environment Variables. The command incorporates the creation of the SNS_TOPIC_ARN variable, to be used in steps 12, 14, and 15.

   SNS_TOPIC_ARN=$(aws sns create-topic --name $SNS_TOPIC_NAME --region $AWS_REGION --output text --query 'TopicArn')

   • Copy or make a note of the SNS topic ARN for use within step 17.

2. Copy the following SNS topic policy and save as "sns_topic_policy.json".

       {
         "Sid": "AllowS3Publish",
         "Effect": "Allow",
         "Principal": {
           "Service": "s3.amazonaws.com"
         },
         "Action": "SNS:Publish",
         "Resource": "SNS_TOPIC_ARN",
         "Condition": {
           "StringEquals": {
             "aws:SourceAccount": "ACCOUNT_ID"
           },
           "ArnLike": {
             "aws:SourceArn": "arn:aws:s3:::S3_BUCKET_NAME"
           }
         }
       },
       {
         "Sid": "AllowTECloudInsightsSubscribe",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::381492131706:user/thousandeyes-integrations-user"
         },
         "Action": "SNS:Subscribe",
         "Resource": "SNS_TOPIC_ARN"
       }
     ]
   }

3. Create the SNS_TOPIC_POLICY variable, to be used in the next step, and replace the placeholders in the SNS topic policy with the relevant variables.

   SNS_TOPIC_POLICY=$(cat sns_topic_policy.json | sed "s/SNS_TOPIC_ARN/$SNS_TOPIC_ARN/" | sed "s/ACCOUNT_ID/$ACCOUNT_ID/" | sed "s/S3_BUCKET_NAME/$S3_BUCKET_NAME/")

4. Apply the SNS topic policy to the SNS topic. You will need the SNS_TOPIC_POLICY variable you just created and the SNS_TOPIC_ARN variable you created in step 11.

   aws sns set-topic-attributes \
     --topic-arn $SNS_TOPIC_ARN \
     --attribute-name Policy \
     --attribute-value "$SNS_TOPIC_POLICY"

S3 Bucket Notification Configuration

1. Create the S3_NOTIFICATION_CONFIGURATION variable, using the SNS_TOPIC_ARN variable you created in step 11. You will use this in the next step.

   S3_NOTIFICATION_CONFIGURATION=$(cat <<EOF
   {
     "TopicConfigurations": [
       {
         "Id": "S3ObjectCreatedNotification",
         "TopicArn": "$SNS_TOPIC_ARN",
         "Events": [
           "s3:ObjectCreated:*"
         ]
       }
     ]
   }
   EOF
   )

2. Apply the S3 bucket notification configuration to your S3 bucket. You will need the S3_BUCKET_NAME environment variable you created in Environment Variables and the S3_NOTIFICATION_CONFIGURATION variable you just created.

     aws s3api put-bucket-notification-configuration \
       --bucket $S3_BUCKET_NAME \
       --notification-configuration "$S3_NOTIFICATION_CONFIGURATION"

Completing the Integration in ThousandEyes

1. Back in the ThousandEyes platform, paste the SNS Topic ARN from step 11 into the Simple Notification Service (SNS) Topics ARNs field.

2. Click Test.

   • Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.

3. If testing was successful, click Save.

Verifying the Configuration

In your various AWS consoles (VPC, S3, SNS), you can look to verify that:

• Your flow log record is being recorded in your VPC (see Troubleshoot VPC Flow Logs).

• Your relevant S3 bucket is receiving the records (see View flow log records with Amazon S3)

• Your S3 bucket is sending object-created events to the SNS topic (see Enabling and configuring event notifications using the Amazon S3 console).

• A ThousandEyes user is subscribed to the SNS topic (see Creating a subscription to an Amazon SNS topic).

Using CloudTrail to Log S3 Events

If you use AWS CloudTrail, you can further track your events with the following commands using AWS Athena (see Query AWS CloudTrail logs). Type or paste the following query into your terminal.

SELECT * FROM "default"."cloudtrail_logs_aws_cloudtrail_logs_986837508024_494b5bb3" WHERE eventSource IN ('sts.amazonaws.com', 's3.amazonaws.com', 'sns.amazonaws.com') AND errorCode = 'AccessDenied' AND eventname = 'GetObject' limit 100;