Triage and Remediation
- Remediation
Remediation
Using Console
Using Console
Below are step‑by‑step instructions (AWS Console only) to create a CloudWatch alarm for route table changes using CloudTrail logs.
Prerequisites
- CloudTrail must be enabled and logging management events (at least Write events) for EC2.
- CloudTrail must be delivering logs to a CloudWatch Logs log group.
- If not configured, in the CloudTrail console → Trails → your trail → Edit → CloudWatch Logs and attach/create a log group and an IAM role.
Step 1: Identify/Create the CloudTrail Log Group
- Open CloudTrail console.
- In the left menu, select Trails.
- Click your active trail.
- Scroll to CloudWatch Logs section:
- Note the CloudWatch Logs log group name.
- If none is configured, enable it:
- Click Edit.
- Under CloudWatch Logs, choose or create a log group (e.g.,
/aws/cloudtrail/management). - Choose/create an IAM role as prompted.
- Save.
Step 2: Create a Metric Filter for Route Table Changes
- Open CloudWatch console.
- In the left menu, select Logs → Log groups.
-
Click the CloudTrail log group you identified (e.g.,
/aws/cloudtrail/management). - Go to the Metric filters tab.
- Click Create metric filter.
-
In Filter pattern, use a pattern that matches route-table–related API calls, for example:
- Click Next.
-
For Filter name, enter something like:
RouteTableChangesFilter -
Under Metric details:
- Metric namespace: e.g.,
Security/NetworkChanges - Metric name: e.g.,
RouteTableChanges - Metric value:
1 - Default value: leave blank or
0.
- Metric namespace: e.g.,
- Click Next, then Create metric filter.
Step 3: Create a CloudWatch Alarm on That Metric
- After creating the filter, you’ll see it listed.
Select the filter and click Create alarm (or go to CloudWatch → Alarms → All alarms → Create alarm and select the metric you just created:
Security/NetworkChanges → RouteTableChanges). - In the Specify metric and conditions step:
- Statistic:
Sum - Period: e.g.,
5 minutes - Threshold type:
Static - Condition:
- Greater than
- Threshold value:
0
- This means: alarm if at least one route table change occurs in 5 minutes.
- Statistic:
- Click Next.
Step 4: Configure Alarm Notifications
- In Configure actions:
- Alarm state trigger:
In alarm - Under Notification, choose an SNS topic to send alerts to:
- Select an existing topic (e.g.,
security-alerts), or - Click Create new topic, provide:
- Topic name: e.g.,
route-table-changes-alerts - Email endpoint(s) (security, ops, etc.).
- Topic name: e.g.,
- Select an existing topic (e.g.,
- Confirm email subscription(s) if prompted.
- Alarm state trigger:
- (Optional) Add additional actions (e.g., OpsCenter, EC2 action – normally not needed here).
- Click Next.
Step 5: Name and Create the Alarm
- Provide:
- Alarm name:
RouteTableChangesAlarm - Alarm description: e.g.,
Alerts on any changes to VPC route tables using CloudTrail events.
- Alarm name:
- Review all settings.
- Click Create alarm.
Step 6: Test the Alarm (Optional but Recommended)
- Make a test route table change (e.g., add a non-impactful route in a test VPC and remove it).
- Wait for the metric to update (typically a few minutes).
- Confirm:
- Alarm transitions to ALARM state.
- Notification (email/SNS) is received.
Using CLI
Using CLI
Below is a minimal, end‑to‑end way to enable an alarm for Route Table changes in AWS using the AWS CLI.Assumptions:
This will emit a metric
Once done:
- You already have a CloudTrail trail sending events to a CloudWatch Logs log group (if not, steps 1–2 cover it).
- Replace all placeholder values (e.g.,
MY_TRAIL,my-log-group,my-route-table-changes-metric,my-route-table-changes-alarm, etc.) with your own.
1. (If needed) Configure CloudTrail to send logs to CloudWatch Logs
2. Create a CloudWatch Logs Metric Filter for Route Table changes
The filter will match API calls that modify Route Tables:CreateRouteTableDeleteRouteTableAssociateRouteTableDisassociateRouteTableReplaceRouteTableAssociationReplaceRouteTableRouteCreateRouteDeleteRouteReplaceRoute
Security/my-route-table-changes-metric with value 1 for every matched event.3. Create an SNS topic for notifications (if you don’t have one)
4. Create a CloudWatch Alarm on the metric
Alarm if at least 1 route table change occurs in a 5-minute period:Once done:
- Any change to Route Tables (via the listed API calls) will be logged by CloudTrail into the log group.
- The metric filter will emit a metric per event.
- The CloudWatch alarm will trigger and send a notification via SNS whenever at least one change is detected in the evaluation period.
Using Python
Using Python
Below is one straightforward way to remediate this in AWS using Python/boto3:Goal:
Notes:
- Detect route table changes via CloudTrail
- Send them to CloudWatch Logs
- Use a metric filter and CloudWatch Alarm to trigger notifications (e.g., via SNS)
- You have AWS credentials configured.
- You have
boto3installed. - You have (or will create) an SNS topic and subscription for notifications.
1. Enable/Verify CloudTrail With CloudWatch Logs
CloudTrail must log management events and send them to a CloudWatch Logs log group.- You must have an IAM role (
CloudTrail_CloudWatchLogs_Role) with the proper trust and permissions for CloudTrail → CloudWatch Logs. - Replace
YOUR_ACCOUNT_ID, bucket name, role ARN as needed.
2. Create a Metric Filter for Route Table Changes
Filter CloudTrail events for route-table–related eventNames on EC2.Common relevant events:CreateRoute,DeleteRoute,ReplaceRouteCreateRouteTable,DeleteRouteTableAssociateRouteTable,DisassociateRouteTable,ReplaceRouteTableAssociation
3. Create an SNS Topic (If Needed)
4. Create a CloudWatch Alarm on the Metric
Alarm when at least 1 route table change occurs in a 5-minute period.5. Validate
- Make a test change to a route table.
- Wait a few minutes.
- Verify:
- CloudTrail shows the event.
- Metric in CloudWatch (
SecurityMonitoring/RouteTableChangeCount) increments. - Alarm transitions to ALARM and sends an SNS notification.
Using Terraform
Using Terraform
RouteTableChangesFilter and alarm RouteTableChangesAlarm; if they are currently unmanaged by Terraform, importing or replacing them may overwrite existing settings. After adding this, terraform plan should show one aws_cloudwatch_log_metric_filter and one aws_cloudwatch_metric_alarm (and optionally aws_sns_topic) to be created or updated with the exact pattern, metric, and threshold described above.
