Ec2 instance retirement remediation

​

1. Find EC2 instances with scheduled events

1. Sign in to the AWS Management Console.
2. Go to EC2.
3. In the left menu, select Events under Instances (or Scheduled events, depending on console version).
   • Or go to Instances, then use the filter:
     • Status → Instance status → check With scheduled events.

​

2. Understand the event type and choose action

1. Instance reboot
2. System reboot
3. Instance stop (or stop/start)
4. Retirement (instance or underlying host)
5. Maintenance (e.g., hardware, network, or instance store impact)

​

3. Remediate by event type (via Console)

​

A. Instance/System Reboot

1. From EC2 → Instances, select the affected instance.
2. Choose Instance state → Reboot instance.
3. Confirm the reboot.
4. Verify:
   • After it comes back, check application health and logs.
   • This typically clears the scheduled reboot event.

• Plan a maintenance window before AWS’s scheduled time.
• Then do the manual reboot as above.

​

B. Instance Stop / Stop-Start

1. From EC2 → Instances, select the instance.
2. Ensure you can tolerate downtime and the root volume is EBS (not instance store).
3. Choose Instance state → Stop instance.
4. Wait for state to become stopped.
5. If event calls for stop/start, then:
   • Choose Instance state → Start instance.
6. Verify application and that the scheduled event is cleared.

• If your instance uses an ephemeral (instance store) root volume, data is lost on stop. Back up before stopping (AMI, snapshot, data copy).

​

C. Instance Retirement

1. Create an AMI (if using EBS root):
   • Select the instance → Actions → Image and templates → Create image.
   • Name the AMI and create it.
2. Launch a new instance from that AMI:
   • Go to AMIs, select your new AMI → Launch instance.
   • Use a similar instance type, same VPC, subnet, security groups, IAM role, etc.
3. Update dependencies:
   • Update DNS, load balancer target groups, or other components to point to the new instance.
4. Once traffic and data are fully migrated:
   • Stop/terminate the old instance (according to your decommissioning policy).

• Change to a different instance type or AZ at this time to avoid similar hardware issues.

​

D. Scheduled Maintenance That Requires No Action

• Confirm in the event description that no instance stop/reboot is needed.
• Optionally plan a maintenance window to validate application behavior during/after the window.

​

4. (Optional) Request reschedule of events

1. In EC2 → Events:
   • Select the instance and scheduled event.
2. If available, choose Request reschedule or similar option.
3. Pick a new time window within the allowed range and submit.

• You must complete the required action (reboot/stop/replace) before the AWS deadline.

​

5. Verify remediation

1. Go back to EC2 → Events.
2. Confirm the event is cleared or marked as completed.
3. Validate:
   • Instance Status checks are 2/2 checks passed.
   • Application/services are operational.

​

1. List EC2 instances with scheduled events

aws ec2 describe-instance-status \
  --include-all-instances \
  --query 'InstanceStatuses[?Events].{InstanceId:InstanceId,Events:Events}' \
  --output table

• instance-stop
• instance-reboot
• system-reboot
• system-maintenance
• instance-retirement
• instance-retirement-scheduled

​

2. Prioritize by event time

aws ec2 describe-instance-status \
  --include-all-instances \
  --query 'InstanceStatuses[?Events].{InstanceId:InstanceId,Events:Events[*].{Code:Code,NotBefore:NotBefore}}' \
  --output json

​

3. Decide remediation action per event type

​

A. For instance-stop / instance-retirement

1. If the instance is in an Auto Scaling Group (ASG) or behind a load balancer, ensure draining / replacement is handled.
2. Stop and start the instance to move it to new hardware:

INSTANCE_ID=i-0123456789abcdef0

aws ec2 stop-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-stopped --instance-ids "$INSTANCE_ID"

aws ec2 start-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

3. If you must preserve the instance exactly but hardware migration isn’t sufficient, create a replacement instance:
   # Create AMI
   aws ec2 create-image \
     --instance-id "$INSTANCE_ID" \
     --name "replacement-$(date +%Y%m%d-%H%M%S)" \
     --no-reboot
   
   Then launch a new instance from that AMI and update DNS/ELB/Target Groups/ASGs as appropriate.

​

B. For instance-reboot

INSTANCE_ID=i-0123456789abcdef0

aws ec2 reboot-instances --instance-ids "$INSTANCE_ID"

​

C. For system-reboot / system-maintenance

INSTANCE_ID=i-0123456789abcdef0

aws ec2 stop-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-stopped --instance-ids "$INSTANCE_ID"

aws ec2 start-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

​

4. Confirm the event is cleared

aws ec2 describe-instance-status \
  --instance-ids "$INSTANCE_ID" \
  --include-all-instances \
  --query 'InstanceStatuses[0].Events' \
  --output json

​

5. (Optional) Script to remediate all “stop/retirement” events

# Get all instance IDs with instance-stop/instance-retirement events
INSTANCES=$(aws ec2 describe-instance-status \
  --include-all-instances \
  --query "InstanceStatuses[?Events[?Code=='instance-stop'||Code=='instance-retirement']].InstanceId" \
  --output text)

for ID in $INSTANCES; do
  echo "Remediating $ID..."
  aws ec2 stop-instances --instance-ids "$ID"
  aws ec2 wait instance-stopped --instance-ids "$ID"
  aws ec2 start-instances --instance-ids "$ID"
  aws ec2 wait instance-running --instance-ids "$ID"
done

​

1. Understand the “misconfiguration”

• Detecting upcoming scheduled events.
• Taking controlled action yourself before AWS does (e.g., stop/start, replace instance, move workload).

​

2. Prerequisites

1. Install boto3:
   pip install boto3
   
2. Configure AWS credentials/region (e.g., using aws configure or environment variables).
3. IAM role/user must have at least:
   • ec2:DescribeInstanceStatus
   • ec2:StopInstances
   • ec2:StartInstances
   • ec2:RebootInstances
   • ec2:TerminateInstances (only if you choose to terminate/replace)
   • ec2:DescribeInstances

​

3. Detect scheduled events via Python

import boto3

ec2 = boto3.client('ec2')

def get_instances_with_scheduled_events():
    instances_with_events = []

    paginator = ec2.get_paginator('describe_instance_status')
    page_iterator = paginator.paginate(IncludeAllInstances=True)

    for page in page_iterator:
        for status in page.get('InstanceStatuses', []):
            events = status.get('Events', [])
            if events:
                instances_with_events.append({
                    'InstanceId': status['InstanceId'],
                    'Events': events
                })

    return instances_with_events

if __name__ == "__main__":
    instances = get_instances_with_scheduled_events()
    for i in instances:
        print(i)

• instance-reboot
• system-reboot
• instance-retirement
• system-maintenance
• instance-stop

​

4. Decide remediation logic

• instance-reboot / system-reboot:
  → You may choose to proactively reboot during a maintenance window.
• instance-stop:
  → Stop/start the instance on your controlled schedule (note: instance-store volumes are lost).
• instance-retirement:
  → Replace the instance (create a new one from AMI/snapshot, attach EIP, update ASG, etc.).

• Detects events.
• If instance-stop or instance-retirement is scheduled, it stops the instance (to force change under your control) and then starts it back.

​

5. Example remediation script (simple)

import boto3
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2')

ACTIONABLE_EVENT_CODES = {
    'instance-stop',
    'instance-retirement',
    'system-reboot',
    'instance-reboot'
}

def get_instances_with_actionable_events():
    affected = []
    paginator = ec2.get_paginator('describe_instance_status')
    page_iterator = paginator.paginate(IncludeAllInstances=True)

    for page in page_iterator:
        for status in page.get('InstanceStatuses', []):
            events = status.get('Events', [])
            if not events:
                continue

            actionable_events = [
                e for e in events if e['Code'] in ACTIONABLE_EVENT_CODES
            ]
            if actionable_events:
                affected.append({
                    'InstanceId': status['InstanceId'],
                    'Events': actionable_events
                })
    return affected

def stop_instance(instance_id):
    print(f"Stopping instance {instance_id}")
    ec2.stop_instances(InstanceIds=[instance_id])
    waiter = ec2.get_waiter('instance_stopped')
    waiter.wait(InstanceIds=[instance_id])
    print(f"Instance {instance_id} stopped")

def start_instance(instance_id):
    print(f"Starting instance {instance_id}")
    ec2.start_instances(InstanceIds=[instance_id])
    waiter = ec2.get_waiter('instance_running')
    waiter.wait(InstanceIds=[instance_id])
    print(f"Instance {instance_id} running")

def remediate_instance(instance):
    instance_id = instance['InstanceId']
    events = instance['Events']

    # Simple strategy:
    # - For any actionable event: stop & start instance within your window
    # Adjust this per event type if needed.
    event_codes = {e['Code'] for e in events}
    print(f"Remediating {instance_id} for events: {event_codes}")

    # Example: for retire/stop/reboot, do controlled stop/start
    try:
        stop_instance(instance_id)
        start_instance(instance_id)
    except ClientError as e:
        print(f"Error remediating {instance_id}: {e}")

def main():
    affected_instances = get_instances_with_actionable_events()
    if not affected_instances:
        print("No instances with actionable scheduled events.")
        return

    for instance in affected_instances:
        remediate_instance(instance)

if __name__ == "__main__":
    main()

​

6. Hardening as an operational process

• Run this script as a scheduled job (e.g., cron, Lambda with CloudWatch Events).
• Optionally:
  • Filter by tags (DescribeInstances) so you only remediate certain workloads.
  • Integrate notifications (SNS/Slack) before/after remediation.
  • Use different logic per event type (e.g., rebuild instance for instance-retirement).

# There is no Terraform argument or resource that can prevent or clear
# AWS EC2 scheduled events; these are initiated by AWS infrastructure.

# Terraform can only manage the instance configuration itself, e.g.:
resource "aws_instance" "example" {
  ami           = "AMI_ID"          # replace with your AMI ID
  instance_type = "INSTANCE_TYPE"   # e.g. "t3.micro"

  tags = {
    Name = "EXAMPLE_NAME"
  }
}

• Use the AWS Console: EC2 → Instances → select instance → “Scheduled events” tab → follow the guidance; for stop/retire events, stop/start or replace the instance (possibly via an Auto Scaling Group or launch template managed in Terraform).
• Or use AWS CLI: for example, stop/start or replace the instance (aws ec2 stop-instances, start-instances, or create a new instance and update any Terraform-managed references to its ID or to an Auto Scaling Group that replaces it).