Cloud 16 min read

AWS CLI First #8: Persistent Block Storage & Golden Images: EBS Volumes, KMS & Custom AMIs

Master enterprise AWS block storage and image lifecycle from the terminal. Learn how to provision gp3 EBS volumes, configure KMS encryption, format & mount filesystems with UUIDs, live-expand storage without downtime, and bake reusable Golden AMIs.

Mohammad Rizky Prawira portrait
Mohammad Rizky Prawira
terminal Series Guide • Part 8 of 17

AWS with CLI: The Terminal-First Mastery Path

Part of the CLI-first comprehensive learning path from foundational setup to production cloud architectures.

View Full Learning Path arrow_forward

In Part 7, we launched our first hardened Graviton compute instance with stateful Security Groups, User Data bootstrapping, and IMDSv2 token enforcement.

However, a virtual machine without persistent storage management and immutable packaging is an operational liability:

  1. Storage Persistence: What happens to application data when an instance is stopped or replaced?
  2. Performance Economics: How do we configure high-throughput storage without overpaying for unused disk capacity?
  3. Zero-Downtime Operations: How do we expand disk volumes live while the operating system and database are actively reading and writing?
  4. Disaster Recovery: How do we migrate disks across physical Availability Zone boundaries?
  5. Immutable Fleet Packaging: How do we bake configured servers into Golden AMIs so Auto Scaling groups can launch pre-warmed nodes in under 20 seconds?

In this guide, we will master Elastic Block Store (EBS), AWS Key Management Service (KMS) encryption, Linux Nitro NVMe device mapping, and Custom Golden AMI lifecycles entirely from the command line.

💰 Estimated Lab Cost & Resource Architecture

ComponentUsage in LabAWS Free TierOn-Demand Rate
Zero-Cost Public VPC + IGWBase Network Foundation100% Free$0.00
t4g.micro Compute (2 Nodes)Node 1 (1 hr) + Node 2 (15 mins)750 hrs/month Free~$0.0105 total
Public IPv4 Addresses2 Public IPs (~1 hr)Standard Rate~$0.0100 total
Secondary gp3 Disks (10–20 GB)Zone A & Restored Zone B Volumes30 GB/month Free~$0.0033 / hr
Snapshots & Custom AMIS3-backed Snapshots (~200 MB delta)5 GB/month Free< $0.0001
KMS Encryption (aws/ebs)AWS Managed CMK Envelope Encryption100% Free$0.00
Total Estimated Cost1–2 Hour Hands-On Session~$0.01 Total~$0.02 – $0.03 Total

[!TIP] Modern gp3 volumes include 3,000 baseline IOPS and 125 MB/s throughput for $0.00 extra. Running the automated Phase 7 Teardown Script cleans up all volumes, snapshots, AMIs, and instances, bringing your ongoing bill immediately back to $0.00.

Prerequisites: Base Compute Node Verification

This guide requires a running EC2 instance in Public Subnet A (ap-southeast-1a) from Part 7.

Option A: If your EC2 instance from Part 7 is already running

Retrieve your instance ID and Availability Zone:

INSTANCE_ID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=Production-WebServer-01" "Name=instance-state-name,Values=running" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Reservations[0].Instances[0].InstanceId" \
  --output text)

AZ=$(aws ec2 describe-instances \
  --instance-ids "$INSTANCE_ID" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Reservations[0].Instances[0].Placement.AvailabilityZone" \
  --output text)

echo "Target Instance: $INSTANCE_ID"
echo "Target Availability Zone: $AZ"

Option B: Fast-Track 1-Click Base Node Rebuild

If you tore down your instance after Part 7, spin up the base VPC and Graviton web server in 30 seconds:

cat <<'EOF' > setup-base-node.sh
#!/usr/bin/env bash
set -e
PROFILE="cloud-dev-admin"
REGION="ap-southeast-1"

# 1. Setup Public VPC (Idempotent: Reuses existing VPC/Subnet if present)
VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Production-VPC" --query "Vpcs[0].VpcId" --output text --region "$REGION" --profile "$PROFILE" 2>/dev/null || true)
if [ -z "$VPC_ID" ] || [ "$VPC_ID" = "None" ]; then
  VPC_ID=$(aws ec2 create-vpc --cidr-block "10.0.0.0/16" --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=Production-VPC}]' --region "$REGION" --profile "$PROFILE" --query "Vpc.VpcId" --output text)
  aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-support '{"Value":true}' --region "$REGION" --profile "$PROFILE"
  aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-hostnames '{"Value":true}' --region "$REGION" --profile "$PROFILE"
fi

PUB_SUB_A=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-A" --query "Subnets[0].SubnetId" --output text --region "$REGION" --profile "$PROFILE" 2>/dev/null || true)
if [ -z "$PUB_SUB_A" ] || [ "$PUB_SUB_A" = "None" ]; then
  PUB_SUB_A=$(aws ec2 create-subnet --vpc-id "$VPC_ID" --cidr-block "10.0.1.0/24" --availability-zone "${REGION}a" --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Public-Subnet-A}]' --region "$REGION" --profile "$PROFILE" --query "Subnet.SubnetId" --output text)
  aws ec2 modify-subnet-attribute --subnet-id "$PUB_SUB_A" --map-public-ip-on-launch --region "$REGION" --profile "$PROFILE"
fi

IGW_ID=$(aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$VPC_ID" --query "InternetGateways[0].InternetGatewayId" --output text --region "$REGION" --profile "$PROFILE" 2>/dev/null || true)
if [ -z "$IGW_ID" ] || [ "$IGW_ID" = "None" ]; then
  IGW_ID=$(aws ec2 create-internet-gateway --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=Production-IGW}]' --region "$REGION" --profile "$PROFILE" --query "InternetGateway.InternetGatewayId" --output text)
  aws ec2 attach-internet-gateway --internet-gateway-id "$IGW_ID" --vpc-id "$VPC_ID" --region "$REGION" --profile "$PROFILE"
fi

PUB_RTB_ID=$(aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Route-Table" --query "RouteTables[0].RouteTableId" --output text --region "$REGION" --profile "$PROFILE" 2>/dev/null || true)
if [ -z "$PUB_RTB_ID" ] || [ "$PUB_RTB_ID" = "None" ]; then
  PUB_RTB_ID=$(aws ec2 create-route-table --vpc-id "$VPC_ID" --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=Public-Route-Table}]' --region "$REGION" --profile "$PROFILE" --query "RouteTable.RouteTableId" --output text)
  aws ec2 create-route --route-table-id "$PUB_RTB_ID" --destination-cidr-block "0.0.0.0/0" --gateway-id "$IGW_ID" --region "$REGION" --profile "$PROFILE"
  aws ec2 associate-route-table --route-table-id "$PUB_RTB_ID" --subnet-id "$PUB_SUB_A" --region "$REGION" --profile "$PROFILE"
fi

# 2. Security Group & Key Pair (Idempotent)
SG_ID=$(aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$VPC_ID" "Name=group-name,Values=WebServer-SG" --query "SecurityGroups[0].GroupId" --output text --region "$REGION" --profile "$PROFILE" 2>/dev/null || true)
if [ -z "$SG_ID" ] || [ "$SG_ID" = "None" ]; then
  SG_ID=$(aws ec2 create-security-group --group-name "WebServer-SG" --description "Web Server SG" --vpc-id "$VPC_ID" --region "$REGION" --profile "$PROFILE" --query "GroupId" --output text)
  aws ec2 authorize-security-group-ingress --group-id "$SG_ID" --protocol tcp --port 80 --cidr "0.0.0.0/0" --region "$REGION" --profile "$PROFILE"
  MY_IP=$(curl -s https://checkip.amazonaws.com)
  aws ec2 authorize-security-group-ingress --group-id "$SG_ID" --protocol tcp --port 22 --cidr "${MY_IP}/32" --region "$REGION" --profile "$PROFILE"
fi

if [ ! -f Production-Server-Key.pem ]; then
  aws ec2 delete-key-pair --key-name "Production-Server-Key" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
  aws ec2 create-key-pair --key-name "Production-Server-Key" --key-type ed25519 --query "KeyMaterial" --output text --region "$REGION" --profile "$PROFILE" > Production-Server-Key.pem
  chmod 400 Production-Server-Key.pem
fi

# 3. Create User Data Bootstrapping Script (Installs Nginx for Golden AMI baking)
cat <<'USERDATA' > user-data.sh
#!/usr/bin/env bash
dnf update -y
dnf install -y nginx
systemctl enable --now nginx

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone)
PRIV_IP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4)
ARCH=$(uname -m)

cat <<HTML > /usr/share/nginx/html/index.html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>AWS CLI Hardened Compute Node</title>
  <style>
    body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #0f172a; color: #f8fafc; display: flex; justify-content: center; align-items: center; height: 100vh; margin: 0; }
    .card { background: #1e293b; padding: 2.5rem; border-radius: 12px; border: 1px solid #334155; box-shadow: 0 10px 25px rgba(0,0,0,0.5); max-width: 500px; width: 100%; }
    h1 { color: #38bdf8; font-size: 1.5rem; margin-top: 0; border-bottom: 1px solid #334155; padding-bottom: 0.75rem; }
    .stat { display: flex; justify-content: space-between; padding: 0.5rem 0; border-bottom: 1px solid #1e293b; font-family: monospace; font-size: 0.9rem; }
    .label { color: #94a3b8; }
    .val { color: #4ade80; font-weight: bold; }
    .badge { display: inline-block; padding: 0.25rem 0.75rem; background: #0369a1; color: #e0f2fe; border-radius: 9999px; font-size: 0.75rem; margin-top: 1rem; }
  </style>
</head>
<body>
  <div class="card">
    <h1>🚀 EC2 Node Online (AWS CLI First)</h1>
    <div class="stat"><span class="label">Instance ID:</span><span class="val">${INSTANCE_ID}</span></div>
    <div class="stat"><span class="label">Availability Zone:</span><span class="val">${AZ}</span></div>
    <div class="stat"><span class="label">Private IPv4:</span><span class="val">${PRIV_IP}</span></div>
    <div class="stat"><span class="label">CPU Architecture:</span><span class="val">${ARCH} (Graviton)</span></div>
    <div class="stat"><span class="label">IMDS Security:</span><span class="val">IMDSv2 Enforced</span></div>
    <div class="badge">Bootstrapped via cloud-init User Data</div>
  </div>
</body>
</html>
HTML
USERDATA

# 4. Launch Instance with User Data
AMI_ID=$(aws ssm get-parameter --name "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64" --region "$REGION" --profile "$PROFILE" --query "Parameter.Value" --output text 2>/dev/null || aws ec2 describe-images --owners amazon --filters "Name=name,Values=al2023-ami-2023*-arm64" "Name=state,Values=available" "Name=architecture,Values=arm64" --region "$REGION" --profile "$PROFILE" --query "reverse(sort_by(Images, &CreationDate))[0].ImageId" --output text)
INSTANCE_ID=$(aws ec2 run-instances --image-id "$AMI_ID" --instance-type "t4g.micro" --key-name "Production-Server-Key" --security-group-ids "$SG_ID" --subnet-id "$PUB_SUB_A" --user-data file://user-data.sh --metadata-options "HttpTokens=required" --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Production-WebServer-01}]' --region "$REGION" --profile "$PROFILE" --query "Instances[0].InstanceId" --output text)

echo "Waiting for instance $INSTANCE_ID..."
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID" --region "$REGION" --profile "$PROFILE"
AZ="${REGION}a"

echo "=== Base Node Ready ==="
echo "INSTANCE_ID=$INSTANCE_ID"
echo "AZ=$AZ"
EOF

source setup-base-node.sh

Phase 1: Provisioning & Attaching an Encrypted gp3 Secondary EBS Volume

The Architecture: Instance Store vs. EBS & Volume Types

AWS provides two primary categories of block storage:

+-----------------------------------------------------------------------------------+
|                        Instance Store vs. Elastic Block Store                     |
+-----------------------------------------------------------------------------------+

 1. INSTANCE STORE (Host-Attached Ephemeral Storage):
    * Physical NVMe SSDs soldered directly onto the host server chassis.
    * Performance: Ultra-low microsecond latency, millions of IOPS.
    * CRITICAL TRAP: EPHEMERAL. If the instance stops, terminates, or underlying
      hardware faults occur, ALL DATA IS PERMANENTLY LOST!
    * Use Case: High-speed caching, temporary swap space, Redis scratch buffers.

 2. ELASTIC BLOCK STORE - EBS (Network-Attached Virtual SAN):
    * Dedicated storage volume connected over a high-speed 100Gbps AWS network fabric.
    * Performance: Up to 256,000 IOPS and 4,000 MB/s throughput (`io2 Block Express`).
    * PERSISTENT: Survives instance stops, reboots, and terminations independently.
    * Use Case: Databases (PostgreSQL, MySQL), OS root drives, file servers.

The gp3 vs. gp2 Revolution:

In older cloud architectures, AWS used gp2 volumes where IOPS were tied directly to disk size (3 IOPS per GB). To get 3,000 IOPS, engineers were forced to buy a 1,000 GB disk!

With modern gp3, AWS decoupled storage size from performance:

  • Baseline Free Performance: Every gp3 volume gets 3,000 IOPS and 125 MB/s throughput included for free, even on a tiny 10 GB disk.
  • Cost Advantage: gp3 is 20% cheaper per GB ($0.08/GB-month) than gp2 ($0.10/GB-month).
+-----------------------------------------------------------------------------------+
|                            EBS Volume Types Comparison                            |
+-------------------+----------------+------------------+---------------------------+
| Volume Type       | Max IOPS       | Max Throughput   | Ideal Workload            |
+-------------------+----------------+------------------+---------------------------+
| gp3 (General)     | 16,000         | 1,000 MB/s       | Web servers, Dev/Test, DB |
| io2 Block Express | 256,000        | 4,000 MB/s       | Mission-critical Oracle/SAP|
| st1 (Throughput)  | 500 (HDD)      | 500 MB/s         | Big Data, Log Processing  |
| sc1 (Cold HDD)    | 250 (HDD)      | 250 MB/s         | Infrequently accessed data|
+-------------------+----------------+------------------+---------------------------+

The Physical AZ-Lock Rule:

[!WARNING] An EBS volume is physically located in a specific datacenter rack inside one Availability Zone. A volume created in ap-southeast-1a cannot be attached to an EC2 instance in ap-southeast-1b. To move it across AZs, you must snapshot it first!

Step 1: Create a 10 GB KMS-Encrypted gp3 Volume

We will create a 10 GB volume with AWS Key Management Service (KMS) encryption and wait for it to enter the available state:

VOL_ID=$(aws ec2 create-volume \
  --availability-zone "$AZ" \
  --size 10 \
  --volume-type gp3 \
  --encrypted \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=Production-Data-Volume},{Key=Environment,Value=Production}]' \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "VolumeId" \
  --output text)

aws ec2 wait volume-available --volume-ids "$VOL_ID" --region ap-southeast-1 --profile cloud-dev-admin

Step 2: Attach the Volume to the Running EC2 Instance

Attach the volume as /dev/sdf and wait for attachment to complete:

aws ec2 attach-volume \
  --volume-id "$VOL_ID" \
  --instance-id "$INSTANCE_ID" \
  --device "/dev/sdf" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin

aws ec2 wait volume-in-use --volume-ids "$VOL_ID" --region ap-southeast-1 --profile cloud-dev-admin

🔍 How to Validate Volume Attachment:

aws ec2 describe-volumes \
  --volume-ids "$VOL_ID" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Volumes[*].[VolumeId, State, Size, VolumeType, Encrypted, Attachments[0].InstanceId, Attachments[0].Device]" \
  --output table

Expected Output:

-------------------------------------------------------------------------------------------
|                                     DescribeVolumes                                     |
+---------------------+---------+----+------+------+----------------------+---------------+
|  vol-0123456789abc  |  in-use | 10 | gp3  | True |  i-0987654321fedcba  |  /dev/sdf     |
+---------------------+---------+----+------+------+----------------------+---------------+

Phase 2: Linux Filesystem Formatting, Mounting & Persistent /etc/fstab (UUID)

The Architecture: Nitro NVMe Device Mapping & the Boot Hang Trap

On modern AWS Nitro System instances (such as Graviton t4g, c6g, m7g), virtual block devices are attached via high-speed NVMe controllers.

  • Although we requested attachment as /dev/sdf, the Linux kernel discovers it as an NVMe device: /dev/nvme1n1 (where nvme0n1 is the root OS disk).
  • The /etc/fstab Disaster: If you hardcode /dev/nvme1n1 or /dev/sdf inside /etc/fstab, device order can shift upon reboot. If the drive fails to mount, the Linux kernel halts and drops into emergency rescue mode, taking your web server offline!
  • The Solution: Always mount using the filesystem’s UUID and include the nofail mount option.
+-----------------------------------------------------------------------------------+
|                     Linux Nitro Block Device Mapping Architecture                 |
+-----------------------------------------------------------------------------------+

 AWS API Request:  Attach Volume  ==> Device Name: /dev/sdf
                                             |
                                             v (Nitro Virtualization Layer)
 Linux Kernel:     Discovered as  ==> /dev/nvme1n1
                                             |
                                             v (Format with XFS)
 Filesystem UUID:  f47ac10b-58cc-4372-a567-0e02b2c3d479
                                             |
                                             v (Mount into Directory)
 Mount Point:      /data (Persistent Storage)

Step 1: Connect to EC2 Instance via SSH

Fetch the instance’s public IP and connect:

PUBLIC_IP=$(aws ec2 describe-instances \
  --instance-ids "$INSTANCE_ID" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Reservations[0].Instances[0].PublicIpAddress" \
  --output text)

echo "Connecting to ec2-user@$PUBLIC_IP..."
ssh -i Production-Server-Key.pem -o StrictHostKeyChecking=no ec2-user@$PUBLIC_IP

Step 2: Format & Mount the Volume inside Linux

Run the following commands individually inside the EC2 SSH session:

1. Inspect block devices (Notice /dev/nvme1n1 with 10 GB size):

lsblk

2. Format the disk with high-performance XFS filesystem:

sudo mkfs -t xfs /dev/nvme1n1

3. Create the mount target directory:

sudo mkdir -p /data

4. Mount the volume:

sudo mount /dev/nvme1n1 /data

5. Write test application data:

echo "Production Application Storage Active" | sudo tee /data/app.log

Step 3: Configure Safe Boot Mounting in /etc/fstab (UUID + nofail)

1. Retrieve the unique UUID of the new filesystem:

UUID=$(sudo blkid -s UUID -o value /dev/nvme1n1)

2. Add the mount entry to /etc/fstab with the nofail option:

echo "UUID=${UUID}  /data  xfs  defaults,nofail  0  2" | sudo tee -a /etc/fstab

3. Test /etc/fstab without rebooting (unmount and mount all):

sudo umount /data
sudo mount -a

4. Verify that data persisted across remount:

cat /data/app.log

🔍 How to Validate Linux Mount:

df -h /data

Expected Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1     10G  104M  9.9G   2% /data

Phase 3: Live EBS Volume Expansion Without Downtime (Elastic Volumes)

The Architecture: Elastic Volumes & In-Flight File System Growth

In traditional on-premise infrastructure, resizing a disk required taking servers offline, backing up partitions, and re-partitioning.

With AWS Elastic Volumes:

  1. You modify the EBS volume size on the AWS control plane while the virtual machine is actively processing I/O.
  2. The volume enters modifying state, expanding capacity in seconds.
  3. You run a single Linux command (xfs_growfs or resize2fs) to extend the live filesystem into the newly allocated space with 0.00 seconds of downtime.
+-----------------------------------------------------------------------------------+
|                        Live Elastic Volume Expansion Flow                         |
+-----------------------------------------------------------------------------------+

 1. Initial State:      [ 10 GB EBS Volume ] ===> Mounted on /data (10 GB XFS)
                                 |
 2. AWS CLI Modify:     aws ec2 modify-volume --size 20
                                 |
 3. Storage Layer:      [ 20 GB EBS Volume ] ===> (Operating system still sees 10 GB)
                                 |
 4. Linux xfs_growfs:   sudo xfs_growfs -d /data
                                 |
 5. Final State:        [ 20 GB EBS Volume ] ===> Mounted on /data (20 GB XFS) - 0 Downtime!

Step 1: Expand Volume from 10 GB to 20 GB via AWS CLI

Open a new terminal tab on your local machine:

aws ec2 modify-volume \
  --volume-id "$VOL_ID" \
  --size 20 \
  --region ap-southeast-1 \
  --profile cloud-dev-admin

Step 2: Extend the Live Linux Filesystem (Zero Downtime)

Switch back to your EC2 SSH session:

1. Inspect block devices (Notice nvme1n1 is now 20 GB at the physical hardware layer):

lsblk

2. Grow the active XFS filesystem live:

sudo xfs_growfs -d /data

🔍 How to Validate Live Disk Growth:

df -h /data

Expected Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1     20G  175M   20G   1% /data

Volume expanded from 10 GB to 20 GB live with zero application downtime or data loss!

Phase 4: Point-in-Time EBS Snapshots & Cross-AZ Migration

The Architecture: Incremental Snapshots & Bypassing the AZ-Lock

  • EBS Snapshots: Point-in-time block-level backups stored redundantly in Amazon S3.
  • Incremental Mechanics: Only modified disk blocks are saved on subsequent snapshots, drastically lowering storage costs.
  • AZ Migration: Because EBS volumes cannot cross Availability Zones, taking a snapshot allows you to restore a brand-new volume in any Availability Zone or even copy the snapshot to another AWS Region.
+-----------------------------------------------------------------------------------+
|                        Cross-AZ Disaster Recovery Migration                       |
+-----------------------------------------------------------------------------------+

 [ Availability Zone: ap-southeast-1a ]
   * EC2 Instance (WebServer-01)
   * EBS Volume (vol-data-10gb)  ===> 1. Take Snapshot
                                           |
                                           v
                             [ Amazon S3 Backend Storage ]
                               (Point-in-Time Snapshot: snap-012345)
                                           |
                                           v 2. Restore Volume into Zone B
 [ Availability Zone: ap-southeast-1b ]
   * New EBS Volume (vol-data-zoneB) ===> Attached to Standby Compute Node!

Step 1: Create a Point-in-Time Snapshot

Run on your local machine:

SNAP_ID=$(aws ec2 create-snapshot \
  --volume-id "$VOL_ID" \
  --description "Point-in-time backup of Production-Data-Volume" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=Production-Data-Snapshot}]' \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "SnapshotId" \
  --output text)

aws ec2 wait snapshot-completed --snapshot-ids "$SNAP_ID" --region ap-southeast-1 --profile cloud-dev-admin

Step 2: Restore the Snapshot into Availability Zone B (ap-southeast-1b)

RESTORED_VOL_ID=$(aws ec2 create-volume \
  --snapshot-id "$SNAP_ID" \
  --availability-zone "ap-southeast-1b" \
  --volume-type gp3 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=Restored-Data-Volume-ZoneB}]' \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "VolumeId" \
  --output text)

aws ec2 wait volume-available --volume-ids "$RESTORED_VOL_ID" --region ap-southeast-1 --profile cloud-dev-admin

🔍 How to Validate Cross-AZ Restoration:

aws ec2 describe-volumes \
  --volume-ids "$RESTORED_VOL_ID" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Volumes[*].[VolumeId, AvailabilityZone, Size, State, SnapshotId]" \
  --output table

Expected Output:

---------------------------------------------------------------------------------------
|                                   DescribeVolumes                                   |
+---------------------+-------------------+----+------------+-------------------------+
|  vol-0998877665544  |  ap-southeast-1b  | 20 | available  |  snap-0123456789abcdef0 |
+---------------------+-------------------+----+------------+-------------------------+

Phase 5: Baking a Custom Golden AMI (Image Creation Lifecycle)

The Architecture: Mutable Config vs. Immutable Golden AMIs

In high-scale cloud fleets, relying on user-data.sh to download packages (dnf install, compile software, configure firewalls) on every new server creates massive problems:

  1. Slow Auto Scaling: If traffic spikes, new instances take 3 to 7 minutes to boot and configure before they can accept user traffic.
  2. Upstream Failure Risk: If GitHub, NPM, or package repositories have an outage, new instances fail to boot.

The Golden AMI Solution: Configure your server once, test it, and bake it into a pre-configured Amazon Machine Image (AMI). New instances launch from the Golden AMI in under 20 seconds with zero external dependencies!

+-----------------------------------------------------------------------------------+
|                            The Golden AMI Pipeline                                |
+-----------------------------------------------------------------------------------+

 [ Master EC2 Instance ] (Nginx + Hardened Configs + Custom Code)
            |
            v  aws ec2 create-image
 [ Custom Golden AMI: Production-Golden-App-v1.0 ]
            |
            +------------------------+------------------------+
            |                        |                        |
            v                        v                        v
 [ Fleet Instance 1 ]     [ Fleet Instance 2 ]     [ Fleet Instance 3 ]
  (Ready in 15s)           (Ready in 15s)           (Ready in 15s)

Step 1: Bake the Golden AMI from the Live Instance

CUSTOM_AMI_ID=$(aws ec2 create-image \
  --instance-id "$INSTANCE_ID" \
  --name "Production-Golden-App-v1.0" \
  --description "Hardened Amazon Linux 2023 ARM64 with pre-configured Nginx and telemetry" \
  --no-reboot \
  --tag-specifications 'ResourceType=image,Tags=[{Key=Name,Value=Production-Golden-App-v1.0}]' \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "ImageId" \
  --output text)

aws ec2 wait image-available --image-ids "$CUSTOM_AMI_ID" --region ap-southeast-1 --profile cloud-dev-admin

Step 2: Launch a Second Node from the Custom Golden AMI

Launch a new instance using $CUSTOM_AMI_ID with zero User Data script:

PUB_SUB_A=$(aws ec2 describe-subnets --filters "Name=tag:Name,Values=Public-Subnet-A" --region ap-southeast-1 --profile cloud-dev-admin --query "Subnets[0].SubnetId" --output text)
SG_ID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=WebServer-SG" --region ap-southeast-1 --profile cloud-dev-admin --query "SecurityGroups[0].GroupId" --output text)

GOLDEN_INSTANCE_ID=$(aws ec2 run-instances \
  --image-id "$CUSTOM_AMI_ID" \
  --instance-type "t4g.micro" \
  --key-name "Production-Server-Key" \
  --security-group-ids "$SG_ID" \
  --subnet-id "$PUB_SUB_A" \
  --metadata-options "HttpTokens=required" \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Production-Fleet-Node-02}]' \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Instances[0].InstanceId" \
  --output text)

aws ec2 wait instance-running --instance-ids "$GOLDEN_INSTANCE_ID" --region ap-southeast-1 --profile cloud-dev-admin

🔍 How to Validate Golden AMI Deployment:

Fetch the second node’s public IP and test immediately:

NODE2_IP=$(aws ec2 describe-instances \
  --instance-ids "$GOLDEN_INSTANCE_ID" \
  --region ap-southeast-1 \
  --profile cloud-dev-admin \
  --query "Reservations[0].Instances[0].PublicIpAddress" \
  --output text)

echo "Testing Pre-Warmed Node: http://${NODE2_IP}"
curl -s "http://${NODE2_IP}" | grep -E "Instance ID|CPU Architecture"

Nginx started automatically and served traffic immediately upon initial boot with zero bootstrap delay!

Phase 6: Advanced Storage Patterns: Multi-Attach & AWS DLM Automation

1. io2 Multi-Attach (Shared Clustered Storage)

Standard EBS volumes can only attach to one EC2 instance at a time.

With io2 Multi-Attach, you can attach a single Provisioned IOPS SSD volume to up to 16 AWS Nitro instances in the same Availability Zone simultaneously.

  • Requirement: Must use a cluster-aware filesystem (e.g., GFS2, GlusterFS, or Oracle RAC) that manages distributed write locks to prevent data corruption.

2. AWS Data Lifecycle Manager (DLM) Automated Snapshots

Instead of writing manual cron scripts to snapshot EBS volumes, use AWS Data Lifecycle Manager (DLM) to automate snapshot creation, cross-region replication, and automated retention policies (e.g., keep 7 daily snapshots and purge older backups):

# Example DLM Policy targeting volumes with tag 'Environment=Production'
aws dlm get-lifecycle-policies --region ap-southeast-1 --profile cloud-dev-admin

Phase 7: Automated Teardown & Resource Cleanup

Clean up all storage volumes, snapshots, AMIs, and compute nodes created during this lab:

echo "=== Starting Part 8 Cleanup ==="

# 1. Terminate EC2 Instances
echo "Terminating instances..."
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" "$GOLDEN_INSTANCE_ID" --region ap-southeast-1 --profile cloud-dev-admin
aws ec2 wait instance-terminated --instance-ids "$INSTANCE_ID" "$GOLDEN_INSTANCE_ID" --region ap-southeast-1 --profile cloud-dev-admin

# 2. Delete Secondary EBS Volumes
echo "Deleting EBS volumes..."
aws ec2 delete-volume --volume-id "$VOL_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
aws ec2 delete-volume --volume-id "$RESTORED_VOL_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true

# 3. Deregister Golden AMI and Delete its Backing Snapshot
echo "Deregistering Custom Golden AMI..."
AMI_SNAP_ID=$(aws ec2 describe-images --image-ids "$CUSTOM_AMI_ID" --region ap-southeast-1 --profile cloud-dev-admin --query "Images[0].BlockDeviceMappings[0].Ebs.SnapshotId" --output text 2>/dev/null || true)
aws ec2 deregister-image --image-id "$CUSTOM_AMI_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
if [ -n "$AMI_SNAP_ID" ] && [ "$AMI_SNAP_ID" != "None" ]; then
  aws ec2 delete-snapshot --snapshot-id "$AMI_SNAP_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
fi

# 4. Delete Data Volume Snapshot
echo "Deleting data snapshot..."
aws ec2 delete-snapshot --snapshot-id "$SNAP_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true

# 5. Clean up Security Group, Key Pair and VPC
echo "Cleaning up Security Group and Key Pair..."
SG_ID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=WebServer-SG" --query "SecurityGroups[0].GroupId" --output text --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true)
if [ -n "$SG_ID" ] && [ "$SG_ID" != "None" ]; then
  aws ec2 delete-security-group --group-id "$SG_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
fi
aws ec2 delete-key-pair --key-name "Production-Server-Key" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true

echo "Cleaning up Public VPC..."
VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Production-VPC" --query "Vpcs[0].VpcId" --output text --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true)
if [ -n "$VPC_ID" ] && [ "$VPC_ID" != "None" ]; then
  # 1. Delete Subnets first (releases route table associations)
  SUB_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-A" --query "Subnets[0].SubnetId" --output text --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true)
  if [ -n "$SUB_ID" ] && [ "$SUB_ID" != "None" ]; then
    aws ec2 delete-subnet --subnet-id "$SUB_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
  fi

  # 2. Delete Custom Route Table (now unassociated)
  RTB_ID=$(aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Route-Table" --query "RouteTables[0].RouteTableId" --output text --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true)
  if [ -n "$RTB_ID" ] && [ "$RTB_ID" != "None" ]; then
    aws ec2 delete-route-table --route-table-id "$RTB_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
  fi

  # 3. Detach and Delete Internet Gateway
  IGW_ID=$(aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$VPC_ID" --query "InternetGateways[0].InternetGatewayId" --output text --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true)
  if [ -n "$IGW_ID" ] && [ "$IGW_ID" != "None" ]; then
    aws ec2 detach-internet-gateway --internet-gateway-id "$IGW_ID" --vpc-id "$VPC_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
    aws ec2 delete-internet-gateway --internet-gateway-id "$IGW_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
  fi

  # 4. Delete VPC
  aws ec2 delete-vpc --vpc-id "$VPC_ID" --region ap-southeast-1 --profile cloud-dev-admin 2>/dev/null || true
fi

# 6. Clean up local files
rm -f setup-base-node.sh Production-Server-Key.pem user-data.sh

echo "=== Part 8 Teardown Complete! Zero Storage, Compute or Network Cost Remaining. ==="

Quick Reference: Essential EBS, Snapshot & AMI CLI Commands

ActionCLI Command
Create Encrypted gp3 Volumeaws ec2 create-volume --size <gb> --volume-type gp3 --availability-zone <az> --encrypted
Attach Volumeaws ec2 attach-volume --volume-id <vol> --instance-id <inst> --device /dev/sdf
Live Expand Volumeaws ec2 modify-volume --volume-id <vol> --size <new_size>
Grow Linux XFS Filesystemsudo xfs_growfs -d /data
Create Snapshotaws ec2 create-snapshot --volume-id <vol> --description <desc>
Restore Volume from Snapshotaws ec2 create-volume --snapshot-id <snap> --availability-zone <az> --volume-type gp3
Bake Custom Golden AMIaws ec2 create-image --instance-id <inst> --name <name> --no-reboot
Deregister AMIaws ec2 deregister-image --image-id <ami>

Summary & What’s Next

In this eighth installment of AWS with CLI, we mastered persistent storage and golden image management:

  1. We evaluated Instance Store vs. EBS and leveraged the economics of gp3 (3,000 baseline IOPS free).
  2. We formatted Nitro NVMe block devices (/dev/nvme1n1) and mounted them safely using UUID and nofail.
  3. We executed a live, zero-downtime volume expansion from 10 GB to 20 GB.
  4. We bypassed the physical AZ-lock using point-in-time snapshots and migrated storage to ap-southeast-1b.
  5. We baked a hardened Golden AMI and launched an instant-ready fleet node in under 20 seconds.

Now that we know how to configure storage and bake machine images, how do we manage and operate fleets of EC2 instances without ever opening Port 22 SSH?

In Part 9, we explore Zero-Trust Fleet Operations: AWS Systems Manager (SSM) Session Manager & Run Command, learning how to access private instances with zero open inbound ports and execute fleet-wide commands from our terminal!

Related & Recommended Guides

Continue exploring related systems architectures and engineering field notes.

Cloud15 min read

AWS CLI First #9: Zero-Trust Fleet Operations: AWS Systems Manager (SSM) Session Manager & Run Command

Eliminate Port 22 SSH forever. Learn how to manage EC2 instances with zero open inbound ports using AWS Systems Manager (SSM) Session Manager, enable encrypted CloudWatch session auditing, tunnel private ports, and run fleet-wide remote commands via CLI.

Read article arrow_forward
Cloud14 min read

AWS CLI First #4: AWS Global Infrastructure Mastery: Regions, Availability Zones & AZ IDs via CLI

Master the physical foundation of AWS. Explore AWS global geography, the 4 golden rules of region selection, Availability Zone physical architecture, AZ-ID mapping across multi-account organizations, and managing opt-in regions entirely from the terminal.

Read article arrow_forward