Cloud 18 min read

AWS CLI First #12: S3 Mastery & Distributed Files: Bucket Policies, Lifecycle, Encryption & EFS

Master cloud storage from the terminal: enforce S3 bucket policies with TLS and KMS CMK encryption, configure tiered lifecycle rules, optimize multi-threaded CLI sync, and mount Multi-AZ Elastic File System (EFS) shares.

Mohammad Rizky Prawira portrait
Mohammad Rizky Prawira
terminal Series Guide • Part 12 of 17

AWS with CLI: The Terminal-First Mastery Path

Part of the CLI-first comprehensive learning path from foundational setup to production cloud architectures.

In modern cloud computing, compute is ephemeral, but data is perpetual. While EC2 compute instances scale up and down or get replaced by Auto Scaling Groups, enterprise applications require robust, highly available storage that spans across multiple servers and regions.

AWS provides two distinct distributed storage primitives:

  1. Amazon Simple Storage Service (S3): Massive, exabyte-scale object storage designed for 99.999999999% (11 9’s) of durability, accessible via REST APIs.
  2. Amazon Elastic File System (EFS): Fully managed, serverless, cloud-native NFSv4 shared file system that can be concurrently mounted across hundreds of EC2 instances and containers.

In this guide, we master both storage paradigms purely from the terminal. We will lock down S3 buckets with strict TLS and KMS bucket policies, configure automated tiered lifecycle rules to slash storage bills by 70%, optimize high-throughput multi-part CLI sync, and provision a Multi-AZ EFS shared file system mounted concurrently across our fleet.


💰 Estimated Lab Cost & Resource Architecture

1. Cost Breakdown for this Lab

All resources in this lab are provisioned on-demand in the Asia Pacific (Singapore) ap-southeast-1 region:

ResourceQuantityUnit Cost (Hourly)Estimated Lab Cost (1 Hour)
Amazon S3 Standard Storage< 100 MB Test Data$0.025 / GB / month< $0.001
Amazon S3 API Operations~100 PUT/GET calls$0.005 / 1,000 requests< $0.001
Elastic File System (EFS)1 File System (< 10 MB)$0.30 / GB / month< $0.001
AWS KMS Customer Managed Key1 Test Key$1.00 / month (prorated hourly)~$0.002
EC2 Graviton Test Node (t4g.micro)1 Node (Optional for EFS test)$0.0084 / hour~$0.009
Total Estimated Lab Cost~$0.013 USD

2. Architecture Diagram

+----------------------------------------------------------------+
|                   Enterprise Storage Architecture              |
+-------------------------------+--------------------------------+
                                |
        +-----------------------+-----------------------+
        |                                               |
        v                                               v
+-------------------------------+ +------------------------------+
|     Amazon S3 Object Store    | |  Amazon Elastic File System  |
|  - Block Public Access (All)  | |  - Protocol: NFSv4.1 POSIX   |
|  - SSE-KMS Customer Managed   | |  - Performance: General Purpose
|  - TLS 1.2+ Enforced Policy   | |  - Multi-AZ Mount Targets    |
+---------------+---------------+ +--------------+---------------+
                |                                |
                v                                v
+-------------------------------+ +------------------------------+
|  Lifecycle Tiering Pipeline   | |    Concurrent Client Mount   |
|  - Day 0:  S3 Standard        | |  - EC2 Worker Node A (AZ 1a) |
|  - Day 30: S3 Standard-IA     | |  - EC2 Worker Node B (AZ 1b) |
|  - Day 90: S3 Glacier Flex    | |  - Shared Directory: /mnt/efs|
|  - Day 365: Auto Expiration   | |                              |
+-------------------------------+ +------------------------------+

🛠️ Phase 1: Secure S3 Bucket Provisioning & Block Public Access

Every S3 bucket name is globally unique across all AWS accounts worldwide. By default, misconfigured S3 buckets have historically been the leading cause of data leaks. Today, AWS mandates security-first provisioning.

1. Generating a Globally Unique Bucket Name

Let’s generate a unique bucket identifier combining your AWS Account ID and region:

REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"

ACCOUNT_ID=$(aws sts get-caller-identity --profile "$PROFILE" --query "Account" --output text)
BUCKET_NAME="production-data-vault-${ACCOUNT_ID}-${REGION}"

echo "Target Bucket Name: $BUCKET_NAME"

2. Provisioning the S3 Bucket

In all regions other than us-east-1, the S3 API requires specifying a LocationConstraint:

aws s3api create-bucket \
  --bucket "$BUCKET_NAME" \
  --create-bucket-configuration LocationConstraint="$REGION" \
  --region "$REGION" \
  --profile "$PROFILE"

echo "Bucket $BUCKET_NAME successfully created."

3. Enforcing Account-Level & Bucket-Level Block Public Access

Never rely on console defaults. Explicitly enforce the four S3 Public Access Block settings via CLI:

aws s3api put-public-access-block \
  --bucket "$BUCKET_NAME" \
  --public-access-block-configuration '{
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true,
    "BlockPublicPolicy": true,
    "RestrictPublicBuckets": true
  }' \
  --profile "$PROFILE"

echo "Public Access Block enforced (100% Private)."

🔐 Phase 2: Security & Encryption: Enforcing TLS 1.2+ and KMS CMK

A bucket is not enterprise-ready until it enforces Encryption-in-Transit (TLS) and Encryption-at-Rest (KMS) through a declarative Bucket Policy.

1. Creating a Dedicated Customer Managed Key (CMK)

While AWS S3 offers default AES-256 (AES256), enterprise compliance frameworks (PCI-DSS, HIPAA, SOC 2) require Customer Managed Keys (CMK) with audited rotation:

KMS_KEY_ID=$(aws kms create-key \
  --description "KMS Key for S3 Data Vault Encryption" \
  --tags "TagKey=Environment,TagValue=Production" \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "KeyMetadata.KeyId" \
  --output text)

aws kms create-alias \
  --alias-name "alias/s3-data-vault-key" \
  --target-key-id "$KMS_KEY_ID" \
  --region "$REGION" \
  --profile "$PROFILE"

KMS_KEY_ARN=$(aws kms describe-key --key-id "$KMS_KEY_ID" --region "$REGION" --profile "$PROFILE" --query "KeyMetadata.Arn" --output text)
echo "KMS Key ARN: $KMS_KEY_ARN"

2. Setting Default Bucket Encryption to KMS

Configure the bucket to automatically encrypt every uploaded object with our KMS CMK:

aws s3api put-bucket-encryption \
  --bucket "$BUCKET_NAME" \
  --server-side-encryption-configuration "{
    \"Rules\": [
      {
        \"ApplyServerSideEncryptionByDefault\": {
          \"SSEAlgorithm\": \"aws:kms\",
          \"KMSMasterKeyID\": \"$KMS_KEY_ARN\"
        },
        \"BucketKeyEnabled\": true
      }
    ]
  }" \
  --profile "$PROFILE"

echo "Default SSE-KMS encryption with S3 Bucket Keys enabled."

[!TIP] S3 Bucket Keys (FinOps): Enabling BucketKeyEnabled: true decreases KMS API request traffic by up to 99%, reducing KMS operational costs when reading and writing thousands of objects!

3. Crafting a Strict Security Bucket Policy (Deny Insecure HTTP)

We write a bucket policy that denies any request that does not use HTTPS (aws:SecureTransport: false):

cat <<POLICY > bucket-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceTLSRequestsOnly",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::$BUCKET_NAME",
        "arn:aws:s3:::$BUCKET_NAME/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}
POLICY

aws s3api put-bucket-policy \
  --bucket "$BUCKET_NAME" \
  --policy file://bucket-policy.json \
  --profile "$PROFILE"

rm -f bucket-policy.json
echo "TLS 1.2+ In-Transit Enforcement Policy attached."

⚡ Phase 3: High-Throughput S3 CLI Operations & Versioning

The AWS CLI contains two command families for S3:

  1. aws s3api: Low-level, 1-to-1 mappings with AWS REST API endpoints.
  2. aws s3: High-level commands (cp, sync, mv, rm) featuring multi-threaded uploads and automated multipart chunking.

1. Enabling S3 Object Versioning

Versioning protects against accidental deletions and malicious ransomware overrides:

aws s3api put-bucket-versioning \
  --bucket "$BUCKET_NAME" \
  --versioning-configuration Status=Enabled \
  --profile "$PROFILE"

echo "S3 Versioning Enabled."

2. Multi-Threaded Sync with File Filtering

Create a local sample directory with assorted files:

mkdir -p ./test-upload/logs ./test-upload/config
echo "Server Config v1.0" > ./test-upload/config/app.conf
echo "Database Connection" > ./test-upload/config/db.conf
echo "2026-09-08 HTTP 200 OK" > ./test-upload/logs/access.log
echo "Temporary Cache File" > ./test-upload/temp.tmp

# Sync only configuration and log files, excluding temporary files
aws s3 sync ./test-upload/ "s3://${BUCKET_NAME}/backups/" \
  --exclude "*.tmp" \
  --profile "$PROFILE"

# Verify uploaded files
aws s3 ls "s3://${BUCKET_NAME}/backups/" --recursive --profile "$PROFILE"

3. Testing Version History Retrieval

Modify app.conf and re-upload to witness versioning in action:

echo "Server Config v2.0 (Updated)" > ./test-upload/config/app.conf
aws s3 cp ./test-upload/config/app.conf "s3://${BUCKET_NAME}/backups/config/app.conf" --profile "$PROFILE"

# Inspect versions
aws s3api list-object-versions \
  --bucket "$BUCKET_NAME" \
  --prefix "backups/config/app.conf" \
  --profile "$PROFILE" \
  --query "Versions[*].[VersionId,IsLatest,LastModified]" \
  --output table

📉 Phase 4: S3 Lifecycle Policies: Automated Tiering (FinOps)

Storing inactive files in S3 Standard costs $0.025/GB/month. In contrast:

  • S3 Standard-Infrequent Access (IA): $0.0125/GB/month (50% cheaper)
  • S3 Glacier Flexible Retrieval: $0.0036/GB/month (85% cheaper)
  • S3 Glacier Deep Archive: $0.00099/GB/month (96% cheaper)

Let’s configure an automated lifecycle policy that shifts objects down the storage hierarchy and purges obsolete versions:

cat <<'EOF' > lifecycle.json
{
  "Rules": [
    {
      "ID": "AutomatedCostOptimizationRule",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "backups/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "GLACIER"
        }
      ],
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 180
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket "$BUCKET_NAME" \
  --lifecycle-configuration file://lifecycle.json \
  --profile "$PROFILE"

rm -f lifecycle.json
echo "Automated Lifecycle Tiering successfully registered."

🗄️ Phase 5: Elastic File System (EFS) Multi-AZ Provisioning

While S3 stores objects, distributed applications (WordPress, CI/CD runners, machine learning datasets, shared state clusters) require a POSIX-compliant shared file system.

Amazon EFS provides elastic storage that automatically grows and shrinks without provisioning disk sizes.

                  +----------------------------------------------+
                  |         Amazon EFS File System (NFSv4)       |
                  +----------------------+-----------------------+
                                         |
            +----------------------------+----------------------------+
            |                                                         |
            v                                                         v
+-----------------------------+                           +-----------------------------+
|    Availability Zone 1a     |                           |    Availability Zone 1b     |
|  Mount Target: 10.0.1.x     |                           |  Mount Target: 10.0.2.x     |
|  Subnet: Public-Subnet-A    |                           |  Subnet: Public-Subnet-B    |
+-----------------------------+                           +-----------------------------+

1. Discovering Multi-AZ Networking Foundation

If continuing from previous labs, retrieve our VPC and Subnet IDs:

REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"

VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Production-VPC" --region "$REGION" --profile "$PROFILE" --query "Vpcs[0].VpcId" --output text)
PUB_SUB_A=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-A" --region "$REGION" --profile "$PROFILE" --query "Subnets[0].SubnetId" --output text)
PUB_SUB_B=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-B" --region "$REGION" --profile "$PROFILE" --query "Subnets[0].SubnetId" --output text)

2. Creating EFS Dedicated Security Group (Port 2049)

NFS communication operates strictly over TCP port 2049. We create an EFS Security Group authorizing ingress exclusively from our application tier:

EFS_SG_ID=$(aws ec2 create-security-group \
  --group-name "EFS-Shared-SG" \
  --description "Inbound NFS access from App Tier" \
  --vpc-id "$VPC_ID" \
  --tag-specifications 'ResourceType=security-group,Tags=[{Key=Name,Value=EFS-Shared-SG}]' \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "GroupId" \
  --output text)

APP_SG_ID=$(aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$VPC_ID" "Name=group-name,Values=App-Tier-SG" --region "$REGION" --profile "$PROFILE" --query "SecurityGroups[0].GroupId" --output text 2>/dev/null || true)

if [ -n "$APP_SG_ID" ] && [ "$APP_SG_ID" != "None" ]; then
  # Authorize inbound NFS from application instances
  aws ec2 authorize-security-group-ingress \
    --group-id "$EFS_SG_ID" \
    --protocol tcp \
    --port 2049 \
    --source-group "$APP_SG_ID" \
    --region "$REGION" \
    --profile "$PROFILE"
else
  # Fallback to VPC CIDR if App SG is not active
  aws ec2 authorize-security-group-ingress \
    --group-id "$EFS_SG_ID" \
    --protocol tcp \
    --port 2049 \
    --cidr "10.0.0.0/16" \
    --region "$REGION" \
    --profile "$PROFILE"
fi

echo "EFS Security Group Created: $EFS_SG_ID"

3. Provisioning the Encrypted EFS File System

Create an encrypted EFS file system with General Purpose performance mode:

EFS_ID=$(aws efs create-file-system \
  --creation-token "production-shared-efs" \
  --performance-mode "generalPurpose" \
  --throughput-mode "bursting" \
  --encrypted \
  --tags "Key=Name,Value=Production-Shared-Storage" \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "FileSystemId" \
  --output text)

echo "EFS File System Created: $EFS_ID"

# Wait for file system to become available
while true; do
  STATUS=$(aws efs describe-file-systems --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "FileSystems[0].LifeCycleState" --output text)
  if [ "$STATUS" = "available" ]; then
    echo "EFS File System is Available."
    break
  fi
  echo "EFS status: $STATUS (waiting 5s)..."
  sleep 5
done

4. Creating Multi-AZ Mount Targets

For instances in different Availability Zones to access EFS, we must deploy a Mount Target into each respective subnet:

# Mount Target in Zone A
MT_A=$(aws efs create-mount-target \
  --file-system-id "$EFS_ID" \
  --subnet-id "$PUB_SUB_A" \
  --security-groups "$EFS_SG_ID" \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "MountTargetId" \
  --output text)

# Mount Target in Zone B
MT_B=$(aws efs create-mount-target \
  --file-system-id "$EFS_ID" \
  --subnet-id "$PUB_SUB_B" \
  --security-groups "$EFS_SG_ID" \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "MountTargetId" \
  --output text)

echo "Mount Targets Created: Zone A ($MT_A), Zone B ($MT_B)"

# Wait for mount targets to transition to available
echo "Waiting for Mount Targets to initialize..."
sleep 15

🖥️ Phase 6: Mounting EFS on EC2 Instances via CLI

To mount EFS, Linux nodes install the amazon-efs-utils helper package and execute standard NFS mount commands.

The Mount Command Syntax

On any instance in your VPC, mounting the shared storage is as simple as:

# 1. Install EFS helper utilities
# dnf install -y amazon-efs-utils

# 2. Create mount directory
# mkdir -p /mnt/shared-data

# 3. Mount EFS with in-transit encryption (TLS)
# mount -t efs -o tls ${EFS_ID}:/ /mnt/shared-data

# 4. Configure persistence across reboots in /etc/fstab
# echo "${EFS_ID}:/ /mnt/shared-data efs _netdev,tls 0 0" >> /etc/fstab

Verify the active mount targets and network interfaces via CLI:

aws efs describe-mount-targets \
  --file-system-id "$EFS_ID" \
  --region "$REGION" \
  --profile "$PROFILE" \
  --query "MountTargets[*].[MountTargetId,SubnetId,LifeCycleState,IpAddress]" \
  --output table

🧹 Phase 7: 1-Click Hardened Lab Teardown

To ensure complete cost elimination ($0.00 idle spend), we delete the EFS Mount Targets, the EFS File System, clean out all S3 object versions, and delete the KMS key alias:

cat <<'EOF' > destroy-storage-lab.sh
#!/usr/bin/env bash
set -euo pipefail

REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"
ACCOUNT_ID=$(aws sts get-caller-identity --profile "$PROFILE" --query "Account" --output text)
BUCKET_NAME="production-data-vault-${ACCOUNT_ID}-${REGION}"

echo "=== 1. Deleting EFS Mount Targets & File Systems ==="
EFS_ID=$(aws efs describe-file-systems --region "$REGION" --profile "$PROFILE" --query "FileSystems[?Tags[?Value=='Production-Shared-Storage']].FileSystemId|[0]" --output text 2>/dev/null || true)

if [ -n "$EFS_ID" ] && [ "$EFS_ID" != "None" ]; then
  echo "Deleting Mount Targets for $EFS_ID..."
  for mt in $(aws efs describe-mount-targets --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "MountTargets[*].MountTargetId" --output text 2>/dev/null || true); do
    echo "Deleting Mount Target: $mt"
    aws efs delete-mount-target --mount-target-id "$mt" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
  done

  echo "Waiting for Mount Targets to fully delete..."
  while true; do
    REMAINING=$(aws efs describe-mount-targets --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "MountTargets" --output text 2>/dev/null || true)
    if [ -z "$REMAINING" ]; then
      break
    fi
    sleep 5
  done

  echo "Deleting EFS File System: $EFS_ID..."
  aws efs delete-file-system --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi

# Delete EFS Security Group
EFS_SG_ID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=EFS-Shared-SG" --region "$REGION" --profile "$PROFILE" --query "SecurityGroups[0].GroupId" --output text 2>/dev/null || true)
if [ -n "$EFS_SG_ID" ] && [ "$EFS_SG_ID" != "None" ]; then
  echo "Deleting EFS Security Group: $EFS_SG_ID..."
  aws ec2 delete-security-group --group-id "$EFS_SG_ID" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi

echo "=== 2. Purging S3 Bucket & Object Versions ==="
if aws s3api head-bucket --bucket "$BUCKET_NAME" --profile "$PROFILE" 2>/dev/null; then
  echo "Purging all object versions from $BUCKET_NAME..."
  # Delete all object versions
  aws s3api delete-objects --bucket "$BUCKET_NAME" \
    --delete "$(aws s3api list-object-versions --bucket "$BUCKET_NAME" --profile "$PROFILE" \
      --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)" \
    --profile "$PROFILE" 2>/dev/null || true

  # Delete all delete markers
  aws s3api delete-objects --bucket "$BUCKET_NAME" \
    --delete "$(aws s3api list-object-versions --bucket "$BUCKET_NAME" --profile "$PROFILE" \
      --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' --output json)" \
    --profile "$PROFILE" 2>/dev/null || true

  echo "Deleting empty S3 Bucket: $BUCKET_NAME..."
  aws s3api delete-bucket --bucket "$BUCKET_NAME" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi

echo "=== 3. Cleaning up KMS Keys ==="
KMS_KEY_ID=$(aws kms list-aliases --region "$REGION" --profile "$PROFILE" --query "Aliases[?AliasName=='alias/s3-data-vault-key'].TargetKeyId|[0]" --output text 2>/dev/null || true)
if [ -n "$KMS_KEY_ID" ] && [ "$KMS_KEY_ID" != "None" ]; then
  echo "Deleting KMS Alias..."
  aws kms delete-alias --alias-name "alias/s3-data-vault-key" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
  echo "Scheduling KMS Key deletion (7 days window)..."
  aws kms schedule-key-deletion --key-id "$KMS_KEY_ID" --pending-window-in-days 7 --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi

rm -rf ./test-upload destroy-storage-lab.sh
echo "=== Part 12 Teardown Complete! Zero Storage Resources Remaining. ==="
EOF

chmod +x destroy-storage-lab.sh
./destroy-storage-lab.sh

🏆 Key Takeaways & Enterprise Best Practices

  1. Block Public Access as Default: Always activate the 4-point Public Access Block on every S3 bucket upon creation to eliminate accidental data exposure.
  2. Mandate TLS 1.2+ via Policy: Enforce aws:SecureTransport: false denials in S3 bucket policies so unencrypted HTTP requests are rejected at the edge.
  3. Use S3 Bucket Keys: Enable BucketKeyEnabled: true to slash KMS request costs by 99% on high-traffic data pipelines.
  4. Automate Lifecycle Transitions: Let AWS move cold data from Standard to Standard-IA and Glacier automatically to optimize storage costs without manual engineering overhead.
  5. EFS for POSIX Shared State: When multiple compute nodes need concurrent read-write access to a unified file tree, EFS provides a serverless, multi-AZ NFS solution.

🔮 What’s Next in Part 13?

Now that we have mastered object and file storage, what about structured relational data?

In Part 13: Provisioning RDS PostgreSQL Multi-AZ & DynamoDB with CLI Parameter Groups, we dive into managed database infrastructure. We will provision a private Multi-AZ RDS PostgreSQL database cluster, configure fine-tuned parameter groups, manage automated snapshots, connect securely from our compute fleet, and provision a high-throughput serverless DynamoDB table!

Related & Recommended Guides

Continue exploring related systems architectures and engineering field notes.