AWS CLI First #12: S3 Mastery & Distributed Files: Bucket Policies, Lifecycle, Encryption & EFS
Master cloud storage from the terminal: enforce S3 bucket policies with TLS and KMS CMK encryption, configure tiered lifecycle rules, optimize multi-threaded CLI sync, and mount Multi-AZ Elastic File System (EFS) shares.
AWS with CLI: The Terminal-First Mastery Path
Part of the CLI-first comprehensive learning path from foundational setup to production cloud architectures.
In modern cloud computing, compute is ephemeral, but data is perpetual. While EC2 compute instances scale up and down or get replaced by Auto Scaling Groups, enterprise applications require robust, highly available storage that spans across multiple servers and regions.
AWS provides two distinct distributed storage primitives:
- Amazon Simple Storage Service (S3): Massive, exabyte-scale object storage designed for 99.999999999% (11 9’s) of durability, accessible via REST APIs.
- Amazon Elastic File System (EFS): Fully managed, serverless, cloud-native NFSv4 shared file system that can be concurrently mounted across hundreds of EC2 instances and containers.
In this guide, we master both storage paradigms purely from the terminal. We will lock down S3 buckets with strict TLS and KMS bucket policies, configure automated tiered lifecycle rules to slash storage bills by 70%, optimize high-throughput multi-part CLI sync, and provision a Multi-AZ EFS shared file system mounted concurrently across our fleet.
💰 Estimated Lab Cost & Resource Architecture
1. Cost Breakdown for this Lab
All resources in this lab are provisioned on-demand in the Asia Pacific (Singapore) ap-southeast-1 region:
| Resource | Quantity | Unit Cost (Hourly) | Estimated Lab Cost (1 Hour) |
|---|---|---|---|
| Amazon S3 Standard Storage | < 100 MB Test Data | $0.025 / GB / month | < $0.001 |
| Amazon S3 API Operations | ~100 PUT/GET calls | $0.005 / 1,000 requests | < $0.001 |
| Elastic File System (EFS) | 1 File System (< 10 MB) | $0.30 / GB / month | < $0.001 |
| AWS KMS Customer Managed Key | 1 Test Key | $1.00 / month (prorated hourly) | ~$0.002 |
EC2 Graviton Test Node (t4g.micro) | 1 Node (Optional for EFS test) | $0.0084 / hour | ~$0.009 |
| Total Estimated Lab Cost | ~$0.013 USD |
2. Architecture Diagram
+----------------------------------------------------------------+
| Enterprise Storage Architecture |
+-------------------------------+--------------------------------+
|
+-----------------------+-----------------------+
| |
v v
+-------------------------------+ +------------------------------+
| Amazon S3 Object Store | | Amazon Elastic File System |
| - Block Public Access (All) | | - Protocol: NFSv4.1 POSIX |
| - SSE-KMS Customer Managed | | - Performance: General Purpose
| - TLS 1.2+ Enforced Policy | | - Multi-AZ Mount Targets |
+---------------+---------------+ +--------------+---------------+
| |
v v
+-------------------------------+ +------------------------------+
| Lifecycle Tiering Pipeline | | Concurrent Client Mount |
| - Day 0: S3 Standard | | - EC2 Worker Node A (AZ 1a) |
| - Day 30: S3 Standard-IA | | - EC2 Worker Node B (AZ 1b) |
| - Day 90: S3 Glacier Flex | | - Shared Directory: /mnt/efs|
| - Day 365: Auto Expiration | | |
+-------------------------------+ +------------------------------+
🛠️ Phase 1: Secure S3 Bucket Provisioning & Block Public Access
Every S3 bucket name is globally unique across all AWS accounts worldwide. By default, misconfigured S3 buckets have historically been the leading cause of data leaks. Today, AWS mandates security-first provisioning.
1. Generating a Globally Unique Bucket Name
Let’s generate a unique bucket identifier combining your AWS Account ID and region:
REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"
ACCOUNT_ID=$(aws sts get-caller-identity --profile "$PROFILE" --query "Account" --output text)
BUCKET_NAME="production-data-vault-${ACCOUNT_ID}-${REGION}"
echo "Target Bucket Name: $BUCKET_NAME"
2. Provisioning the S3 Bucket
In all regions other than us-east-1, the S3 API requires specifying a LocationConstraint:
aws s3api create-bucket \
--bucket "$BUCKET_NAME" \
--create-bucket-configuration LocationConstraint="$REGION" \
--region "$REGION" \
--profile "$PROFILE"
echo "Bucket $BUCKET_NAME successfully created."
3. Enforcing Account-Level & Bucket-Level Block Public Access
Never rely on console defaults. Explicitly enforce the four S3 Public Access Block settings via CLI:
aws s3api put-public-access-block \
--bucket "$BUCKET_NAME" \
--public-access-block-configuration '{
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}' \
--profile "$PROFILE"
echo "Public Access Block enforced (100% Private)."
🔐 Phase 2: Security & Encryption: Enforcing TLS 1.2+ and KMS CMK
A bucket is not enterprise-ready until it enforces Encryption-in-Transit (TLS) and Encryption-at-Rest (KMS) through a declarative Bucket Policy.
1. Creating a Dedicated Customer Managed Key (CMK)
While AWS S3 offers default AES-256 (AES256), enterprise compliance frameworks (PCI-DSS, HIPAA, SOC 2) require Customer Managed Keys (CMK) with audited rotation:
KMS_KEY_ID=$(aws kms create-key \
--description "KMS Key for S3 Data Vault Encryption" \
--tags "TagKey=Environment,TagValue=Production" \
--region "$REGION" \
--profile "$PROFILE" \
--query "KeyMetadata.KeyId" \
--output text)
aws kms create-alias \
--alias-name "alias/s3-data-vault-key" \
--target-key-id "$KMS_KEY_ID" \
--region "$REGION" \
--profile "$PROFILE"
KMS_KEY_ARN=$(aws kms describe-key --key-id "$KMS_KEY_ID" --region "$REGION" --profile "$PROFILE" --query "KeyMetadata.Arn" --output text)
echo "KMS Key ARN: $KMS_KEY_ARN"
2. Setting Default Bucket Encryption to KMS
Configure the bucket to automatically encrypt every uploaded object with our KMS CMK:
aws s3api put-bucket-encryption \
--bucket "$BUCKET_NAME" \
--server-side-encryption-configuration "{
\"Rules\": [
{
\"ApplyServerSideEncryptionByDefault\": {
\"SSEAlgorithm\": \"aws:kms\",
\"KMSMasterKeyID\": \"$KMS_KEY_ARN\"
},
\"BucketKeyEnabled\": true
}
]
}" \
--profile "$PROFILE"
echo "Default SSE-KMS encryption with S3 Bucket Keys enabled."
[!TIP] S3 Bucket Keys (FinOps): Enabling
BucketKeyEnabled: truedecreases KMS API request traffic by up to 99%, reducing KMS operational costs when reading and writing thousands of objects!
3. Crafting a Strict Security Bucket Policy (Deny Insecure HTTP)
We write a bucket policy that denies any request that does not use HTTPS (aws:SecureTransport: false):
cat <<POLICY > bucket-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnforceTLSRequestsOnly",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::$BUCKET_NAME",
"arn:aws:s3:::$BUCKET_NAME/*"
],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
}
]
}
POLICY
aws s3api put-bucket-policy \
--bucket "$BUCKET_NAME" \
--policy file://bucket-policy.json \
--profile "$PROFILE"
rm -f bucket-policy.json
echo "TLS 1.2+ In-Transit Enforcement Policy attached."
⚡ Phase 3: High-Throughput S3 CLI Operations & Versioning
The AWS CLI contains two command families for S3:
aws s3api: Low-level, 1-to-1 mappings with AWS REST API endpoints.aws s3: High-level commands (cp,sync,mv,rm) featuring multi-threaded uploads and automated multipart chunking.
1. Enabling S3 Object Versioning
Versioning protects against accidental deletions and malicious ransomware overrides:
aws s3api put-bucket-versioning \
--bucket "$BUCKET_NAME" \
--versioning-configuration Status=Enabled \
--profile "$PROFILE"
echo "S3 Versioning Enabled."
2. Multi-Threaded Sync with File Filtering
Create a local sample directory with assorted files:
mkdir -p ./test-upload/logs ./test-upload/config
echo "Server Config v1.0" > ./test-upload/config/app.conf
echo "Database Connection" > ./test-upload/config/db.conf
echo "2026-09-08 HTTP 200 OK" > ./test-upload/logs/access.log
echo "Temporary Cache File" > ./test-upload/temp.tmp
# Sync only configuration and log files, excluding temporary files
aws s3 sync ./test-upload/ "s3://${BUCKET_NAME}/backups/" \
--exclude "*.tmp" \
--profile "$PROFILE"
# Verify uploaded files
aws s3 ls "s3://${BUCKET_NAME}/backups/" --recursive --profile "$PROFILE"
3. Testing Version History Retrieval
Modify app.conf and re-upload to witness versioning in action:
echo "Server Config v2.0 (Updated)" > ./test-upload/config/app.conf
aws s3 cp ./test-upload/config/app.conf "s3://${BUCKET_NAME}/backups/config/app.conf" --profile "$PROFILE"
# Inspect versions
aws s3api list-object-versions \
--bucket "$BUCKET_NAME" \
--prefix "backups/config/app.conf" \
--profile "$PROFILE" \
--query "Versions[*].[VersionId,IsLatest,LastModified]" \
--output table
📉 Phase 4: S3 Lifecycle Policies: Automated Tiering (FinOps)
Storing inactive files in S3 Standard costs $0.025/GB/month. In contrast:
- S3 Standard-Infrequent Access (IA): $0.0125/GB/month (50% cheaper)
- S3 Glacier Flexible Retrieval: $0.0036/GB/month (85% cheaper)
- S3 Glacier Deep Archive: $0.00099/GB/month (96% cheaper)
Let’s configure an automated lifecycle policy that shifts objects down the storage hierarchy and purges obsolete versions:
cat <<'EOF' > lifecycle.json
{
"Rules": [
{
"ID": "AutomatedCostOptimizationRule",
"Status": "Enabled",
"Filter": {
"Prefix": "backups/"
},
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
],
"NoncurrentVersionTransitions": [
{
"NoncurrentDays": 30,
"StorageClass": "GLACIER"
}
],
"NoncurrentVersionExpiration": {
"NoncurrentDays": 180
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}
EOF
aws s3api put-bucket-lifecycle-configuration \
--bucket "$BUCKET_NAME" \
--lifecycle-configuration file://lifecycle.json \
--profile "$PROFILE"
rm -f lifecycle.json
echo "Automated Lifecycle Tiering successfully registered."
🗄️ Phase 5: Elastic File System (EFS) Multi-AZ Provisioning
While S3 stores objects, distributed applications (WordPress, CI/CD runners, machine learning datasets, shared state clusters) require a POSIX-compliant shared file system.
Amazon EFS provides elastic storage that automatically grows and shrinks without provisioning disk sizes.
+----------------------------------------------+
| Amazon EFS File System (NFSv4) |
+----------------------+-----------------------+
|
+----------------------------+----------------------------+
| |
v v
+-----------------------------+ +-----------------------------+
| Availability Zone 1a | | Availability Zone 1b |
| Mount Target: 10.0.1.x | | Mount Target: 10.0.2.x |
| Subnet: Public-Subnet-A | | Subnet: Public-Subnet-B |
+-----------------------------+ +-----------------------------+
1. Discovering Multi-AZ Networking Foundation
If continuing from previous labs, retrieve our VPC and Subnet IDs:
REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"
VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Production-VPC" --region "$REGION" --profile "$PROFILE" --query "Vpcs[0].VpcId" --output text)
PUB_SUB_A=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-A" --region "$REGION" --profile "$PROFILE" --query "Subnets[0].SubnetId" --output text)
PUB_SUB_B=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=Public-Subnet-B" --region "$REGION" --profile "$PROFILE" --query "Subnets[0].SubnetId" --output text)
2. Creating EFS Dedicated Security Group (Port 2049)
NFS communication operates strictly over TCP port 2049. We create an EFS Security Group authorizing ingress exclusively from our application tier:
EFS_SG_ID=$(aws ec2 create-security-group \
--group-name "EFS-Shared-SG" \
--description "Inbound NFS access from App Tier" \
--vpc-id "$VPC_ID" \
--tag-specifications 'ResourceType=security-group,Tags=[{Key=Name,Value=EFS-Shared-SG}]' \
--region "$REGION" \
--profile "$PROFILE" \
--query "GroupId" \
--output text)
APP_SG_ID=$(aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$VPC_ID" "Name=group-name,Values=App-Tier-SG" --region "$REGION" --profile "$PROFILE" --query "SecurityGroups[0].GroupId" --output text 2>/dev/null || true)
if [ -n "$APP_SG_ID" ] && [ "$APP_SG_ID" != "None" ]; then
# Authorize inbound NFS from application instances
aws ec2 authorize-security-group-ingress \
--group-id "$EFS_SG_ID" \
--protocol tcp \
--port 2049 \
--source-group "$APP_SG_ID" \
--region "$REGION" \
--profile "$PROFILE"
else
# Fallback to VPC CIDR if App SG is not active
aws ec2 authorize-security-group-ingress \
--group-id "$EFS_SG_ID" \
--protocol tcp \
--port 2049 \
--cidr "10.0.0.0/16" \
--region "$REGION" \
--profile "$PROFILE"
fi
echo "EFS Security Group Created: $EFS_SG_ID"
3. Provisioning the Encrypted EFS File System
Create an encrypted EFS file system with General Purpose performance mode:
EFS_ID=$(aws efs create-file-system \
--creation-token "production-shared-efs" \
--performance-mode "generalPurpose" \
--throughput-mode "bursting" \
--encrypted \
--tags "Key=Name,Value=Production-Shared-Storage" \
--region "$REGION" \
--profile "$PROFILE" \
--query "FileSystemId" \
--output text)
echo "EFS File System Created: $EFS_ID"
# Wait for file system to become available
while true; do
STATUS=$(aws efs describe-file-systems --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "FileSystems[0].LifeCycleState" --output text)
if [ "$STATUS" = "available" ]; then
echo "EFS File System is Available."
break
fi
echo "EFS status: $STATUS (waiting 5s)..."
sleep 5
done
4. Creating Multi-AZ Mount Targets
For instances in different Availability Zones to access EFS, we must deploy a Mount Target into each respective subnet:
# Mount Target in Zone A
MT_A=$(aws efs create-mount-target \
--file-system-id "$EFS_ID" \
--subnet-id "$PUB_SUB_A" \
--security-groups "$EFS_SG_ID" \
--region "$REGION" \
--profile "$PROFILE" \
--query "MountTargetId" \
--output text)
# Mount Target in Zone B
MT_B=$(aws efs create-mount-target \
--file-system-id "$EFS_ID" \
--subnet-id "$PUB_SUB_B" \
--security-groups "$EFS_SG_ID" \
--region "$REGION" \
--profile "$PROFILE" \
--query "MountTargetId" \
--output text)
echo "Mount Targets Created: Zone A ($MT_A), Zone B ($MT_B)"
# Wait for mount targets to transition to available
echo "Waiting for Mount Targets to initialize..."
sleep 15
🖥️ Phase 6: Mounting EFS on EC2 Instances via CLI
To mount EFS, Linux nodes install the amazon-efs-utils helper package and execute standard NFS mount commands.
The Mount Command Syntax
On any instance in your VPC, mounting the shared storage is as simple as:
# 1. Install EFS helper utilities
# dnf install -y amazon-efs-utils
# 2. Create mount directory
# mkdir -p /mnt/shared-data
# 3. Mount EFS with in-transit encryption (TLS)
# mount -t efs -o tls ${EFS_ID}:/ /mnt/shared-data
# 4. Configure persistence across reboots in /etc/fstab
# echo "${EFS_ID}:/ /mnt/shared-data efs _netdev,tls 0 0" >> /etc/fstab
Verify the active mount targets and network interfaces via CLI:
aws efs describe-mount-targets \
--file-system-id "$EFS_ID" \
--region "$REGION" \
--profile "$PROFILE" \
--query "MountTargets[*].[MountTargetId,SubnetId,LifeCycleState,IpAddress]" \
--output table
🧹 Phase 7: 1-Click Hardened Lab Teardown
To ensure complete cost elimination ($0.00 idle spend), we delete the EFS Mount Targets, the EFS File System, clean out all S3 object versions, and delete the KMS key alias:
cat <<'EOF' > destroy-storage-lab.sh
#!/usr/bin/env bash
set -euo pipefail
REGION="ap-southeast-1"
PROFILE="cloud-dev-admin"
ACCOUNT_ID=$(aws sts get-caller-identity --profile "$PROFILE" --query "Account" --output text)
BUCKET_NAME="production-data-vault-${ACCOUNT_ID}-${REGION}"
echo "=== 1. Deleting EFS Mount Targets & File Systems ==="
EFS_ID=$(aws efs describe-file-systems --region "$REGION" --profile "$PROFILE" --query "FileSystems[?Tags[?Value=='Production-Shared-Storage']].FileSystemId|[0]" --output text 2>/dev/null || true)
if [ -n "$EFS_ID" ] && [ "$EFS_ID" != "None" ]; then
echo "Deleting Mount Targets for $EFS_ID..."
for mt in $(aws efs describe-mount-targets --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "MountTargets[*].MountTargetId" --output text 2>/dev/null || true); do
echo "Deleting Mount Target: $mt"
aws efs delete-mount-target --mount-target-id "$mt" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
done
echo "Waiting for Mount Targets to fully delete..."
while true; do
REMAINING=$(aws efs describe-mount-targets --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" --query "MountTargets" --output text 2>/dev/null || true)
if [ -z "$REMAINING" ]; then
break
fi
sleep 5
done
echo "Deleting EFS File System: $EFS_ID..."
aws efs delete-file-system --file-system-id "$EFS_ID" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi
# Delete EFS Security Group
EFS_SG_ID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=EFS-Shared-SG" --region "$REGION" --profile "$PROFILE" --query "SecurityGroups[0].GroupId" --output text 2>/dev/null || true)
if [ -n "$EFS_SG_ID" ] && [ "$EFS_SG_ID" != "None" ]; then
echo "Deleting EFS Security Group: $EFS_SG_ID..."
aws ec2 delete-security-group --group-id "$EFS_SG_ID" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi
echo "=== 2. Purging S3 Bucket & Object Versions ==="
if aws s3api head-bucket --bucket "$BUCKET_NAME" --profile "$PROFILE" 2>/dev/null; then
echo "Purging all object versions from $BUCKET_NAME..."
# Delete all object versions
aws s3api delete-objects --bucket "$BUCKET_NAME" \
--delete "$(aws s3api list-object-versions --bucket "$BUCKET_NAME" --profile "$PROFILE" \
--query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)" \
--profile "$PROFILE" 2>/dev/null || true
# Delete all delete markers
aws s3api delete-objects --bucket "$BUCKET_NAME" \
--delete "$(aws s3api list-object-versions --bucket "$BUCKET_NAME" --profile "$PROFILE" \
--query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' --output json)" \
--profile "$PROFILE" 2>/dev/null || true
echo "Deleting empty S3 Bucket: $BUCKET_NAME..."
aws s3api delete-bucket --bucket "$BUCKET_NAME" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi
echo "=== 3. Cleaning up KMS Keys ==="
KMS_KEY_ID=$(aws kms list-aliases --region "$REGION" --profile "$PROFILE" --query "Aliases[?AliasName=='alias/s3-data-vault-key'].TargetKeyId|[0]" --output text 2>/dev/null || true)
if [ -n "$KMS_KEY_ID" ] && [ "$KMS_KEY_ID" != "None" ]; then
echo "Deleting KMS Alias..."
aws kms delete-alias --alias-name "alias/s3-data-vault-key" --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
echo "Scheduling KMS Key deletion (7 days window)..."
aws kms schedule-key-deletion --key-id "$KMS_KEY_ID" --pending-window-in-days 7 --region "$REGION" --profile "$PROFILE" 2>/dev/null || true
fi
rm -rf ./test-upload destroy-storage-lab.sh
echo "=== Part 12 Teardown Complete! Zero Storage Resources Remaining. ==="
EOF
chmod +x destroy-storage-lab.sh
./destroy-storage-lab.sh
🏆 Key Takeaways & Enterprise Best Practices
- Block Public Access as Default: Always activate the 4-point Public Access Block on every S3 bucket upon creation to eliminate accidental data exposure.
- Mandate TLS 1.2+ via Policy: Enforce
aws:SecureTransport: falsedenials in S3 bucket policies so unencrypted HTTP requests are rejected at the edge. - Use S3 Bucket Keys: Enable
BucketKeyEnabled: trueto slash KMS request costs by 99% on high-traffic data pipelines. - Automate Lifecycle Transitions: Let AWS move cold data from Standard to Standard-IA and Glacier automatically to optimize storage costs without manual engineering overhead.
- EFS for POSIX Shared State: When multiple compute nodes need concurrent read-write access to a unified file tree, EFS provides a serverless, multi-AZ NFS solution.
🔮 What’s Next in Part 13?
Now that we have mastered object and file storage, what about structured relational data?
In Part 13: Provisioning RDS PostgreSQL Multi-AZ & DynamoDB with CLI Parameter Groups, we dive into managed database infrastructure. We will provision a private Multi-AZ RDS PostgreSQL database cluster, configure fine-tuned parameter groups, manage automated snapshots, connect securely from our compute fleet, and provision a high-throughput serverless DynamoDB table!