AWS OpenTofu #115: Final Level 100 Challenge

Welcome to the Final Level 100 Challenge.

Throughout this series, you have progressed from understanding raw AWS primitives without IaC (Lab 101) to building production-grade multi-tier environments (Lab 114).

In everyday software engineering, senior platform architects are not handed code tutorials. Instead, you receive an Enterprise Architecture Request for Proposal (RFP) from leadership with business requirements, compliance mandates, and security guardrails.

This lab is your Graduation Exam.

You will be handed the official architecture specification for Project Sentinel—a mission-critical financial transaction ingestion system. You will design the topology, implement the OpenTofu code, validate live connectivity, resolve a real-time infrastructure drift crisis, and perform a zero-waste teardown audit.

---

🎯 The Exam Objectives

Phase	Milestone	Deliverables
Phase 1: Architecture & IaC Implementation	Build from RFP Specification	Multi-file codebase, custom VPC, multi-AZ subnets, S3 security, IAM least privilege, compute tier
Phase 2: Operational Validation	Functional & Security Verification	HTTP endpoint verification, zero-open-port SSM access, in-guest IAM S3 validation
Phase 3: The Drift Crisis	Live Incident Remediation	Detecting out-of-band ClickOps drift via tofu plan, evaluating triage path, reconciling baseline
Phase 4: Zero-Waste Audit	Complete Infrastructure Teardown	tofu destroy, AWS CLI resource enumeration check, cost validation ($0.00 lingering footprint)

---

💰 Estimated Lab Cost

Resource	Configuration	Free Tier Cost	Standard Account (Past Free Tier)
Amazon EC2 Virtual Machines	2x t3.micro On-Demand (~1 hour)	$0.00	~$0.024 / hour
Amazon EBS Storage	2x 20 GB gp3 Root Volumes (~1 hour)	$0.00	~$0.004 / hour
Public IPv4 Addresses	2x Auto-assigned IPv4 (~1 hour)	$0.00	$0.010 / hour ($0.005/ea)
Custom VPC & Subnets	1 VPC, 4 Subnets, 2 Route Tables, 1 IGW	$0.00	$0.00 (Always 100% Free)
Amazon S3 Storage	1 Secured Storage Bucket	$0.00	$0.00 (Negligible < $0.001)
AWS Systems Manager (SSM)	Zero-trust interactive console access	$0.00	$0.00 (Always 100% Free)
Total Estimated Cost	Complete Graduation Exam (~1 hour)	$0.00	~$0.038 (~4 cents total)

---

📋 Client Architecture Specification (The RFP)

Client: FinTech Corp

Project: “Project Sentinel” (Transaction Staging Environment)

“FinTech Corp is deploying Project Sentinel to ingest and validate high-volume financial transaction payloads. Security and compliance auditors require an isolated staging environment with strict network segmentation, zero exposed management ports, encrypted data at rest, and least-privilege identity boundaries.”

Mandatory Requirements:

1. Network Topology

• VPC CIDR: Dedicated VPC with CIDR 10.100.0.0/16.
• DNS: enable_dns_hostnames = true and enable_dns_support = true.
• High Availability: Subnets must span two distinct Availability Zones in us-east-1 (dynamically discovered via data source).
• Public Ingestion Subnets:
  • 10.100.1.0/24 (AZ-1)
  • 10.100.2.0/24 (AZ-2)
  • Must auto-assign public IPv4 addresses (map_public_ip_on_launch = true).
• Private Data Subnets:
  • 10.100.10.0/24 (AZ-1)
  • 10.100.20.0/24 (AZ-2)
• Internet Gateway: Attached to VPC with default route (0.0.0.0/0) configured on the Public Route Table.
• Private Route Table: Local VPC routing only; zero default internet routes.

2. Storage & Compliance (S3)

• Bucket Naming: sentinel-tx-records-<random-hex> (globally unique).
• Public Access: 100% blocked via aws_s3_bucket_public_access_block.
• Encryption: Default AES-256 Server-Side Encryption (aws_s3_bucket_server_side_encryption_configuration).
• Auditability: Object versioning must be Enabled.
• In-Transit Security: Bucket policy must explicitly Deny all non-TLS traffic (aws:SecureTransport = false).
• Clean Destruction: force_destroy = true enabled for automated lab teardown.

3. Compute & Hardening (EC2)

• Fleet: 2x t3.micro compute instances placed in the two public subnets.
• Operating System: Latest official Amazon Linux 2023 AMI discovered via aws_ami data source (owners = ["amazon"]).
• Storage: 20 GB gp3 root EBS volumes with encrypted = true.
• IMDSv2 Enforced: Metadata endpoint enabled with http_tokens = "required".
• Web Ingestion Port: Listening on port 8080 (not standard 80).
• Zero Inbound Management Ports: Port 22 SSH must be strictly closed to the entire universe.
• Administration: Out-of-band management via AWS Systems Manager Session Manager.

4. Least-Privilege Identity (IAM)

• Instances must assume an IAM role with an attached instance profile.
• AWS Managed Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore.
• Custom Scoped Policy: EC2 instances must be allowed to s3:PutObject, s3:GetObject, and s3:ListBucket strictly against the Project Sentinel S3 bucket. Access to any other S3 bucket in the account must be denied.

5. Code Quality & Standards

• All code organized across focused .tf files (providers.tf, variables.tf, locals.tf, vpc.tf, security.tf, storage.tf, iam.tf, compute.tf, outputs.tf).
• Strict use of input variables with custom validation rules.
• Common tags applied globally via default_tags.

---

🛑 Challenge Mode: Build It Yourself First!

Before scrolling down to review the reference implementation, pause and build this environment yourself:

1. Create directory ~/opentofu-labs/lab-115-final-challenge/.
2. Write your .tf configurations based strictly on the RFP requirements above.
3. Run tofu fmt, tofu validate, and tofu apply.
4. Test connectivity using curl http://<ip>:8080 and aws ssm start-session.

Once you have completed your attempt—or if you need an architectural review—proceed to the canonical reference implementation below.

---

🏛️ Reference Implementation Architecture

Here is the production-grade, canonical reference architecture that fulfills 100% of the RFP requirements.

1. providers.tf

# providers.tf
terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = local.common_tags
  }
}

---

2. variables.tf

# variables.tf
variable "aws_region" {
  description = "Target AWS deployment region"
  type        = string
  default     = "us-east-1"
}

variable "environment" {
  description = "Deployment lifecycle environment"
  type        = string
  default     = "staging"

  validation {
    condition     = contains(["staging", "prod"], var.environment)
    error_message = "Environment must be either 'staging' or 'prod'."
  }
}

variable "project_name" {
  description = "Project code identifier"
  type        = string
  default     = "sentinel"
}

variable "vpc_cidr" {
  description = "VPC network IP range"
  type        = string
  default     = "10.100.0.0/16"

  validation {
    condition     = can(cidrnetmask(var.vpc_cidr))
    error_message = "Must be a valid IPv4 CIDR block."
  }
}

variable "api_port" {
  description = "Custom API ingestion port"
  type        = number
  default     = 8080

  validation {
    condition     = var.api_port > 1024 && var.api_port < 65535
    error_message = "API port must be a non-privileged port between 1025 and 65534."
  }
}

---

3. locals.tf

# locals.tf
locals {
  name_prefix = "${var.project_name}-${var.environment}"

  common_tags = {
    Project     = var.project_name
    Environment = var.environment
    ManagedBy   = "OpenTofu"
    Lab         = "115-Final-Level-100-Challenge"
    Compliance  = "SOC2-Staging"
  }
}

---

4. vpc.tf

# vpc.tf
data "aws_availability_zones" "available" {
  state = "available"
}

# 1. Custom VPC
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${local.name_prefix}-vpc"
  }
}

# 2. Internet Gateway
resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${local.name_prefix}-igw"
  }
}

# 3. Public Subnet 1 (AZ-A)
resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.100.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${local.name_prefix}-public-1"
    Tier = "Public"
  }
}

# 4. Public Subnet 2 (AZ-B)
resource "aws_subnet" "public_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.100.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = true

  tags = {
    Name = "${local.name_prefix}-public-2"
    Tier = "Public"
  }
}

# 5. Private Subnet 1 (AZ-A)
resource "aws_subnet" "private_1" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.100.10.0/24"
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name = "${local.name_prefix}-private-1"
    Tier = "Private"
  }
}

# 6. Private Subnet 2 (AZ-B)
resource "aws_subnet" "private_2" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.100.20.0/24"
  availability_zone = data.aws_availability_zones.available.names[1]

  tags = {
    Name = "${local.name_prefix}-private-2"
    Tier = "Private"
  }
}

# 7. Route Tables
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }

  tags = {
    Name = "${local.name_prefix}-public-rt"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${local.name_prefix}-private-rt"
  }
}

# 8. Associations
resource "aws_route_table_association" "pub_1" {
  subnet_id      = aws_subnet.public_1.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "pub_2" {
  subnet_id      = aws_subnet.public_2.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "priv_1" {
  subnet_id      = aws_subnet.private_1.id
  route_table_id = aws_route_table.private.id
}

resource "aws_route_table_association" "priv_2" {
  subnet_id      = aws_subnet.private_2.id
  route_table_id = aws_route_table.private.id
}

---

5. security.tf

# security.tf

# Ingestion Security Group (Port 8080, Zero Port 22 SSH)
resource "aws_security_group" "api_ingestion" {
  name        = "${local.name_prefix}-api-sg"
  description = "FinTech Ingestion API - Port 8080 only"
  vpc_id      = aws_vpc.main.id

  # Inbound Port 8080
  ingress {
    description = "Allow Ingestion API Traffic"
    from_port   = var.api_port
    to_port     = var.api_port
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # STRICT COMPLIANCE: Zero Port 22 SSH Ingress Rules!

  egress {
    description = "Outbound HTTPS and services"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${local.name_prefix}-api-sg"
  }
}

---

6. storage.tf

# storage.tf

resource "random_id" "s3_suffix" {
  byte_length = 4
}

resource "aws_s3_bucket" "tx_records" {
  bucket        = "${local.name_prefix}-records-${random_id.s3_suffix.hex}"
  force_destroy = true

  tags = {
    Name = "${local.name_prefix}-tx-records"
  }
}

resource "aws_s3_bucket_ownership_controls" "tx_records" {
  bucket = aws_s3_bucket.tx_records.id
  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

resource "aws_s3_bucket_public_access_block" "tx_records" {
  bucket = aws_s3_bucket.tx_records.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "tx_records" {
  bucket = aws_s3_bucket.tx_records.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tx_records" {
  bucket = aws_s3_bucket.tx_records.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# TLS Enforced In-Transit
data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid     = "DenyInsecureHTTP"
    effect  = "Deny"
    actions = ["s3:*"]
    resources = [
      aws_s3_bucket.tx_records.arn,
      "${aws_s3_bucket.tx_records.arn}/*"
    ]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

resource "aws_s3_bucket_policy" "tls_only" {
  bucket = aws_s3_bucket.tx_records.id
  policy = data.aws_iam_policy_document.deny_insecure_transport.json
}

---

7. iam.tf

# iam.tf

# 1. EC2 Trust Relationship
data "aws_iam_policy_document" "ec2_trust" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "sentinel_node" {
  name               = "${local.name_prefix}-node-role"
  assume_role_policy = data.aws_iam_policy_document.ec2_trust.json

  tags = {
    Name = "${local.name_prefix}-node-role"
  }
}

# 2. SSM Session Manager Core Attachment
resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.sentinel_node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

# 3. Fine-Grained Least-Privilege S3 Access
data "aws_iam_policy_document" "s3_least_privilege" {
  statement {
    sid       = "AllowAppBucketList"
    effect    = "Allow"
    actions   = ["s3:ListBucket"]
    resources = [aws_s3_bucket.tx_records.arn]
  }

  statement {
    sid     = "AllowAppBucketOperations"
    effect  = "Allow"
    actions = [
      "s3:PutObject",
      "s3:GetObject"
    ]
    resources = ["${aws_s3_bucket.tx_records.arn}/*"]
  }
}

resource "aws_iam_policy" "s3_access" {
  name        = "${local.name_prefix}-s3-policy"
  description = "Scoped read/write to Sentinel transaction storage"
  policy      = data.aws_iam_policy_document.s3_least_privilege.json
}

resource "aws_iam_role_policy_attachment" "s3_attach" {
  role       = aws_iam_role.sentinel_node.name
  policy_arn = aws_iam_policy.s3_access.arn
}

# 4. Instance Profile
resource "aws_iam_instance_profile" "sentinel" {
  name = "${local.name_prefix}-instance-profile"
  role = aws_iam_role.sentinel_node.name
}

---

8. compute.tf

# compute.tf

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "state"
    values = ["available"]
  }
}

locals {
  sentinel_bootstrap = <<-EOF
    #!/bin/bash
    dnf update -y
    dnf install -y nginx awscli
    
    # Configure Nginx to listen on Port 8080
    sed -i 's/listen       80;/listen       8080;/' /etc/nginx/nginx.conf
    sed -i 's/listen       \[::\]:80;/listen       \[::\]:8080;/' /etc/nginx/nginx.conf

    systemctl enable nginx
    systemctl start nginx

    TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
    INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
    AVAIL_ZONE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone)

    echo "Sentinel Transaction Healthcheck: ONLINE at $AVAIL_ZONE" > /usr/share/nginx/html/index.html

    # Upload startup receipt to S3 via IAM Role
    echo "Receipt: Node $INSTANCE_ID booted in $AVAIL_ZONE" > /tmp/receipt.txt
    aws s3 cp /tmp/receipt.txt s3://${aws_s3_bucket.tx_records.id}/startup-$INSTANCE_ID.txt
  EOF
}

# Node A
resource "aws_instance" "node_a" {
  ami                  = data.aws_ami.al2023.id
  instance_type        = "t3.micro"
  subnet_id            = aws_subnet.public_1.id
  iam_instance_profile = aws_iam_instance_profile.sentinel.name

  vpc_security_group_ids = [aws_security_group.api_ingestion.id]

  root_block_device {
    volume_size           = 20
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }

  user_data = local.sentinel_bootstrap

  tags = {
    Name = "${local.name_prefix}-node-a"
  }
}

# Node B
resource "aws_instance" "node_b" {
  ami                  = data.aws_ami.al2023.id
  instance_type        = "t3.micro"
  subnet_id            = aws_subnet.public_2.id
  iam_instance_profile = aws_iam_instance_profile.sentinel.name

  vpc_security_group_ids = [aws_security_group.api_ingestion.id]

  root_block_device {
    volume_size           = 20
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }

  user_data = local.sentinel_bootstrap

  tags = {
    Name = "${local.name_prefix}-node-b"
  }
}

---

9. outputs.tf

# outputs.tf

output "node_a_endpoint" {
  description = "Ingestion API endpoint Node A"
  value       = "http://${aws_instance.node_a.public_ip}:${var.api_port}"
}

output "node_b_endpoint" {
  description = "Ingestion API endpoint Node B"
  value       = "http://${aws_instance.node_b.public_ip}:${var.api_port}"
}

output "s3_bucket_name" {
  description = "Project Sentinel transaction records bucket"
  value       = aws_s3_bucket.tx_records.id
}

output "ssm_command_node_a" {
  description = "SSM Session Manager connection command"
  value       = "aws ssm start-session --target ${aws_instance.node_a.id}"
}

---

⚡ Phase 2: Operational Validation Battery

Apply the reference architecture:

tofu init
tofu apply -auto-approve

Wait roughly 90 seconds for bootstrapping to complete. Then execute the four validation checks:

Test 1: Ingestion API Ingress Verification

curl -Iv --connect-timeout 5 $(tofu output -raw node_a_endpoint)
curl -Iv --connect-timeout 5 $(tofu output -raw node_b_endpoint)

Expected Result: Both nodes return HTTP/1.1 200 OK on custom port 8080.

Test 2: Ingress Hardening Audit

curl -Iv --connect-timeout 3 http://$(tofu state show aws_instance.node_a | grep public_ip | awk '{print $3}' | tr -d '"'):80

Expected Result: Connection times out! Port 80 is strictly blocked by the Security Group.

Test 3: S3 Startup Receipts Verification

aws s3 ls s3://$(tofu output -raw s3_bucket_name)

Expected Result: Both startup-i-xxxx.txt receipts exist, proving compute IAM authorization succeeded.

Test 4: Zero-Port Out-of-Band SSM Access

$(tofu output -raw ssm_command_node_a)

Expected Result: Terminal immediately opens an encrypted interactive shell without SSH keys! Type exit to return.

---

🚨 Phase 3: The Live Drift Crisis

Now, test your Day-2 incident response skills.

1. Simulate an Unauthorized ClickOps Change

A rogue administrator opens SSH port 22 directly in AWS:

SG_ID=$(tofu state show aws_security_group.api_ingestion | grep '^    id ' | awk '{print $3}' | tr -d '"')

aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 22 \
  --cidr 0.0.0.0/0

echo "Rogue Rule Injected into SG: $SG_ID"

2. Detect the Drift with OpenTofu

Run:

tofu plan

OpenTofu immediately flags the intrusion:

Note: Objects have changed outside of OpenTofu
...
  # aws_security_group.api_ingestion will be updated in-place
  ~ resource "aws_security_group" "api_ingestion" {
      ~ ingress = [
          - {
              - cidr_blocks      = [ "0.0.0.0/0" ]
              - from_port        = 22
              - protocol         = "tcp"
              - to_port          = 22
            },
            ...
        ]
    }
Plan: 0 to add, 1 to change, 0 to destroy.

3. Reconcile and Enforce Compliance Baseline

Run:

tofu apply -auto-approve

OpenTofu automatically closes the unauthorized port 22 hole, restoring the SOC2 compliance posture!

---

🧹 Phase 4: Zero-Waste Teardown & Final Audit

Clean up all cloud resources:

tofu destroy -auto-approve

The Post-Teardown Cloud Audit

Verify that your AWS account has a clean, $0.00 lingering footprint:

# 1. Audit Running EC2 Instances
aws ec2 describe-instances \
  --filters "Name=tag:Project,Values=sentinel" "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].InstanceId" --output text

# 2. Audit Custom VPCs
aws ec2 describe-vpcs \
  --filters "Name=tag:Project,Values=sentinel" \
  --query "Vpcs[].VpcId" --output text

# 3. Audit S3 Buckets
aws s3 ls | grep sentinel

All three commands should return empty output. Your infrastructure was provisioned, verified, healed, and dismantled with zero waste.

---

🎓 Level 100 Graduation: What You Have Achieved

Congratulations! You have completed all 15 labs of the AWS + OpenTofu Level 100 Learning Lab Series.

Your New Engineering Superpowers:

1. Infrastructure as Code Mastery: You no longer rely on brittle Web Console ClickOps. You express cloud infrastructure as clean, version-controlled, declarative code.
2. Enterprise AWS Architecture: You understand the AWS Global Infrastructure, multi-AZ VPC design, public/private subnet isolation, and route table mechanics.
3. Zero-Trust Security Engineering: You have permanently replaced vulnerable Port 22 SSH with AWS Systems Manager Session Manager, hardened IMDSv2, encrypted EBS volumes, and scoped IAM policies.
4. Data Protection & Storage: You configure S3 buckets with default encryption, object versioning, public access blocks, and mandatory TLS bucket policies.
5. Day-2 Operations & Governance: You understand the internal structure of terraform.tfstate, diagnose complex network failures layer by layer, remediate configuration drift, and import legacy assets into code.

---

🗺️ What’s Next? (Preview of Level 200)

Now that you have mastered the foundational building blocks, you are ready for Level 200: Intermediate Cloud Engineering:

• Lab 201: Remote State Backends with Amazon S3 & DynamoDB Distributed Locking
• Lab 202: Reusable OpenTofu Modules: Authoring, Versioning & Publishing
• Lab 203: Multi-Tier Web Applications with Application Load Balancers (ALBs) & Target Groups
• Lab 204: Auto Scaling Groups (ASG), Launch Templates & Self-Healing Fleets
• Lab 205: Relational Database Service (RDS) Multi-AZ PostgreSQL & Subnet Groups
• Lab 206: Serverless Microservices with AWS Lambda, API Gateway & DynamoDB
• Lab 207: Automated GitOps CI/CD Pipelines using GitHub Actions & AWS OIDC (Zero Long-Lived Keys!)

Thank you for learning with the AWS + OpenTofu Level 100 Series!