AWS OpenTofu #111: Security Groups & Network Troubleshooting

It is 2:15 AM on a Saturday. Your pager goes off: production web instances are reporting degraded health checks. You try to send an HTTP request:

curl -Iv --connect-timeout 5 http://54.214.120.45

The terminal cursor blinks. Five seconds pass. Then the dreaded output:

* Trying 54.214.120.45:80...
* Connection timed out after 5000 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 5000 milliseconds

No HTTP 500 Internal Server Error. No 403 Forbidden. No TCP RST (Connection Refused). Just absolute, agonizing silence.

Junior engineers react with “guess-and-check ClickOps”: they randomly modify Security Group rules, open 0.0.0.0/0 across all ports, restart instances, and inadvertently introduce massive security vulnerabilities without ever discovering the root cause.

Senior cloud engineers never guess. They use a systematic diagnostic mental model that traces packets through the AWS networking stack layer by layer.

In this lab, you will master the art and science of AWS network troubleshooting. We will explore the mechanics of stateful firewalls vs. stateless access lists, intentionally deploy four common architectural bugs using OpenTofu, observe their exact failure signatures, and fix them with surgical precision.

---

🎯 What You Will Learn

AWS Network Mechanics	Diagnostic Methodology	OpenTofu IaC Hardening
• Stateful Security Groups vs. Stateless NACLs	• The 5-layer AWS packet traversal mental model	• Security Group referencing (SG-to-SG chaining)
• The “Ephemeral Port Trap” (TCP ports 1024–65535)	• Differentiating timeouts (DROP) from refuses (REJECT)	• Subnet IP auto-assignment vs. static ENIs
• VPC Router default routes vs. local routes	• AWS VPC Reachability Analyzer via the AWS CLI	• Route table associations and edge cases
• Public IP translation (1:1 NAT at the IGW boundary)	• In-guest Linux socket analysis (ss, curl, ip route)	• Clean declarative infrastructure recovery

---

💰 Estimated Lab Cost

Resource	Configuration	Free Tier Cost	Standard Account (Past Free Tier)
Amazon EC2 Virtual Machine	1x t3.micro On-Demand (~1 hour)	$0.00	~$0.012 / hour
Amazon EBS Storage	20 GB gp3 Root Volume (~1 hour)	$0.00	~$0.002 / hour ($0.08/GB-mo)
Public IPv4 Address	Auto-assigned IPv4 (~1 hour)	$0.00	$0.005 / hour (AWS IPv4 charge)
Custom VPC, Subnets & IGW	1 VPC, 2 Subnets, 1 IGW, Custom NACL	$0.00	$0.00 (Always 100% Free)
Route Tables & Security Groups	Custom routing & stateful firewall rules	$0.00	$0.00 (Always 100% Free)
AWS Systems Manager (SSM)	Session Manager interactive access	$0.00	$0.00 (Always 100% Free)
Total Estimated Cost	Complete Hands-On Session (~1 hour)	$0.00	~$0.019 (~2 cents total)

[!NOTE] All networking configuration (VPC, Subnets, Route Tables, Internet Gateways, Security Groups, and NACLs) is 100% free forever on AWS. Your only minimal expense during this lab is the running t3.micro instance (~2 cents if past Free Tier). We will tear everything down at the end of the lab.

---

🧠 The AWS Packet Traversal Mental Model

When a network packet travels from the public internet to an application running inside an EC2 instance, it must traverse five distinct layers. If traffic fails, it is failing at exactly one of these checkpoints:

                  PUBLIC INTERNET (Client)
                             │
                             ▼
  [Layer 1: Edge]     Internet Gateway (IGW)
                      • 1:1 NAT mapping (Public IP <-> Private IP)
                      • Requires Elastic IP or Auto-assigned Public IP
                             │
                             ▼
  [Layer 2: Routing]  VPC Route Table
                      • Must have: 0.0.0.0/0 -> igw-xxxxxxxx
                      • Subnet must be explicitly associated
                             │
                             ▼
  [Layer 3: Boundary] Network ACL (NACL) — STATELESS
                      • Evaluates Inbound rules in numerical order
                      • Must also allow Outbound Ephemeral Ports!
                             │
                             ▼
  [Layer 4: ENI]      Security Group (SG) — STATEFUL
                      • Attached directly to the virtual ENI
                      • Automatically tracks established connections
                             │
                             ▼
  [Layer 5: OS Guest] Linux Operating System
                      • Service must be running (e.g. nginx on :80)
                      • Local OS firewall (iptables/ufw) must allow port
                      • Socket listening on 0.0.0.0, not 127.0.0.1

The Triage Rule of Thumb: Timeout vs. Connection Refused

Before typing random commands, read your client’s error output:

1. Connection Timed Out (Silence / Dropped Packets):
   • The packet never reached the destination, or the return packet was blocked.
   • Culprits: Missing Route Table entry (0.0.0.0/0), Security Group ingress rule missing, NACL inbound or outbound rule dropping the packet, or no public IP assigned.
2. Connection Refused (Immediate TCP RST):
   • The packet successfully reached the operating system! However, nothing was listening on that port, or the local OS firewall actively rejected it.
   • Culprits: The web server daemon crashed, is not installed, or is bound only to 127.0.0.1 (localhost) instead of 0.0.0.0 (all interfaces).

---

⚖️ Stateful Firewalls vs. Stateless Network ACLs

The single most common source of confusion in AWS networking is the difference between Security Groups and Network ACLs.

+-------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
| Feature                 | Security Groups (aws_security_group)                      | Network ACLs (aws_network_acl)                            |
+-------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
| Attachment Target       | Elastic Network Interface (ENI) level                     | Subnet boundary level                                     |
| Statefulness            | STATEFUL (Connection Tracking)                            | STATELESS (Zero memory of connections)                    |
| Rule Evaluation         | All rules evaluated; ALLOW rules only                     | Evaluated in numerical order (100, 200...); ALLOW & DENY  |
| Return Traffic Handling | Automatically allowed regardless of outbound rules        | MUST explicitly allow outbound return traffic             |
| Default Behavior        | Denies all inbound, allows all outbound                   | Default NACL allows all; Custom NACL denies all           |
+-------------------------+-----------------------------------------------------------+-----------------------------------------------------------+

The “Ephemeral Port Trap” Explained

When a web browser connects to your server:

1. Request: Browser sends TCP SYN from client port 54321 (random high port) to server port 80.
2. Response: Server sends TCP SYN-ACK from server port 80 back to client port 54321.

• With a Security Group: Because Security Groups are stateful, the hypervisor tracks the connection in its connection table (conntrack). The response packet is automatically permitted out, even if outbound rules specify nothing about port 54321.
• With a Network ACL: Because NACLs are stateless, the return packet is evaluated as a completely independent connection! If the subnet’s outbound NACL does not explicitly allow outbound traffic to ports 1024–65535 (ephemeral ports), the response packet is silently dropped.

The client experiences a Connection Timed Out, even though your Security Group rules are 100% correct!

---

🛠️ Step 1: Project Setup

Let’s create a dedicated workspace directory for this lab following our standardized naming convention:

mkdir -p ~/opentofu-labs/lab-111-network-troubleshooting
cd ~/opentofu-labs/lab-111-network-troubleshooting

We will build five focused files:

1. providers.tf — AWS provider configuration and standard tagging.
2. vpc.tf — Custom VPC, subnets, Internet Gateway, and Route Tables.
3. security.tf — Stateful Security Groups and a custom Network ACL.
4. compute.tf — EC2 instance and SSM IAM role configuration.
5. outputs.tf — Operational endpoints, instance IDs, and verification commands.

---

📄 Step 2: Provider Configuration

Create providers.tf:

# providers.tf
terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

provider "aws" {
  region = "us-east-1"

  default_tags {
    tags = {
      Project     = "OpenTofu-Level-100"
      Lab         = "111-Network-Troubleshooting"
      ManagedBy   = "OpenTofu"
      Environment = "Lab"
    }
  }
}

---

🌐 Step 3: Base Networking Configuration

Create vpc.tf:

# vpc.tf
data "aws_availability_zones" "available" {
  state = "available"
}

# 1. Dedicated VPC
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "lab111-vpc"
  }
}

# 2. Internet Gateway
resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "lab111-igw"
  }
}

# 3. Public Subnet A (AZ-1)
resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "lab111-public-subnet-1"
  }
}

# 4. Custom Route Table for Public Subnet
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gw.id
  }

  tags = {
    Name = "lab111-public-rt"
  }
}

# 5. Route Table Association
resource "aws_route_table_association" "public_1" {
  subnet_id      = aws_subnet.public_1.id
  route_table_id = aws_route_table.public.id
}

---

🛡️ Step 4: Security Groups & Network ACLs

Create security.tf. Notice how we create both an instance Security Group and a custom Network ACL:

# security.tf

# ------------------------------------------------------------------------------
# 1. Web Security Group (Attached to EC2 ENI)
# ------------------------------------------------------------------------------
resource "aws_security_group" "web" {
  name        = "lab111-web-sg"
  description = "Allow inbound HTTP and all outbound traffic"
  vpc_id      = aws_vpc.main.id

  # Inbound HTTP from anywhere
  ingress {
    description = "Allow HTTP from world"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Zero Inbound Port 22 SSH! (Managed strictly via SSM)

  # Outbound rule: Allow all outbound traffic
  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "lab111-web-sg"
  }
}

# ------------------------------------------------------------------------------
# 2. Custom Network ACL (Attached to Subnet)
# ------------------------------------------------------------------------------
resource "aws_network_acl" "public" {
  vpc_id     = aws_vpc.main.id
  subnet_ids = [aws_subnet.public_1.id]

  # Rule 100: Allow Inbound HTTP (port 80)
  ingress {
    protocol   = "tcp"
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 80
    to_port    = 80
  }

  # Rule 110: Allow Inbound HTTPS (port 443 - needed for SSM communication)
  ingress {
    protocol   = "tcp"
    rule_no    = 110
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 443
    to_port    = 443
  }

  # Rule 120: Allow Inbound Ephemeral Ports (for outbound initiated connections)
  ingress {
    protocol   = "tcp"
    rule_no    = 120
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 1024
    to_port    = 65535
  }

  # Rule 100: Allow Outbound All Traffic
  egress {
    protocol   = "all"
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    Name = "lab111-public-nacl"
  }
}

---

💻 Step 5: Compute & IAM Configuration

Create compute.tf. We will bootstrap a working Nginx web server and attach an SSM IAM instance profile so we have out-of-band console access even when inbound web ports fail:

# compute.tf

# 1. Discover latest Amazon Linux 2023 AMI
data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "state"
    values = ["available"]
  }
}

# 2. IAM Role for SSM Session Manager
resource "aws_iam_role" "ec2_ssm" {
  name = "lab111-ec2-ssm-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.ec2_ssm.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_instance_profile" "ec2_ssm" {
  name = "lab111-ec2-ssm-instance-profile"
  role = aws_iam_role.ec2_ssm.name
}

# 3. EC2 Instance inside Custom Public Subnet
resource "aws_instance" "web" {
  ami                  = data.aws_ami.al2023.id
  instance_type        = "t3.micro"
  subnet_id            = aws_subnet.public_1.id
  iam_instance_profile = aws_iam_instance_profile.ec2_ssm.name

  vpc_security_group_ids = [aws_security_group.web.id]

  # Root block device hardening
  root_block_device {
    volume_size           = 20
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }

  # Hardened IMDSv2
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }

  # Cloud-Init User Data: Install and launch Nginx web server
  user_data = <<-EOF
              #!/bin/bash
              dnf update -y
              dnf install -y nginx
              systemctl enable nginx
              systemctl start nginx
              echo "<h1>AWS OpenTofu Lab 111: Network Diagnostics Online</h1><p>VPC: 10.0.0.0/16 | Subnet: 10.0.1.0/24</p>" > /usr/share/nginx/html/index.html
              EOF

  tags = {
    Name = "lab111-web-server"
  }
}

---

📤 Step 6: Outputs

Create outputs.tf:

# outputs.tf
output "vpc_id" {
  description = "The ID of the custom VPC"
  value       = aws_vpc.main.id
}

output "instance_id" {
  description = "The ID of the test EC2 instance"
  value       = aws_instance.web.id
}

output "public_ip" {
  description = "Public IPv4 address of the EC2 instance"
  value       = aws_instance.web.public_ip
}

output "curl_test_command" {
  description = "Command to verify web connectivity"
  value       = "curl -Iv --connect-timeout 5 http://${aws_instance.web.public_ip}"
}

output "ssm_connect_command" {
  description = "Command to open out-of-band SSM session"
  value       = "aws ssm start-session --target ${aws_instance.web.id}"
}

---

🚀 Step 7: Deploy Baseline Infrastructure

Initialize OpenTofu and apply the baseline configuration:

tofu init
tofu apply -auto-approve

Wait roughly 60 to 90 seconds for the EC2 instance to initialize and the user-data script to start Nginx.

Verify Healthy Baseline Connectivity

Run the curl test output:

curl -Iv --connect-timeout 5 http://$(tofu output -raw public_ip)

You should receive a clean HTTP 200 response:

* Connected to 54.x.x.x port 80
> GET / HTTP/1.1
> Host: 54.x.x.x
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.24.0
< Content-Type: text/html
< 
<h1>AWS OpenTofu Lab 111: Network Diagnostics Online</h1>

Now that we have confirmed our baseline is healthy, let’s enter the troubleshooting clinic.

---

🧪 The Troubleshooting Clinic: 4 Real-World Diagnostic Cases

We will intentionally break our infrastructure four different ways. For each case, you will observe the exact failure symptom, run diagnostic commands, identify the root cause, and apply the surgical fix.

---

🚨 Case A: The “Black Hole Route” (Missing Route to IGW)

1. Introduce the Bug

In vpc.tf, comment out or delete the default route to the Internet Gateway:

# vpc.tf
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  # BUG INTRODUCED: Commented out the IGW default route!
  # route {
  #   cidr_block = "0.0.0.0/0"
  #   gateway_id = aws_internet_gateway.gw.id
  # }

  tags = {
    Name = "lab111-public-rt"
  }
}

Apply the broken change:

tofu apply -auto-approve

2. Observe the Symptom

Run your curl command again:

curl -Iv --connect-timeout 5 http://$(tofu output -raw public_ip)

Symptom:

* Trying 54.x.x.x:80...
* Connection timed out after 5000 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 5000 milliseconds

3. Systematic Diagnosis

Why did it time out? Check your instance’s subnet route table using the AWS CLI:

aws ec2 describe-route-tables \
  --route-table-ids $(aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$(tofu output -raw vpc_id)" "Name=tag:Name,Values=lab111-public-rt" --query "RouteTables[0].RouteTableId" --output text) \
  --query "RouteTables[0].Routes" --output table

Output:

------------------------------------------------------
|                   DescribeRoutes                   |
+------------------+---------------------+-----------+
| DestinationCidr  | GatewayId           | State     |
+------------------+---------------------+-----------+
| 10.0.0.0/16      | local               | active    |
+------------------+---------------------+-----------+

Notice what is missing: There is no route for 0.0.0.0/0 targeting an Internet Gateway (igw-xxxx). The VPC router has no instructions on where to send non-VPC packets, so return traffic is discarded into a black hole!

4. The Fix

Uncomment the route block in vpc.tf and run tofu apply -auto-approve. Re-test with curl to verify recovery.

---

🚨 Case B: The “Ephemeral Port Blackout” (Stateless NACL Trap)

1. Introduce the Bug

Open security.tf. In the aws_network_acl.public resource, remove or comment out rule 120 (the ephemeral port ingress):

# security.tf
resource "aws_network_acl" "public" {
  vpc_id     = aws_vpc.main.id
  subnet_ids = [aws_subnet.public_1.id]

  # Rule 100: Allow Inbound HTTP
  ingress {
    protocol   = "tcp"
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 80
    to_port    = 80
  }

  # BUG INTRODUCED: Ephemeral port ingress rule 120 removed!
  # ingress {
  #   protocol   = "tcp"
  #   rule_no    = 120
  #   action     = "allow"
  #   cidr_block = "0.0.0.0/0"
  #   from_port  = 1024
  #   to_port    = 65535
  # }

  egress {
    protocol   = "all"
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }
}

Apply the change:

tofu apply -auto-approve

2. Observe the Symptom

Connect to your instance via SSM Session Manager:

aws ssm start-session --target $(tofu output -raw instance_id)

Inside the instance, attempt to download a package or ping an external site:

curl -Iv --connect-timeout 5 https://aws.amazon.com

Symptom:

* Trying 52.84.x.x:443...
* Connection timed out after 5000 milliseconds

Wait! In security.tf, the Security Group allows all outbound traffic (egress: protocol = -1). The instance route table has 0.0.0.0/0 -> igw. Why can the EC2 instance not reach external HTTPS endpoints?

3. Systematic Diagnosis

• When the EC2 instance connects outbound to https://aws.amazon.com:443, the EC2 instance chooses a random ephemeral port (e.g. 41284) as its source port.
• The outbound packet leaves the subnet through NACL egress rule 100 (allow all).
• AWS sends back the response to destination port 41284.
• The response arrives at the subnet boundary. The NACL evaluates inbound rules:
  • Rule 100 allows port 80 (Mismatch).
  • Rule * (Default Deny) drops the packet!
• Because NACLs are stateless, the return packet is blocked!

4. The Fix

Restore rule 120 in security.tf:

  ingress {
    protocol   = "tcp"
    rule_no    = 120
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 1024
    to_port    = 65535
  }

Run tofu apply -auto-approve and verify that curl https://aws.amazon.com succeeds immediately from inside the instance.

---

🚨 Case C: The “Public Subnet Without Public IP” Trap

1. Introduce the Bug

What happens if an instance is placed in a subnet with a valid Internet Gateway route, but lacks a public IPv4 address?

In vpc.tf, modify aws_subnet.public_1:

# vpc.tf
resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  
  # BUG INTRODUCED: Set to false
  map_public_ip_on_launch = false

  tags = {
    Name = "lab111-public-subnet-1"
  }
}

And in compute.tf, taint or recreate the instance to observe what happens:

tofu taint aws_instance.web
tofu apply -auto-approve

2. Observe the Symptom

Check the outputs:

tofu output public_ip

Output:

""

The public_ip is completely blank!

3. Root Cause Analysis

An Internet Gateway does not route private RFC 1918 IP addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to the public internet. The IGW performs 1:1 stateless Network Address Translation (NAT):

• Outbound: Replaces private IP (10.0.1.45) with the assigned public IP (54.214.x.x).
• Inbound: Replaces the public IP destination with the instance’s private IP.

If an instance has no public IP or Elastic IP attached, the Internet Gateway refuses to forward the traffic.

4. The Fix

Set map_public_ip_on_launch = true in vpc.tf (or specify associate_public_ip_address = true on aws_instance.web), run tofu taint aws_instance.web && tofu apply -auto-approve, and verify that the instance receives a public IPv4 address.

---

🚨 Case D: Security Group Referencing (SG Chaining vs. Hardcoded IPs)

In real-world architectures, you rarely expose EC2 instances directly to the public internet. Instead, traffic passes through an Application Load Balancer (ALB) to the EC2 backend:

[Internet Client] ➔ (Port 443/80) ➔ [Application Load Balancer] ➔ (Port 80) ➔ [EC2 Instances]

The Anti-Pattern: Hardcoding Private IPs

Junior teams often write backend security group rules like this:

# BAD PRACTICE: Brittle and vulnerable!
ingress {
  from_port   = 80
  to_port     = 80
  protocol    = "tcp"
  cidr_blocks = ["10.0.1.50/32", "10.0.2.84/32"] # Hardcoded ALB private IPs!
}

When the ALB scales or an IP changes, backend connectivity shatters.

The Professional Solution: Security Group Referencing

AWS allows you to use a Security Group ID as the source in another Security Group’s ingress rule:

# 1. ALB Security Group
resource "aws_security_group" "alb" {
  name        = "lab111-alb-sg"
  description = "Public ALB Security Group"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# 2. EC2 Backend Security Group (Chained to ALB SG)
resource "aws_security_group" "backend" {
  name        = "lab111-backend-sg"
  description = "Allow traffic ONLY from ALB"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "Only allow HTTP from the ALB security group"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id] # << SG REFERENCING!
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

With Security Group Referencing:

1. Only packets originating from network interfaces attached to aws_security_group.alb are permitted.
2. Even if another instance in the same subnet (10.0.1.0/24) tries to talk to the backend, it is blocked.
3. No IP addresses are hardcoded. As the ALB scales up and down, connectivity remains 100% resilient.

---

🔍 In-Guest Linux Socket & Network Inspection

When you suspect an issue is inside the EC2 instance rather than AWS infrastructure, connect via SSM Session Manager and use these standard Linux diagnostic commands:

aws ssm start-session --target $(tofu output -raw instance_id)

1. Check if the Service is Listening (ss)

sudo ss -tulpn | grep :80

Expected Output:

tcp   LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=2140,fd=6))

[!IMPORTANT] If the local address is 127.0.0.1:80, the daemon is listening only on localhost. External clients will receive Connection Refused! It must be bound to 0.0.0.0:80 (or :::80 for IPv6).

2. Verify Local Linux Gateway Route (ip route)

ip route show

Expected Output:

default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.x metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.x metric 512 

This confirms that the Linux kernel recognizes 10.0.1.1 (the AWS VPC Router) as its default gateway.

3. Test Local Loopback vs. Network Interface

# Test internal socket
curl -I http://127.0.0.1

# Test private ENI IP
curl -I http://$(hostname -I | awk '{print $1}')

If local loopback works but the private ENI fails, a local Linux firewall (iptables, nftables, or firewalld) is dropping the packets.

---

🔬 Next-Level Tooling: AWS VPC Reachability Analyzer

Did you know AWS has a built-in automated reasoning tool that diagnoses network connectivity paths without sending real packets? It is called the VPC Reachability Analyzer.

You can run an analysis from your terminal using the AWS CLI:

# 1. Create a Reachability Analysis Path from the Internet Gateway to your EC2 Instance
PATH_ID=$(aws ec2 create-network-insights-path \
  --source $(aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$(tofu output -raw vpc_id)" --query "InternetGateways[0].InternetGatewayId" --output text) \
  --destination $(tofu output -raw instance_id) \
  --protocol tcp \
  --destination-port 80 \
  --query "NetworkInsightsPath.NetworkInsightsPathId" \
  --output text)

echo "Created Path: $PATH_ID"

# 2. Trigger the analysis
ANALYSIS_ID=$(aws ec2 start-network-insights-analysis \
  --network-insights-path-id $PATH_ID \
  --query "NetworkInsightsAnalysis.NetworkInsightsAnalysisId" \
  --output text)

echo "Running Analysis: $ANALYSIS_ID"

# 3. Wait 10 seconds, then check the result
sleep 10
aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids $ANALYSIS_ID \
  --query "NetworkInsightsAnalyses[0].[Status, NetworkPathFound]" \
  --output text

If the path is reachable, it outputs:

succeeded    True

If it fails, it provides the exact component that blocked the packet (e.g. SECURITY_GROUP_RULE_MISMATCH or NO_ROUTE_TO_DESTINATION).

---

🧹 Step 8: Clean Teardown

Never leave lab resources running. Clean up all resources with OpenTofu:

tofu destroy -auto-approve

Expected Output:

aws_network_acl.public: Destroying...
aws_instance.web: Destroying... [id=i-0123456789abcdef0]
aws_instance.web: Destruction complete after 30s
aws_security_group.web: Destroying...
aws_route_table_association.public_1: Destroying...
aws_iam_instance_profile.ec2_ssm: Destroying...
aws_iam_role_policy_attachment.ssm_core: Destroying...
aws_route_table.public: Destroying...
aws_subnet.public_1: Destroying...
aws_internet_gateway.gw: Destroying...
aws_iam_role.ec2_ssm: Destroying...
aws_vpc.main: Destroying...
Destroy complete! Resources: 10 destroyed.

Verify that zero resources remain in AWS:

aws ec2 describe-instances --filters "Name=tag:Lab,Values=111-Network-Troubleshooting" "Name=instance-state-name,Values=running" --query "Reservations[].Instances[].InstanceId" --output text

(Should return empty).

---

🏋️ Hands-On Challenge

Test your diagnostic intuition with this challenge:

1. In security.tf, add a second Security Group rule that allows inbound traffic on port 8080.
2. Modify the Nginx configuration inside the EC2 instance to listen on port 8080 instead of port 80.
3. Try connecting with curl http://<public_ip>:8080. It will time out!
4. The Challenge: Identify which layer dropped the packet (Hint: check aws_network_acl.public!). Fix the issue using OpenTofu so that port 8080 works seamlessly.

---

📖 Summary & Key Takeaways

In this lab, you elevated your AWS networking skills from basic provisioning to professional diagnostics:

1. The 5-Layer Mental Model: Always trace packets sequentially: Edge (IGW) ➔ Route Tables ➔ Network ACLs ➔ Security Groups ➔ OS Sockets.
2. Stateful vs. Stateless: Security Groups automatically track return connections; Network ACLs require explicit ephemeral port rules (1024–65535).
3. Symptom Clues: Connection Timed Out means packet drop (SG, NACL, Route, or missing Public IP); Connection Refused means packet arrived, but no OS process was listening.
4. Security Group Chaining: Never hardcode private IP addresses; reference security groups (security_groups = [aws_security_group.frontend.id]) for resilient, least-privilege firewalls.
5. Zero-Trust Recovery: With AWS Systems Manager (SSM) configured via IAM, you can diagnose and repair broken EC2 instances even when all public inbound ports are blocked.

---

🚀 What’s Next? (Lab 112)

Congratulations! You have completed Module 04: Networking & Troubleshooting.

Now, we enter Module 05: State Management & Operations.

Up to this point, our state has lived on our local disk in terraform.tfstate. But in real engineering organizations:

• Multiple engineers collaborate on the same infrastructure simultaneously.
• CI/CD pipelines automate deployments.
• Storing state locally risks state locking collisions, state corruption, and accidental leakage of sensitive credentials.

In [AWS OpenTofu #112: Understanding OpenTofu State], we will:

• Inspect the internal structure of the terraform.tfstate JSON file.
• Master tofu state list, tofu state show, and tofu state rm.
• Understand why state files contain plaintext secrets.
• Lay the groundwork for remote state locking and multi-engineer collaboration.