AWS OpenTofu #201: From Lab Configuration to Infrastructure Project

Welcome to Level 200: Intermediate Cloud Architecture & Reusable IaC Platforms.

In Level 100, your primary objective was learning the mechanics: understanding cloud primitives, writing basic HCL resources, inspecting state files, and diagnosing network packets. To keep tutorials accessible, most labs stuffed providers, variables, networking, compute, and outputs into a single file or a loosely organized directory.

In production engineering, that approach immediately collapses:

• A single 800-line main.tf becomes an unreadable wall of code where five engineers suffer Git merge conflicts every sprint.
• Unpinned provider versions silently upgrade on a teammate’s machine or in a CI/CD runner, introducing breaking schema changes that crash your deployment.
• Resources lack systematic naming and tagging taxonomies, making cost allocation and compliance auditing impossible for FinOps and security teams.
• Secrets and environment-specific configs accidentally get committed to Git because the repository lacks strict boundary files.

In this foundational lab of Level 200, you will make the critical mental leap from writing disposable lab scripts to engineering a production-grade, team-ready Infrastructure as Code project.

---

🎯 Learning Objectives

Project Engineering	Version & Dependency Governance	Operational Rigor
• The standard enterprise OpenTofu project structure	• Semantic versioning operators (~>, >=, =)	• Validating refactoring with zero planned drift
• OpenTofu’s compilation model across multiple .tf files	• Dependency lockfile governance (.terraform.lock.hcl)	• Git hygiene with root and project .gitignore
• Decoupling variables, locals, providers, and outputs	• Enterprise tagging taxonomies via default_tags	• Creating actionable terraform.tfvars.example

---

💰 Estimated Lab Cost

Resource	Configuration	Free Tier Cost	Standard Account (Past Free Tier)
Amazon S3 Storage	1 Metadata Storage Bucket	$0.00	$0.00 (Negligible < $0.001)
AWS KMS Customer Managed Key	1 KMS Key for encryption testing (~1 hour)	$0.00	~$0.0014 / hour ($1/mo pro-rated)
Compute & Networking	Zero EC2 instances or NAT Gateways	$0.00	$0.00
Total Estimated Cost	Complete Hands-On Session (~1 hour)	$0.00	< $0.01 (Less than a penny)

[!NOTE] This lab focuses on project engineering, metadata, and structural refactoring. Total cloud expense is effectively $0.00. We will destroy all test assets at the end of the lab.

---

🧠 Core Concept: How OpenTofu Compiles Directories

A common misconception among beginner engineers is that OpenTofu executes files sequentially—for example, thinking providers.tf runs before main.tf, which runs before outputs.tf.

OpenTofu does not care about filenames.

When you execute tofu plan or tofu apply in a directory:

1. OpenTofu reads every single file ending in .tf within that root folder.
2. It merges all HCL declarations into a single, unified abstract syntax tree (AST) in memory.
3. It constructs a Directed Acyclic Graph (DAG) based on resource references and dependencies—not file order.
4. It ignores non-.tf files (unless explicitly parsed as templates).

Directory on Disk:
├── versions.tf   ──┐
├── providers.tf  ──┼──► [ OpenTofu Lexical Parser ] ──► [ Unified AST in Memory ]
├── variables.tf  ──┤                                            │
├── locals.tf     ──┤                                            ▼
├── main.tf       ──┤                                [ Directed Acyclic Graph ]
└── outputs.tf    ──┘                                            │
                                                                 ▼
                                                     [ Execution Plan & Apply ]

Why File Separation Matters for Humans

If OpenTofu merges everything anyway, why not keep everything in one file?

• Cognitive Load: Searching an 800-line main.tf for an output or a variable definition wastes valuable engineering time.
• Git Branching & Merge Conflicts: If three engineers modify variables, security groups, and outputs simultaneously in a single file, Git cannot cleanly resolve the diffs. Separating files by responsibility keeps pull requests small, focused, and mergeable.
• Standardized Navigation: Any cloud architect stepping into your codebase instantly knows where to find inputs (variables.tf), computed attributes (locals.tf), cloud connections (providers.tf), and exported values (outputs.tf).

---

🏗️ The Production-Grade Project Blueprint

Here is the standard enterprise project layout we will build and standardize on throughout Level 200:

aws-platform/
├── versions.tf               # OpenTofu engine & provider version constraints
├── providers.tf              # Provider configurations, default tags & regions
├── variables.tf              # Input variable declarations with validation
├── terraform.tfvars.example  # Safe template illustrating required inputs
├── locals.tf                 # Computed values, naming formulas & common metadata
├── main.tf                   # Primary resource composition and declarations
├── outputs.tf                # Exported attributes, connection strings & ARNs
├── .gitignore                # State, cache, and secret exclusion rules
└── README.md                 # Architectural context, usage, and inputs/outputs table

Let’s dissect each file’s exact responsibility:

+--------------------------+-----------------------------------------------------------------------------------------+
| File                     | Strict Responsibility                                                                   |
+--------------------------+-----------------------------------------------------------------------------------------+
| versions.tf              | Declares required OpenTofu engine version and provider sources with version bounds.    |
| providers.tf             | Configures provider credentials, target AWS regions, and mandatory default_tags.       |
| variables.tf             | Declares external inputs (types, defaults, descriptions, custom validation rules).     |
| terraform.tfvars.example | Committed example showing how to populate input variables without exposing secrets.    |
| locals.tf                | Computes internal, non-configurable derived expressions, prefixes, and tag maps.       |
| main.tf                  | Instantiates core infrastructure resources and orchestrates child modules.             |
| outputs.tf               | Exposes operational endpoints, ARNs, and IDs for consumers and CI/CD pipelines.         |
| .gitignore               | Prevents local state, provider binaries, and sensitive variable files from Git.       |
+--------------------------+-----------------------------------------------------------------------------------------+

---

🛠️ Step 1: Workspace Setup & The Monolithic Anti-Pattern

Let’s create our project directory:

mkdir -p ~/opentofu-labs/lab-201-project-structure
cd ~/opentofu-labs/lab-201-project-structure

To understand why project structure matters, we must first experience the mess.

Imagine inheriting the following monolithic monolith_mess.tf—a typical “ClickOps-to-IaC” script where everything is dumped into one unmaintainable file:

# monolith_mess.tf (THE ANTI-PATTERN)
terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "env" {
  type    = string
  default = "dev"
}

resource "aws_kms_key" "app_key" {
  description             = "KMS Key for ${var.env}"
  deletion_window_in_days = 7
  enable_key_rotation     = true
  tags = {
    Environment = var.env
    Project     = "LegacyApp"
    CreatedBy   = "Dave"
  }
}

resource "aws_s3_bucket" "app_bucket" {
  bucket        = "acme-corp-${var.env}-storage-data-9921"
  force_destroy = true
  tags = {
    Environment = var.env
    Project     = "LegacyApp"
    Department  = "Finance"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "s3_encrypt" {
  bucket = aws_s3_bucket.app_bucket.id
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.app_key.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

output "bucket_arn" {
  value = aws_s3_bucket.app_bucket.arn
}

Write this monolithic configuration to disk so we can execute our refactoring workflow:

cat <<'EOF' > monolith_mess.tf
terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "env" {
  type    = string
  default = "dev"
}

resource "aws_kms_key" "app_key" {
  description             = "KMS Key for app encryption"
  deletion_window_in_days = 7
  enable_key_rotation     = true
  tags = {
    Environment = "dev"
    Project     = "LegacyApp"
  }
}

resource "aws_s3_bucket" "app_bucket" {
  bucket_prefix = "acme-corp-dev-storage-"
  force_destroy = true
  tags = {
    Environment = "dev"
    Project     = "LegacyApp"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "s3_encrypt" {
  bucket = aws_s3_bucket.app_bucket.id
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.app_key.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

output "bucket_arn" {
  value = aws_s3_bucket.app_bucket.arn
}
EOF

Initialize and deploy this baseline configuration:

tofu init
tofu apply -auto-approve

Verify that the resources exist in state:

tofu state list

Output:

aws_kms_key.app_key
aws_s3_bucket.app_bucket
aws_s3_bucket_server_side_encryption_configuration.s3_encrypt

Now, our mission begins: Refactor this entire project into a clean, modular structure without modifying or replacing a single live AWS resource.

---

📦 Step 2: Extracting Version & Provider Constraints

The first step in refactoring is isolating engine and provider requirements.

Understanding Semantic Version Operators

In production, unpinned or loosely pinned provider versions are dangerous. If AWS releases a major provider version (v6.0.0) containing breaking argument deprecations, an unpinned version = ">= 5.0" will download the new version and break your pipeline.

Use semantic version constraints deliberately:

Operator    Syntax             Meaning
──────────────────────────────────────────────────────────────────────────
Exact       version = "5.50.0" Strictly download only 5.50.0 (Extremely rigid)
Minimum     version = ">= 5.50" Download 5.50.0 or any newer version (Risk of v6 breakage)
Pessimistic version = "~> 5.50" Allows 5.50.0 up to < 6.0.0 (RECOMMENDED FOR PROVIDERS)
Patch-only  version = "~> 5.50.0" Allows 5.50.0 up to < 5.51.0 (Strict bugfixes only)

Create versions.tf:

# versions.tf
terraform {
  # Minimum OpenTofu engine version
  required_version = ">= 1.8.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

The Role of .terraform.lock.hcl

When you run tofu init, OpenTofu generates a dependency lockfile named .terraform.lock.hcl. It records the exact version of the provider downloaded and cryptographic checksums for multiple operating system architectures (Linux x86, macOS ARM, Windows).

[!IMPORTANT] Always commit .terraform.lock.hcl to Git! This guarantees that every engineer and every CI/CD runner compiles infrastructure against the exact same provider binary.

---

🌐 Step 3: Centralizing Provider Configuration & Default Tags

Next, create providers.tf.

Notice the critical enterprise feature: default_tags. In Level 100, we manually repeated tags on every individual resource. In Level 200, we inject governance tags automatically into every AWS resource that supports tagging:

# providers.tf
provider "aws" {
  region = var.aws_region

  default_tags {
    tags = local.common_tags
  }
}

Any tag specified in local.common_tags is automatically propagated to all VPCs, S3 buckets, EC2 instances, KMS keys, and databases without duplicating a single line of HCL.

---

🎛️ Step 4: Strict Input Validation in variables.tf

Never accept raw, unvalidated strings in production code. Use typed variables, descriptive documentation, and custom validation blocks to reject invalid configurations at compile time.

Create variables.tf:

# variables.tf

variable "aws_region" {
  description = "Target AWS region for resource deployment"
  type        = string
  default     = "us-east-1"
}

variable "environment" {
  description = "Deployment environment tier (dev, staging, prod)"
  type        = string
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "The environment variable must be one of: dev, staging, prod."
  }
}

variable "project_name" {
  description = "Standardized project code name"
  type        = string
  default     = "platform-core"

  validation {
    condition     = can(regex("^[a-z0-9-]+$", var.project_name))
    error_message = "Project name must contain only lowercase alphanumeric characters and hyphens."
  }
}

variable "owner_email" {
  description = "Contact email for resource ownership and cost attribution"
  type        = string
  default     = "[email protected]"

  validation {
    condition     = can(regex("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$", var.owner_email))
    error_message = "Must provide a valid email address for cost attribution."
  }
}

Now create a template for collaborators: terraform.tfvars.example:

# terraform.tfvars.example
# Copy this file to terraform.tfvars and customize for your local workspace.
# DO NOT COMMIT terraform.tfvars to Git if it contains sensitive values!

aws_region   = "us-east-1"
environment  = "dev"
project_name = "platform-core"
owner_email  = "[email protected]"

---

🏷️ Step 5: Establishing Derived Metadata in locals.tf

Input variables should capture raw facts (dev, platform-core).

Local values (locals.tf) should compute derived business logic, standardized naming conventions, and shared tag maps.

Create locals.tf:

# locals.tf

locals {
  # Standardized resource naming prefix: e.g., "platform-core-dev"
  name_prefix = "${var.project_name}-${var.environment}"

  # Centralized enterprise governance tags
  common_tags = {
    Project     = var.project_name
    Environment = var.environment
    ManagedBy   = "OpenTofu"
    Owner       = var.owner_email
    Repository  = "https://github.com/organization/aws-platform"
    CostCenter  = "Core-Infrastructure-9901"
  }
}

Notice how clean this is: if the company changes its CostCenter code or adds a compliance tag, you update locals.tf in one spot, and your entire infrastructure inherits the change.

---

🏛️ Step 6: Pure Resource Declarations in main.tf

Now that versions, providers, variables, and locals are cleanly decoupled, main.tf contains only clean resource definitions.

Notice how resource arguments reference local.name_prefix and variables:

Create main.tf:

# main.tf

# 1. Customer Managed KMS Key
resource "aws_kms_key" "app_key" {
  description             = "KMS Master Key for ${local.name_prefix}"
  deletion_window_in_days = 7
  enable_key_rotation     = true

  tags = {
    Name = "${local.name_prefix}-kms-key"
  }
}

# 2. Secure Application Storage Bucket
resource "aws_s3_bucket" "app_bucket" {
  bucket_prefix = "${local.name_prefix}-storage-"
  force_destroy = true

  tags = {
    Name = "${local.name_prefix}-storage-bucket"
  }
}

# 3. Default Server-Side Encryption using KMS
resource "aws_s3_bucket_server_side_encryption_configuration" "s3_encrypt" {
  bucket = aws_s3_bucket.app_bucket.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.app_key.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

---

📤 Step 7: Structured Outputs in outputs.tf

Create outputs.tf. Always include descriptive text so engineers inspecting outputs understand what the value represents:

# outputs.tf

output "kms_key_arn" {
  description = "Amazon Resource Name of the application KMS key"
  value       = aws_kms_key.app_key.arn
}

output "s3_bucket_name" {
  description = "Unique name of the provisioned application S3 bucket"
  value       = aws_s3_bucket.app_bucket.id
}

output "s3_bucket_arn" {
  description = "ARN of the provisioned application S3 bucket"
  value       = aws_s3_bucket.app_bucket.arn
}

---

🛡️ Step 8: Git Hygiene & The .gitignore Standard

Never allow local state, provider binaries, or secret files to contaminate source control.

Create .gitignore:

# Local OpenTofu cache directories
.terraform/
.terraform.lock.hcl.*

# Local state files and backups
*.tfstate
*.tfstate.*
*.tfstate.backup

# Crash log files
crash.log
crash.*.log

# CLI configuration and variable files containing sensitive data
*.tfvars
*.tfvars.json
!terraform.tfvars.example

# Override files
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# CLI plan files
*.tfplan

[!NOTE] Notice the exception rule: !terraform.tfvars.example. We ignore all *.tfvars files because developers might accidentally put live passwords in them, but we explicitly allow the .example template through.

---

🧪 Step 9: Zero-Drift Refactoring Validation

Now comes the ultimate test of our refactoring:

1. We originally deployed our infrastructure using monolith_mess.tf.
2. We have now split our configuration into versions.tf, providers.tf, variables.tf, locals.tf, main.tf, outputs.tf, and .gitignore.

Now, delete the old monolithic mess:

rm monolith_mess.tf

Look at your directory layout now:

ls -la

Output:

.gitignore
locals.tf
main.tf
outputs.tf
providers.tf
terraform.tfstate
terraform.tfvars.example
variables.tf
versions.tf

1. Check Formatting & Validation

tofu fmt -check
tofu validate

Output:

Success! The configuration is valid.

2. Run the Plan Test (The Moment of Truth!)

tofu plan

Examine the output carefully:

aws_kms_key.app_key: Refreshing state... [id=12345678-1234-1234-1234-123456789012]
aws_s3_bucket.app_bucket: Refreshing state... [id=acme-corp-dev-storage-xxxxxxxx]
aws_s3_bucket_server_side_encryption_configuration.s3_encrypt: Refreshing state...

OpenTofu will perform the following actions:

  # aws_kms_key.app_key will be updated in-place
  ~ resource "aws_kms_key" "app_key" {
        id                      = "12345678-1234-1234-1234-123456789012"
      ~ tags                    = {
          + "CostCenter"  = "Core-Infrastructure-9901"
          + "ManagedBy"   = "OpenTofu"
          + "Name"        = "platform-core-dev-kms-key"
          + "Owner"       = "[email protected]"
          + "Repository"  = "https://github.com/organization/aws-platform"
            # (2 unchanged elements hidden)
        }
        # (7 unchanged attributes hidden)
    }

  # aws_s3_bucket.app_bucket will be updated in-place
  ~ resource "aws_s3_bucket" "app_bucket" {
        id                      = "acme-corp-dev-storage-xxxxxxxx"
      ~ tags                    = {
          + "CostCenter"  = "Core-Infrastructure-9901"
          + "ManagedBy"   = "OpenTofu"
          + "Name"        = "platform-core-dev-storage-bucket"
          + "Owner"       = "[email protected]"
          + "Repository"  = "https://github.com/organization/aws-platform"
            # (2 unchanged elements hidden)
        }
        # (9 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Look at what OpenTofu is proposing:

• 0 to add, 0 to destroy: OpenTofu understands that every single resource in main.tf is the exact same live AWS asset originally created in monolith_mess.tf.
• 2 to change (in-place): The only modification is applying our standardized enterprise governance tags!

Apply the tag update:

tofu apply -auto-approve

Now run tofu plan again:

tofu plan

Output:

No changes. Your infrastructure matches the configuration.

You have executed a flawless, zero-downtime structural refactoring.

---

🛠️ Common Troubleshooting & Gotchas

1. Error: Duplicate resource "aws_s3_bucket" "app_bucket"

• Cause: Leaving the old monolith_mess.tf file in the directory alongside the new main.tf.
• Rule: OpenTofu merges all .tf files in the directory. You cannot have two resource blocks with the exact same type and name across any file in the same folder.

2. Error: Cycle: local.name_prefix -> var.project_name -> local.common_tags

• Cause: Creating circular references between locals and variables.
• Rule: Variables cannot reference locals. Locals can reference variables and other locals, but cannot create closed loops.

3. CI/CD Fails with Provider Checksum Mismatch

• Cause: Developing on macOS ARM64 (Apple Silicon) and pushing to a Linux x86_64 CI/CD runner without updating the lockfile.
• Fix: Run tofu providers lock -platform=linux_amd64 -platform=darwin_arm64 to cache cross-platform checksums in .terraform.lock.hcl.

---

🧹 Step 10: Clean Teardown

Clean up all resources deployed during this lab:

tofu destroy -auto-approve

Output:

aws_s3_bucket_server_side_encryption_configuration.s3_encrypt: Destroying...
aws_s3_bucket.app_bucket: Destroying...
aws_kms_key.app_key: Destroying...
Destroy complete! Resources: 3 destroyed.

Verify that state is empty:

tofu state list

(Returns empty).

---

🏋️ Hands-On Challenge

Before proceeding to Lab 202:

1. In variables.tf, add a new variable named retention_days of type number with a default of 30.
2. Add a custom validation block requiring retention_days to be either 30, 60, or 90.
3. In locals.tf, create a derived tag "DataRetention" = "${var.retention_days}-days".
4. In terraform.tfvars.example, document this new variable.
5. Run tofu validate to prove your validation syntax is flawless.

---

📖 Summary & Key Takeaways

In this lab, you upgraded your Infrastructure as Code practice from script-writing to project engineering:

1. Compilation Model: OpenTofu evaluates all .tf files in a folder into a single lexical scope. File separation exists for human readability, code reviews, and Git branching.
2. Standard File Structure:
   • versions.tf pins engine and provider versions.
   • providers.tf injects regional configuration and default_tags.
   • variables.tf enforces typed, validated inputs.
   • locals.tf computes business logic, prefixes, and metadata.
   • main.tf instantiates resources cleanly.
   • outputs.tf reports structured connection endpoints.
3. Pessimistic Pinning (~>): Always use pessimistic version constraints for providers (version = "~> 5.50") and commit .terraform.lock.hcl to ensure cross-team build reproducibility.
4. Zero-Drift Refactoring: Refactoring code across separate files does not alter resource identity in terraform.tfstate.

---

🚀 What’s Next? (Lab 202)

Now that we know how to structure a clean root project, how do we package infrastructure so it can be reused across dozens of different applications and teams?

In [AWS OpenTofu #202: Your First Reusable OpenTofu Module], we will:

• Deconstruct the boundary between Root Modules and Child Modules.
• Build an encapsulated, reusable VPC networking child module.
• Design strict, clean input and output contracts.
• Call the module locally using source = "./modules/vpc".