AWS OpenTofu #105: Data Sources & AWS Discovery

In a classic infrastructure deployment incident, a developer hardcoded an Amazon Machine Image (AMI) ID into their deployment code:

# ❌ The Anti-Pattern: Hardcoded Regional IDs
ami = "ami-0c55b159cbfafe1f0"

The code worked flawlessly in us-east-1 (N. Virginia) for six months.

Then came the mandate: deploy an identical replica in ap-southeast-1 (Singapore) for an enterprise expansion. The pipeline executed and crashed immediately:

Error: InvalidAMIID.NotFound: The image id '[ami-0c55b159cbfafe1f0]' does not exist

Why? Because AMI IDs are strictly regional. The identical Amazon Linux 2023 OS image has a completely different identifier in every AWS region on Earth.

Worse yet, three months later, the original us-east-1 pipeline failed too. Why? AWS had deprecated and deregistered that specific AMI snapshot in favor of a newer release with critical Linux kernel security patches.

Even beyond AMIs:

• How does your code know the 12-digit AWS Account ID it is executing inside without asking the user to type it?
• How does your code dynamically discover which Availability Zones are active in the target region instead of hardcoding us-east-1a (which might not exist or may lack capacity in a different account)?
• How do you construct globally unique names using the current account ID and region dynamically?

The answer is OpenTofu Data Sources.

In this lab, you will learn how to query the live AWS control plane dynamically at plan-time. You will discover environment metadata, fetch active Availability Zones, securely look up official AMIs, and combine data sources with managed resources to eliminate hardcoding forever.

---

🎯 What You Will Learn

AWS Discovery Primitives	OpenTofu Data Blocks	Engineering Standards
• Regional AMI scoping & lifecycle	• data block syntax vs resource blocks	• Preventing AMI supply-chain attacks via owners
• STS caller identity (GetCallerIdentity)	• Querying account, region, & partition	• Dynamic multi-AZ distribution
• Availability Zone status (available vs impaired)	• Filtering AMIs by name, architecture, & type	• Zero-cost environment inspection
• AWS Partition boundaries (aws, aws-cn, gov)	• Consuming data attributes in resources & outputs	• Safe handling of most_recent search filters

---

💰 Estimated Lab Cost

Resource	Configuration	Free Tier Cost	Standard Account (Past Free Tier)
AWS Data Queries (Describe*, Get*)	STS, EC2 Describe, AMI filters	$0.00	$0.00 (Always Free)
Amazon S3 Parameterized Bucket	Temporary test bucket (<15 min)	$0.00	<$0.00001
OpenTofu CLI	Local binary execution	$0.00	$0.00 (Free & Open Source)
Total Estimated Cost	Complete Hands-On Session	$0.00	$0.00

[!NOTE] Zero-Cost Read Operations: OpenTofu data blocks only issue read-only discovery queries against AWS APIs. These metadata requests are permanently free in all AWS accounts regardless of Free Tier status. The temporary S3 bucket created for demonstration costs effectively $0.00 when destroyed after verification.

---

🔍 Resources vs. Data Sources: The Critical Difference

Every OpenTofu block you have written so far has been a resource block. Now we introduce the data block:

+-------------------------------------------------------------------------------+
| RESOURCE BLOCK: MANAGED INFRASTRUCTURE                                        |
|                                                                               |
|   resource "aws_s3_bucket" "vault" { ... }                                    |
|                                                                               |
|   - Purpose:  CREATE, UPDATE, and DESTROY infrastructure                      |
|   - Action:   Issues POST / PUT / DELETE API requests to AWS                  |
|   - State:    Recorded in terraform.tfstate under "mode": "managed"           |
|   - Teardown: Deleted when you run 'tofu destroy'                             |
+-------------------------------------------------------------------------------+
                                      VS
+-------------------------------------------------------------------------------+
| DATA BLOCK: READ-ONLY DISCOVERY                                               |
|                                                                               |
|   data "aws_ami" "amazon_linux_2023" { ... }                                  |
|                                                                               |
|   - Purpose:  QUERY and READ existing cloud data into your code               |
|   - Action:   Issues GET / Describe* API requests to AWS                      |
|   - State:    Cached in terraform.tfstate under "mode": "data"                |
|   - Teardown: NEVER deleted by 'tofu destroy' (it only reads external data)   |
+-------------------------------------------------------------------------------+

---

🏗️ The Data Discovery Architecture

Here is how data flows from AWS APIs into our OpenTofu configuration:

+-------------------------------------------------------------------------------+
| AWS CONTROL PLANE (APIs)                                                      |
|                                                                               |
|   [ STS: GetCallerIdentity ]        ==> Account ID: "123456789012"            |
|   [ EC2: DescribeAvailabilityZones ]==> Active AZs: ["apse1-a", "apse1-b", ..]|
|   [ EC2: DescribeImages ]           ==> Latest AL2023 AMI: "ami-01234567..."  |
+-------------------------------------------------------------------------------+
                                      │
                                      │ Evaluated during 'tofu plan' & 'apply'
                                      ▼
+-------------------------------------------------------------------------------+
| OPENTOFU WORKING DIRECTORY                                                    |
|                                                                               |
|   data.aws_caller_identity.current.account_id                                 |
|   data.aws_availability_zones.available.names[0]                              |
|   data.aws_ami.amazon_linux_2023.id                                           |
|                                     │                                         |
|                                     ▼                                         |
|   [ Dynamically Parameterized Resources & Outputs (Zero Hardcoded Values) ]   |
+-------------------------------------------------------------------------------+

---

📁 Step 1: Set Up Lab Directory

Navigate to your ~/opentofu-labs repository and create the directory for Lab 105:

cd ~/opentofu-labs
mkdir -p lab-105-data-sources
cd lab-105-data-sources

We will structure our project using the standard multi-file layout:

lab-105-data-sources/
├── providers.tf             # Engine and AWS Provider setup
├── data.tf                  # All data source queries
├── main.tf                  # Resources that consume the discovered data
├── variables.tf             # Configurable search parameters
└── outputs.tf               # Exported discoveries

---

⚙️ Step 2: Configure Providers (providers.tf)

Create providers.tf:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project   = "OpenTofu-Level-100"
      ManagedBy = "OpenTofu"
      LabPart   = "105"
    }
  }
}

Create variables.tf:

variable "aws_region" {
  type        = string
  description = "The target AWS region for discovery queries."
  default     = "ap-southeast-1"
}

---

🔎 Step 3: Declare Data Sources (data.tf)

Create data.tf. We will declare three essential categories of data sources:

# ==============================================================================
# 1. ENVIRONMENT METADATA (Account, Region, Partition)
# ==============================================================================

# Queries AWS STS to fetch the current authenticated account and IAM identity
data "aws_caller_identity" "current" {}

# Queries the current active region from the provider
data "aws_region" "current" {}

# Queries the AWS Partition (standard 'aws', China 'aws-cn', or US GovCloud 'aws-us-gov')
data "aws_partition" "current" {}

# ==============================================================================
# 2. GLOBAL NETWORKING DISCOVERY (Active Availability Zones)
# ==============================================================================

# Dynamically discovers all healthy, available AZs in the configured region
data "aws_availability_zones" "available" {
  state = "available"

  # Optional filter: exclude Local Zones or Wavelength Zones, keeping standard AZs
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

# ==============================================================================
# 3. COMPUTE DISCOVERY (Latest Official Amazon Linux 2023 AMI)
# ==============================================================================

# Queries the AWS EC2 AMI catalog for the most recent official Amazon Linux 2023 x86_64 image
data "aws_ami" "amazon_linux_2023" {
  most_recent = true
  owners      = ["amazon"] # CRITICAL: Guarantees we only trust official Amazon images!

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-kernel-6.1-x86_64"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

---

🛡️ Security Deep Dive: Why owners = ["amazon"] is Mandatory

When querying AMIs in AWS, anyone can publish a public AMI to the global community catalog.

If you omit the owners constraint or search by name alone:

# 🚨 DANGEROUS: Supply chain vulnerability!
data "aws_ami" "insecure_lookup" {
  most_recent = true
  # Missing owners = ["amazon"]!

  filter {
    name   = "name"
    values = ["al2023-ami-*"]
  }
}

A malicious actor could name their rogue AMI al2023-ami-2023.99.9-backdoor, publish it publicly, and your OpenTofu code would automatically select the attacker’s infected image because it has the most recent creation timestamp!

[!CAUTION] Always restrict owners when using data "aws_ami".

• For official Amazon Linux / EKS: owners = ["amazon"]
• For official Ubuntu: owners = ["099720109477"] (Canonical)
• For official Red Hat: owners = ["309956199498"] (Red Hat)
• For your company’s private Golden AMIs: owners = ["self"] or your specific AMI builder Account ID.

---

📤 Step 4: Expose Discovered Data (outputs.tf)

Create outputs.tf to inspect the results of our discovery queries:

# Environment Metadata
output "account_id" {
  description = "The 12-digit AWS Account ID executing this code."
  value       = data.aws_caller_identity.current.account_id
}

output "caller_arn" {
  description = "The IAM Identity executing this session."
  value       = data.aws_caller_identity.current.arn
}

output "region_name" {
  description = "The AWS Region where code is running."
  value       = data.aws_region.current.name
}

output "partition" {
  description = "The AWS partition (aws, aws-cn, or aws-us-gov)."
  value       = data.aws_partition.current.partition
}

# Availability Zones
output "available_az_names" {
  description = "List of all active Availability Zone names in this region."
  value       = data.aws_availability_zones.available.names
}

output "available_az_ids" {
  description = "List of physical Availability Zone IDs."
  value       = data.aws_availability_zones.available.zone_ids
}

output "primary_az" {
  description = "The first available AZ chosen dynamically."
  value       = data.aws_availability_zones.available.names[0]
}

# Latest AMI Details
output "latest_ami_id" {
  description = "The dynamic AMI ID for Amazon Linux 2023 in this region."
  value       = data.aws_ami.amazon_linux_2023.id
}

output "latest_ami_name" {
  description = "The descriptive name of the discovered AMI."
  value       = data.aws_ami.amazon_linux_2023.name
}

output "latest_ami_creation_date" {
  description = "When this official image was released by AWS."
  value       = data.aws_ami.amazon_linux_2023.creation_date
}

---

🧪 Step 5: Test Discovery Without Touching Infrastructure

Now let’s run OpenTofu to see data discovery in action:

tofu init
tofu plan

Examine the terminal output:

data.aws_partition.current: Reading...
data.aws_caller_identity.current: Reading...
data.aws_region.current: Reading...
data.aws_availability_zones.available: Reading...
data.aws_ami.amazon_linux_2023: Reading...
data.aws_partition.current: Read complete after 0s
data.aws_region.current: Read complete after 0s
data.aws_caller_identity.current: Read complete after 1s
data.aws_availability_zones.available: Read complete after 1s
data.aws_ami.amazon_linux_2023: Read complete after 1s

Changes to Outputs:
  + account_id               = "123456789012"
  + available_az_ids         = [
      + "apse1-az1",
      + "apse1-az2",
      + "apse1-az3",
    ]
  + available_az_names       = [
      + "ap-southeast-1a",
      + "ap-southeast-1b",
      + "ap-southeast-1c",
    ]
  + caller_arn               = "arn:aws:iam::123456789012:user/admin-prawira"
  + latest_ami_creation_date = "2026-03-04T22:15:30.000Z"
  + latest_ami_id            = "ami-0123456789abcdef0"
  + latest_ami_name          = "al2023-ami-2023.6.20260304.0-kernel-6.1-x86_64"
  + partition                = "aws"
  + primary_az               = "ap-southeast-1a"
  + region_name              = "ap-southeast-1"

You can apply this plan to save these new output values to the OpenTofu state, without changing any real infrastructure.

Notice what OpenTofu reported:

• Read complete: It queried live AWS REST endpoints during the plan phase.
• Zero resources planned: No servers, disks, or buckets are being created.
• Dynamic attributes populated: Real account IDs, real physical AZ IDs, and the exact current AMI ID are available immediately.

Run tofu apply to save these discoveries into your local state:

tofu apply -auto-approve

---

🔄 Step 6: Test Multi-Region Portability

Here is the true test: What happens if we switch regions?

Run tofu plan pointing to us-east-1 (North Virginia):

tofu plan -var="aws_region=us-east-1"

Look at the output:

  ~ latest_ami_id      = "ami-0123456789abcdef0" -> "ami-0c7217cdde317cfec"
  ~ primary_az         = "ap-southeast-1a" -> "us-east-1a"
  ~ region_name        = "ap-southeast-1" -> "us-east-1"
  ~ available_az_names = [
      ~ "ap-southeast-1a",
      ~ "ap-southeast-1b",
      ~ "ap-southeast-1c",
    ] -> [
      + "us-east-1a",
      + "us-east-1b",
      + "us-east-1c",
      + "us-east-1d",
      + "us-east-1e",
      + "us-east-1f",
    ]

Zero code changes required.

• OpenTofu automatically queried the us-east-1 EC2 endpoint.
• It discovered that us-east-1 has 6 Availability Zones instead of Singapore’s 3.
• It fetched the correct regional AMI ID for North Virginia (ami-0c7217cdde317cfec).

---

🪣 Step 7: Combine Data Sources with Resources (main.tf)

Data sources become truly powerful when you feed their discovered attributes directly into managed resources.

Let’s create a managed S3 bucket whose name is dynamically constructed using your real account ID and current region—guaranteeing global uniqueness without any hardcoded names.

Create main.tf:

locals {
  # Construct a globally unique bucket name using discovered AWS account ID & region
  unique_bucket_name = "vault-${data.aws_caller_identity.current.account_id}-${data.aws_region.current.name}-105"
}

# Resource that consumes our discovered data
resource "aws_s3_bucket" "discovered_vault" {
  bucket        = local.unique_bucket_name
  force_destroy = true

  tags = {
    Name       = local.unique_bucket_name
    OwnerArn   = data.aws_caller_identity.current.arn
    TargetAZ   = data.aws_availability_zones.available.names[0]
    Partition  = data.aws_partition.current.partition
    BaseAmiRef = data.aws_ami.amazon_linux_2023.id
  }
}

Add an output for the new bucket in outputs.tf:

output "discovered_bucket_name" {
  description = "Bucket created using dynamic account and regional discovery."
  value       = aws_s3_bucket.discovered_vault.id
}

Apply the changes:

tofu apply -auto-approve

Output:

aws_s3_bucket.discovered_vault: Creating...
aws_s3_bucket.discovered_vault: Creation complete after 2s [id=vault-123456789012-ap-southeast-1-105]

Outputs:
discovered_bucket_name = "vault-123456789012-ap-southeast-1-105"

Notice the tags on this bucket: it has your caller ARN, your target primary AZ, and the latest AMI reference dynamically attached.

---

🧹 Step 8: Clean Teardown (tofu destroy)

Tear down the test bucket:

tofu destroy -auto-approve

Output:

aws_s3_bucket.discovered_vault: Destroying...
aws_s3_bucket.discovered_vault: Destruction complete after 1s

Destroy complete! Resources: 1 destroyed.

Notice that tofu destroy only destroyed the managed S3 bucket. The data sources are not resources, so they were left untouched.

---

🛠️ Common Troubleshooting & Gotchas

1. Error: Your query returned no results. Please change your search criteria and try again.

• Cause: Your filter criteria in data "aws_ami" was too narrow, or the requested architecture is not available in that region.
• Fix: Check AWS CLI to verify image names matching your wildcard:

  aws ec2 describe-images \
    --owners amazon \
    --filters "Name=name,Values=al2023-ami-*-kernel-6.1-x86_64" \
    --query "Images[0].Name"

2. Error: multiple EC2 AMIs matched; use additional constraints to arrive at a single AMI

• Cause: Your filter matched multiple images and you omitted most_recent = true.
• Fix: Always include most_recent = true inside data "aws_ami" when searching with wildcards (*).

3. State Drift on Dynamic AMIs

• Gotcha: If AWS publishes a new release of Amazon Linux next month, running tofu plan might detect a new AMI ID. If your EC2 instance uses that data source directly, OpenTofu may propose destroying and recreating your virtual machine!
• Best Practice: In production environments, teams use data sources to feed AMI IDs into an AMI pipeline or pin AMIs using explicit semantic version tags, avoiding surprise EC2 replacements. We will explore this in Module 03.

---

🏋️ Hands-On Challenge

Before completing this module:

1. Discover Ubuntu 24.04 LTS (Noble Numbat): Add a second AMI data source to data.tf to query the latest official Ubuntu 24.04 LTS server image.
   • Owner ID for Canonical: "099720109477"
   • Filter pattern for name: "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"
2. Add an output in outputs.tf for ubuntu_24_04_ami_id.
3. Run tofu plan and verify that OpenTofu fetches the correct Ubuntu AMI ID alongside Amazon Linux.

---

📖 Summary & What You Learned

With this lab, you have completed Module 02: Storage & Configuration Patterns!

Here is what you have mastered:

• Data vs. Resources: Resources create and mutate cloud infrastructure; data sources perform read-only queries against existing live cloud APIs.
• Environment Autodiscovery: Queried caller identity (data.aws_caller_identity), current region (data.aws_region), and partition (data.aws_partition).
• Dynamic Resiliency: Used data.aws_availability_zones to dynamically discover active data center zones across regions without hardcoded names.
• AMI Discovery & Supply-Chain Security: Queried official Amazon Linux images while enforcing owners = ["amazon"] to protect against malicious community images.
• Multi-Region Portability: Demonstrated how configurations adapt to entirely different regions and accounts with zero code modifications.

---

🚀 What’s Next? (Module 03: Compute & Identity Security)

Now that you have mastered storage, variables, and data discovery, we step into the core of enterprise cloud engineering: Networking and Compute.

In [AWS OpenTofu #106: Launch Your First EC2 Instance], we will:

• Provision a real virtual server (aws_instance) using our discovered Amazon Linux AMI.
• Configure virtual firewall rules using Security Groups (aws_security_group).
• Attach an encrypted Elastic Block Store (EBS) root volume.
• Understand how OpenTofu manages server compute lifecycles.