Immutable Infrastructure

Immutable infrastructure means servers are never modified after deployment. Instead of SSHing into a server to apply a patch, you build a new image with the patch, deploy new servers from that image, and destroy the old ones. This eliminates configuration drift, snowflake servers, and the “works on my machine” problem.

---

Mutable vs Immutable

Mutable Infrastructure (Pets):
  1. Provision server
  2. SSH in, install software
  3. Configure manually
  4. Apply patches over time
  5. Each server becomes unique ("snowflake")
  
  Problem: Server A has patch v1.2, Server B has v1.1
  Problem: Someone SSHed in and changed a config 6 months ago
  Problem: "Just restart it" doesn't work — state is lost

Immutable Infrastructure (Cattle):
  1. Build image (AMI, container image)
  2. Deploy servers from image
  3. Need a change? Build NEW image
  4. Deploy new servers, destroy old ones
  5. Every server is identical
  
  No SSH, no patching, no drift
  If a server misbehaves: destroy it, spin up a new one

---

Golden Image Pipeline

# Packer: Build machine images
# golden-image.pkr.hcl

source "amazon-ebs" "base" {
  source_ami_filter {
    filters = {
      name = "ubuntu/images/*ubuntu-jammy-22.04*"
    }
    owners = ["099720109477"]  # Canonical
  }
  instance_type = "t3.medium"
  ami_name      = "golden-{{timestamp}}"
}

build {
  sources = ["source.amazon-ebs.base"]
  
  # Install software
  provisioner "shell" {
    scripts = [
      "scripts/base-packages.sh",
      "scripts/security-hardening.sh",
      "scripts/monitoring-agent.sh",
      "scripts/application.sh",
    ]
  }
  
  # Validate image
  provisioner "shell" {
    inline = [
      "sudo /opt/app/healthcheck.sh",
      "sudo lynis audit system --no-colors",
    ]
  }
  
  # Tag with version
  post-processor "manifest" {
    output = "manifest.json"
  }
}

# Pipeline:
# 1. Code change → trigger Packer build
# 2. Packer builds new AMI
# 3. Automated tests run against new AMI
# 4. AMI promoted to staging
# 5. Staging tests pass → promote to production
# 6. Blue-green deployment with new AMI
# 7. Old instances terminated

---

Blue-Green Infrastructure

Blue (Current Production):
  ┌─────────────────────────────┐
  │  Load Balancer              │
  │  ↓                          │
  │  ASG: ami-v1.2 (3 instances)│ ← Serving traffic
  └─────────────────────────────┘

Deploy new version:
  ┌─────────────────────────────┐
  │  Load Balancer              │
  │  ↓              ↓           │
  │  Blue: ami-v1.2  Green: ami-v1.3 │
  │  (3 instances)   (3 instances)    │
  └─────────────────────────────┘
  Run health checks on Green

Switch traffic:
  ┌─────────────────────────────┐
  │  Load Balancer              │
  │                 ↓           │
  │  Blue: ami-v1.2  Green: ami-v1.3 │
  │  (draining)      (serving)       │
  └─────────────────────────────┘

Cleanup:
  Terminate Blue instances
  Green becomes new Blue
  
Rollback: Switch LB back to Blue (instant)

---

Anti-Patterns

Anti-Pattern	Consequence	Fix
SSH into production	Configuration drift, snowflakes	No SSH access, immutable images
In-place patches	Server state diverges	New image for every change
No image testing	Broken images reach production	Automated image validation pipeline
Persistent local state	Lost when server replaced	External state (S3, EBS, database)
Manual image building	Slow, inconsistent	Automated Packer pipeline

Immutable infrastructure is not about never changing — it is about changing by replacement instead of mutation. Every server is born from the same image and dies the same way.