Secrets Rotation Automation

Manual secret rotation is a ticking bomb. Every API key that is rotated by hand is one forgotten rotation away from an outage or a security incident. Automated rotation ensures credentials are fresh, access is auditable, and expired secrets never cause downtime.

---

Why Automate Rotation

Manual rotation failures:
  - API key expires on a Friday night → production down
  - Database password in 47 config files → missed 3 of them
  - SSL certificate expires → entire domain unreachable
  - Shared credentials → cannot audit who accessed what
  - "We rotated it 18 months ago" → attacker had it for 18 months

Automated rotation:
  - Credentials rotated on schedule (every 30-90 days)
  - All consuming services updated atomically
  - Zero-downtime (dual-credential overlap period)
  - Full audit trail (who requested, when rotated)
  - Alerts if rotation fails

---

Rotation Strategies

Strategy 1: Dual-Credential Rotation
  1. Generate new credential (credential B)
  2. Add credential B to all consumers
  3. Verify all consumers work with B
  4. Remove old credential (credential A)
  5. Total time: minutes (automated)

Strategy 2: Gradual Migration
  1. Generate new credential
  2. Both old and new valid simultaneously
  3. Migrate consumers one by one
  4. Revoke old credential after all migrated
  5. Total time: hours to days

Strategy 3: Dynamic Credentials (best)
  1. Application requests credential from Vault
  2. Vault generates unique credential on-the-fly
  3. Credential has short TTL (1 hour)
  4. Application renews or gets new credential
  5. No static credentials to rotate

---

HashiCorp Vault Integration

# Vault dynamic database credentials
resource "vault_database_secret_backend_connection" "postgres" {
  backend     = "database"
  name        = "postgres"
  plugin_name = "postgresql-database-plugin"
  
  postgresql {
    connection_url = "postgresql://{{username}}:{{password}}@db:5432/myapp"
    username       = "vault_admin"
    password       = var.db_admin_password
  }
}

resource "vault_database_secret_backend_role" "app_readonly" {
  backend   = "database"
  name      = "app-readonly"
  db_name   = vault_database_secret_backend_connection.postgres.name
  
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
    "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";",
  ]
  
  revocation_statements = [
    "DROP ROLE IF EXISTS \"{{name}}\";",
  ]
  
  default_ttl = "1h"   # Credentials expire after 1 hour
  max_ttl     = "24h"  # Maximum 24 hours with renewal
}

# Application fetches dynamic credentials
import hvac

client = hvac.Client(url='https://vault.company.com:8200')

# Get dynamic database credential
creds = client.secrets.database.generate_credentials(
    name='app-readonly',
    mount_point='database'
)

db_username = creds['data']['username']  # Unique per request
db_password = creds['data']['password']  # Short-lived
lease_id = creds['lease_id']            # For renewal

# Connect with dynamic credential
connection = psycopg2.connect(
    host='db.company.com',
    dbname='myapp',
    user=db_username,
    password=db_password,
)

---

Certificate Rotation (cert-manager)

# Automated TLS certificate rotation with cert-manager
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: app-tls
spec:
  secretName: app-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - app.company.com
    - api.company.com
  duration: 2160h    # 90 days
  renewBefore: 720h  # Renew 30 days before expiry
  
  # cert-manager automatically:
  # 1. Issues certificate from Let's Encrypt
  # 2. Stores in Kubernetes secret
  # 3. Renews 30 days before expiry
  # 4. Updates secret (pods pick up new cert)

---

Anti-Patterns

Anti-Pattern	Consequence	Fix
Manual rotation	Forgotten, inconsistent, error-prone	Vault dynamic credentials
Long-lived credentials	Extended exposure window	1-24 hour TTLs
Shared credentials	Cannot audit individual access	Unique per-service credentials
No rotation alerts	Rotation failure undetected	Monitor rotation, alert on failure
Hardcoded secrets in code	Rotation requires code deploy	External secret management

The best secret is one that does not exist as a static artifact. Dynamic, short-lived credentials eliminate the rotation problem entirely.