Secrets Rotation & Credential Lifecycle

Static credentials are the #1 attack vector in cloud breaches. AWS reports that leaked access keys cause the majority of their customer security incidents. The fix isn’t just storing secrets in a vault — it’s automating the entire lifecycle: creation, rotation, distribution, and revocation — so no human ever touches a credential.

---

Secret Lifecycle

CREATE → DISTRIBUTE → USE → ROTATE → REVOKE

1. CREATE
   Generated programmatically (never chosen by humans)
   Minimum entropy requirements met
   
2. DISTRIBUTE
   Injected at runtime (env vars, mounted files)
   Never in code, config files, or container images
   
3. USE
   Application retrieves from vault at startup
   Short TTL, cached briefly
   
4. ROTATE
   Automated on schedule (30-90 days)
   Zero-downtime: new credential active before old expires
   
5. REVOKE
   Immediate revocation on compromise
   Automated offboarding revokes all user credentials

---

Zero-Downtime Rotation Pattern

Time 0: 
  Active: Credential A (valid)
  
Time 1 (rotation start):
  Active: Credential A (valid)
  New:    Credential B (created, deployed to app)
  
Time 2 (app updated):
  Active: Credential A (valid, still accepted)
  Active: Credential B (valid, app now using this)
  
Time 3 (grace period):
  Active: Credential B (primary)
  Deprecated: Credential A (still valid for stragglers)
  
Time 4 (rotation complete):
  Active: Credential B (only valid credential)
  Revoked: Credential A (revoked)

---

Rotation Schedules

Secret Type	Rotation Period	Automation
Database passwords	30 days	Vault dynamic secrets
API keys	90 days	Automated rotation + deploy
TLS certificates	90 days (or shorter)	cert-manager auto-renewal
SSH keys	90 days	Certificate-based (no static keys)
Service account tokens	Per-request	Workload identity (no rotation needed)
Encryption keys	365 days	KMS automatic rotation

---

Vault Dynamic Secrets

# Instead of static database credentials,
# Vault generates unique, short-lived credentials per request

import hvac

client = hvac.Client(url='https://vault.internal:8200')

# Request temporary database credentials (TTL: 1 hour)
creds = client.secrets.database.generate_credentials(
    name='order-service-role'
)

db_username = creds['data']['username']  # v-order-svc-abc123
db_password = creds['data']['password']  # auto-generated
ttl = creds['lease_duration']            # 3600 seconds

# Connect to database with temporary credentials
conn = psycopg2.connect(
    host='db.internal',
    user=db_username,
    password=db_password,
    dbname='orders'
)

# When TTL expires, Vault automatically revokes the credentials

---

Leak Detection

Tool	What It Scans	When
Gitleaks	Git history	Pre-commit hook
TruffleHog	Git repos, deep history scan	CI/CD pipeline
AWS GuardDuty	API calls from leaked AWS keys	Continuous
GitHub Secret Scanning	Public repos	Continuous (auto-alert)
GitLab Secret Detection	Commits, MRs	CI/CD pipeline

---

Anti-Patterns

Anti-Pattern	Problem	Fix
Secrets in environment variables (hardcoded)	Visible in process listing, crash dumps	Vault injection, mounted files
Never rotating credentials	Leaked credential valid forever	Automated rotation on schedule
Manual rotation	Human error, rotation skipped	Fully automated rotation
Same credential everywhere	Compromise one = compromise all	Unique credentials per service/environment
No leak detection	Credentials in Git for months undetected	Pre-commit scanning + continuous monitoring

---

Checklist

• Secrets vault deployed (HashiCorp Vault, AWS Secrets Manager)
• Dynamic secrets for databases (no static credentials)
• Rotation automated for all credential types
• Zero-downtime rotation: dual-credential overlap
• Leak detection: pre-commit hooks + CI scanning
• Workload identity for cloud services (no static API keys)
• TLS certificates auto-renewed (cert-manager)
• Offboarding: automated credential revocation
• Emergency rotation: runbook for compromised credentials

---

:::note[Source] This guide is derived from operational intelligence at Garnet Grid Consulting. For secrets management consulting, visit garnetgrid.com. :::