Secrets Management

Secrets — API keys, database passwords, TLS certificates, encryption keys — are the keys to your kingdom. A leaked database password grants access to all customer data. A leaked Stripe key enables unauthorized charges. Secrets management is not about convenience — it is about preventing the most damaging category of security incidents.

---

Secret Categories

Category	Examples	Rotation Frequency
Infrastructure	DB passwords, SSH keys, TLS certs	90 days
Application	API keys, JWT signing keys	180 days
Third-party	Stripe keys, AWS access keys	90-365 days
Encryption	Data-at-rest keys, KMS keys	365 days
User	Service accounts, OAuth tokens	On-demand

---

Architecture Options

HashiCorp Vault

Application → Vault Agent → Vault Server → Secret Backend
              (sidecar)     (HA cluster)   (KV, DB, PKI)

Features:
  - Dynamic secrets (generated on-demand, auto-expired)
  - Secret leasing and renewal
  - Audit logging for every access
  - Multiple auth methods (Kubernetes, AWS IAM, OIDC)

Cloud-Native

AWS: Secrets Manager + Parameter Store
GCP: Secret Manager
Azure: Key Vault

Features:
  - Managed service, no operational overhead
  - Integrated with IAM
  - Automatic rotation for supported services
  - Versioning and audit trails

---

Secret Injection Patterns

Environment Variables

# Kubernetes: Inject from secret
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      env:
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Sidecar / Init Container

# Vault Agent sidecar injects secrets as files
apiVersion: v1
kind: Pod
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-db: "secret/data/db"
    vault.hashicorp.com/role: "order-service"
spec:
  containers:
    - name: app
      # Secrets available at /vault/secrets/db

External Secrets Operator

# Sync cloud secrets to Kubernetes
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: production/order-service/db
        property: password

---

Secret Rotation

Automated Rotation

def rotate_database_password():
    # 1. Generate new password
    new_password = secrets.token_urlsafe(32)
    
    # 2. Update in database
    db.execute(f"ALTER USER app_user PASSWORD '{new_password}'")
    
    # 3. Update in secret store
    vault.write('secret/data/db', {'password': new_password})
    
    # 4. Application picks up new password
    #    (via sidecar reload or env refresh)
    
    # 5. Verify connectivity with new password
    verify_connection(new_password)
    
    # 6. Log rotation event
    audit_log('password_rotated', service='order-db')

Dual-Password Rotation

For zero-downtime rotation:

1. Service uses Password A
2. Generate Password B, add to database (both valid)
3. Update secret store with Password B
4. Wait for all instances to pick up Password B
5. Revoke Password A

---

Anti-Patterns

Anti-Pattern	Consequence	Fix
Secrets in code/Git	Leaked if repo exposed	Secret scanning + vault
Shared secrets across services	Blast radius of leak is huge	Per-service, per-environment secrets
No rotation policy	Leaked secrets valid forever	Automated rotation on schedule
Secrets in environment variables logged	Secrets in crash dumps, log files	Inject as files, never log env vars
No audit trail	”Who accessed what” unknown	Audit every secret access

Secrets management is the foundation of zero-trust security. If anyone can access any secret, your other security measures are meaningless.