Chaos Testing Maturity

Most organizations stop at chaos engineering Level 1: manually running a chaos experiment once, declaring success, and never running it again. Mature chaos engineering is continuous, automated, and integrated into CI/CD. It is not about breaking things — it is about building confidence that your systems handle failure gracefully.

---

Maturity Levels

Level 0: Ad Hoc
  "Let's kill a pod and see what happens"
  No hypothesis, no metrics, no automation
  Occasional, unstructured, heroic efforts

Level 1: Defined
  Hypothesis-driven experiments
  "If we kill the payment service replica, orders should still process"
  Runbooks for experiments, manual execution

Level 2: Repeatable
  Experiments run regularly (monthly GameDays)
  Results tracked and compared over time
  Blast radius controlled and documented

Level 3: Automated
  Chaos experiments in CI/CD pipeline
  Automated canary analysis detects regressions
  Experiments run weekly or daily

Level 4: Continuous
  Chaos runs continuously in production
  Automated response: detect → experiment → validate → report
  Resilience is a measured, tracked metric like availability

---

Hypothesis-Driven Experiments

experiment:
  name: "Payment service instance failure"
  hypothesis: >
    When 1 of 3 payment service replicas is terminated,
    the remaining replicas should absorb traffic with
    no increase in error rate and < 50ms latency increase at p99.
  
  steady_state:
    - metric: error_rate
      value: "< 0.1%"
    - metric: latency_p99
      value: "< 200ms"
    - metric: orders_per_minute
      value: "> 50"
  
  method:
    action: "Kill 1 payment-service pod"
    tool: "chaos-mesh"
    duration: "5 minutes"
    blast_radius: "1 pod in staging"
  
  expected_result:
    - "Remaining pods absorb traffic within 30 seconds"
    - "Error rate stays below 0.5%"
    - "Kubernetes respawns killed pod within 60 seconds"
  
  abort_conditions:
    - "Error rate exceeds 5%"
    - "Latency p99 exceeds 1 second"
    - "More than 10 orders fail"
  
  result:
    status: "PASSED"
    notes: "Traffic redistributed in 8 seconds. No order failures."

---

GameDay Operations

Pre-GameDay (1 week before):
  ☐ Define experiments with hypotheses
  ☐ Identify blast radius and abort conditions
  ☐ Notify stakeholders and on-call teams
  ☐ Prepare monitoring dashboards
  ☐ Review rollback procedures

GameDay Execution:
  ☐ Start with steady-state verification
  ☐ Run experiments in order of increasing blast radius
  ☐ Each experiment: observe → measure → document
  ☐ Abort if conditions met
  ☐ Real-time Slack/video conference for coordination

Post-GameDay:
  ☐ Document all findings
  ☐ Create action items for failures
  ☐ Update runbooks based on learnings
  ☐ Schedule follow-up to verify fixes
  ☐ Share results with engineering org

---

Chaos in CI/CD

# Automated chaos in staging pipeline
stages:
  - name: deploy
    action: Deploy to staging

  - name: warm-up
    action: Wait 5 minutes for traffic baseline

  - name: chaos-experiments
    parallel:
      - experiment: pod-kill
        target: order-service
        count: 1
        verify: error_rate < 0.5%
        
      - experiment: network-latency
        target: payment-service
        delay: 500ms
        verify: upstream_timeout_handling == graceful
        
      - experiment: disk-pressure
        target: database
        fill_percentage: 90%
        verify: alerts_fired AND auto_cleanup_triggered

  - name: validate
    action: Verify all experiments passed

  - name: promote
    condition: all_experiments_passed
    action: Promote to production

---

Anti-Patterns

Anti-Pattern	Consequence	Fix
Break things without hypothesis	No learning, just chaos	Every experiment starts with hypothesis
Only in staging	Staging ≠ production reality	Graduate to production with tight blast radius
No abort conditions	Experiment causes real outage	Automated abort on threshold breach
Run once, never repeat	Confidence decays, regressions appear	Automated, continuous experiments
Skip GameDay postmortem	Same failures repeat	Document findings, track action items

Chaos engineering is not about breaking production. It is about building confidence through controlled experimentation. The goal is to find weaknesses before your customers do.