Performance Testing & Load Engineering

Performance testing answers the question every engineering team avoids until it’s too late: “Will our system handle production load?” The answer is always yes — until Black Friday, a viral tweet, or a marketing campaign drives 10x normal traffic. Performance testing in staging catches bottlenecks before users do.

---

Test Types

Type	Goal	Pattern	Duration
Load test	Behavior at expected peak	Normal traffic volume	10-30 min
Stress test	Find the breaking point	Ramp beyond capacity	Until failure
Soak test	Memory leaks, degradation over time	Steady load, long duration	4-24 hours
Spike test	Sudden traffic surges	1x → 10x → 1x	5-15 min
Breakpoint test	Maximum capacity	Linear ramp	Until SLO breach

---

Load Testing with k6

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const orderLatency = new Trend('order_latency');

export const options = {
  scenarios: {
    // Ramp up to peak load
    peak_load: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 100 },  // Ramp up
        { duration: '5m', target: 100 },  // Steady state
        { duration: '2m', target: 200 },  // Peak
        { duration: '5m', target: 200 },  // Hold peak
        { duration: '2m', target: 0 },    // Ramp down
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<200', 'p(99)<500'],
    errors: ['rate<0.01'],  // < 1% error rate
    order_latency: ['p(95)<300'],
  },
};

export default function () {
  // Browse products
  const products = http.get('https://staging.api.com/products');
  check(products, { 'products 200': (r) => r.status === 200 });
  
  // Place order
  const start = Date.now();
  const order = http.post('https://staging.api.com/orders', 
    JSON.stringify({
      product_id: 'PROD-001',
      quantity: 1,
    }),
    { headers: { 'Content-Type': 'application/json' } }
  );
  orderLatency.add(Date.now() - start);
  
  const success = check(order, { 'order 201': (r) => r.status === 201 });
  errorRate.add(!success);
  
  sleep(1); // Think time between requests
}

---

Performance Budgets

performance_budget:
  api_endpoint:
    "/api/products":
      p50_latency: 50ms
      p95_latency: 150ms
      p99_latency: 300ms
      max_rps: 5000
      error_rate: 0.1%
    
    "/api/orders":
      p50_latency: 100ms
      p95_latency: 250ms
      p99_latency: 500ms
      max_rps: 1000
      error_rate: 0.01%
  
  frontend:
    largest_contentful_paint: 2.5s
    first_input_delay: 100ms
    cumulative_layout_shift: 0.1
    total_bundle_size: 200KB

---

Tool Comparison

Tool	Language	Best For	Distributed
k6	JavaScript	Developer-friendly, CI/CD integration	✅ (k6 Cloud)
Gatling	Scala	JVM applications, complex scenarios	✅
JMeter	Java/GUI	Legacy, GUI-based test design	✅
Locust	Python	Python teams, simple API testing	✅
Artillery	JavaScript/YAML	Quick setup, serverless testing	✅ (Cloud)
Vegeta	Go	Simple HTTP load testing	❌ (single machine)

---

Anti-Patterns

Anti-Pattern	Problem	Fix
Testing in production	Risk of real user impact	Dedicated staging with production-like data
Unrealistic test data	Caches always hot, no variety	Use realistic data volume and distribution
Ignoring think time	Load test sends requests with no delay	Add realistic pauses between requests
Testing single endpoint	Miss bottlenecks in workflows	Test full user flows
No baseline	Can’t tell if performance improved or degraded	Run baseline tests, compare on every change

---

Checklist

• Performance budgets defined per endpoint and page
• Load testing tool selected and integrated into CI/CD
• Baseline performance recorded for comparison
• Realistic test data: volume, variety, distribution
• Test scenarios: load, stress, soak, spike
• Think time and realistic user behavior in tests
• Results analysis: bottleneck identification, capacity planning
• Performance regression: automated detection in CI/CD

---

:::note[Source] This guide is derived from operational intelligence at Garnet Grid Consulting. For performance testing consulting, visit garnetgrid.com. :::