Load Testing Guide - UltraBalancer

Overview

Load test UltraBalancer to validate performance, identify bottlenecks, and ensure it can handle your production traffic patterns.

Apache Bench

Simple HTTP benchmarking tool

wrk

Modern HTTP benchmarking tool

hey

Go-based load generator

k6

Modern load testing for DevOps

Apache Bench (ab)

Installation

Ubuntu/Debian
macOS
CentOS/RHEL

sudo apt install apache2-utils

# Pre-installed
ab -V

sudo yum install httpd-tools

Basic Usage

# 1000 requests, 10 concurrent
ab -n 1000 -c 10 http://localhost:8080/

# With keep-alive
ab -n 10000 -c 100 -k http://localhost:8080/

# POST requests with JSON
ab -n 1000 -c 50 -p data.json -T application/json http://localhost:8080/api

# Custom headers
ab -n 1000 -c 50 -H "Authorization: Bearer token" http://localhost:8080/

Output Analysis

Benchmarking localhost (be patient)
Completed 1000 requests
Finished 1000 requests

Server Software:        UltraBalancer
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        612 bytes

Concurrency Level:      10
Time taken for tests:   0.892 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      810000 bytes
HTML transferred:       612000 bytes
Requests per second:    1121.08 [#/sec] (mean)
Time per request:       8.920 [ms] (mean)
Time per request:       0.892 [ms] (mean, across all concurrent requests)
Transfer rate:          886.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     2    9   3.2      8      25
Waiting:        2    8   3.1      7      24
Total:          2    9   3.2      8      25

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      9
  75%     10
  80%     11
  90%     14
  95%     17
  98%     20
  99%     22
 100%     25 (longest request)

wrk

Installation

# Clone and build
git clone https://github.com/wg/wrk.git
cd wrk
make
sudo cp wrk /usr/local/bin/

# Or via package manager
brew install wrk  # macOS

Basic Usage

# 12 threads, 400 connections, 30 seconds
wrk -t12 -c400 -d30s http://localhost:8080/

# With custom script
wrk -t4 -c100 -d30s -s script.lua http://localhost:8080/

# Latency distribution
wrk -t4 -c100 -d30s --latency http://localhost:8080/

Lua Scripts

post-request.lua

wrk.method = "POST"
wrk.body   = '{"key":"value"}'
wrk.headers["Content-Type"] = "application/json"

request = function()
   return wrk.format(nil, "/api/endpoint")
end

response = function(status, headers, body)
   if status ~= 200 then
      print("Error: " .. status)
   end
end

wrk -t4 -c100 -d30s -s post-request.lua http://localhost:8080/

hey

Installation

# Go install
go install github.com/rakyll/hey@latest

# Or download binary
wget https://hey-release.s3.us-east-2.amazonaws.com/hey_linux_amd64
chmod +x hey_linux_amd64
sudo mv hey_linux_amd64 /usr/local/bin/hey

Usage

# 10000 requests, 50 concurrent
hey -n 10000 -c 50 http://localhost:8080/

# Rate limiting (100 QPS)
hey -n 10000 -q 100 http://localhost:8080/

# POST with body
hey -n 1000 -m POST -H "Content-Type: application/json" \
  -d '{"key":"value"}' http://localhost:8080/api

# Custom headers
hey -n 1000 -H "Authorization: Bearer token" http://localhost:8080/

# Duration-based
hey -z 30s -c 50 http://localhost:8080/

k6

Installation

# Linux
wget https://github.com/grafana/k6/releases/download/v0.47.0/k6-v0.47.0-linux-amd64.tar.gz
tar -xzf k6-v0.47.0-linux-amd64.tar.gz
sudo mv k6 /usr/local/bin/

# macOS
brew install k6

# Windows
choco install k6

Load Test Script

load-test.js

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },  // Ramp up
    { duration: '5m', target: 100 },  // Stay at 100 users
    { duration: '2m', target: 200 },  // Ramp to 200 users
    { duration: '5m', target: 200 },  // Stay at 200 users
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95% of requests under 500ms
    http_req_failed: ['rate<0.01'],    // Error rate under 1%
  },
};

export default function () {
  const res = http.get('http://localhost:8080/');

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);
}

# Run test
k6 run load-test.js

# Run with JSON output
k6 run --out json=results.json load-test.js

# Run with InfluxDB output
k6 run --out influxdb=http://localhost:8086/k6 load-test.js

Advanced k6 Test

advanced-test.js

import http from 'k6/http';
import { check, group } from 'k6';

export const options = {
  scenarios: {
    constant_load: {
      executor: 'constant-vus',
      vus: 50,
      duration: '5m',
    },
    spike_test: {
      executor: 'ramping-vus',
      startTime: '5m',
      stages: [
        { duration: '10s', target: 500 },
        { duration: '1m', target: 500 },
        { duration: '10s', target: 0 },
      ],
    },
  },
  thresholds: {
    'http_req_duration{scenario:constant_load}': ['p(95)<200'],
    'http_req_duration{scenario:spike_test}': ['p(95)<1000'],
  },
};

export default function () {
  group('Homepage', function () {
    const res = http.get('http://localhost:8080/');
    check(res, { 'status 200': (r) => r.status === 200 });
  });

  group('API', function () {
    const res = http.post('http://localhost:8080/api', JSON.stringify({
      key: 'value',
    }), { headers: { 'Content-Type': 'application/json' } });
    check(res, { 'status 200': (r) => r.status === 200 });
  });
}

Test Scenarios

Baseline Performance

# Establish baseline
ab -n 10000 -c 100 http://localhost:8080/ > baseline.txt

# Compare after changes
ab -n 10000 -c 100 http://localhost:8080/ > after-changes.txt
diff baseline.txt after-changes.txt

Stress Testing

# Gradually increase load
for c in 10 50 100 200 500 1000; do
  echo "Testing with $c concurrent connections..."
  ab -n 10000 -c $c http://localhost:8080/ | grep "Requests per second"
  sleep 5
done

Spike Testing

# Sudden load spike
hey -z 10s -c 10 http://localhost:8080/
sleep 5
hey -z 30s -c 1000 http://localhost:8080/  # Spike!
sleep 5
hey -z 10s -c 10 http://localhost:8080/

Endurance Testing

# Long-running test (1 hour)
wrk -t4 -c100 -d1h --latency http://localhost:8080/

# Or with k6
k6 run --duration 1h --vus 100 load-test.js

Monitoring During Tests

Real-time Metrics

# Terminal 1: Run load test
hey -z 60s -c 100 http://localhost:8080/

# Terminal 2: Watch metrics
watch -n 1 'curl -s http://localhost:8080/metrics | jq ".requests_per_second, .avg_response_time_ms"'

# Terminal 3: Monitor system resources
htop

# Terminal 4: Watch connections
watch -n 1 'ss -s'

Results Analysis

Key Metrics to Track

Throughput

Requests per second (RPS): Target varies by application
Transactions per second (TPS): For write-heavy workloads
Bytes per second: Network throughput

Latency

Average latency: Mean response time
p50 (median): 50% of requests faster than this
p95: 95% of requests faster than this
p99: 99% of requests faster than this
Max latency: Worst-case scenario

Errors

Error rate: Percentage of failed requests
Timeout rate: Requests that timed out
Connection errors: Failed to connect

System Resources

CPU usage: Should not hit 100%
Memory usage: Watch for memory leaks
Network bandwidth: Avoid saturation
File descriptors: Ensure sufficient limits

Best Practices

Load Testing Tips

Test in isolated environment - Avoid affecting production
Start small - Begin with low load, increase gradually
Monitor everything - CPU, memory, network, disk
Run multiple iterations - Results should be consistent
Test different scenarios - Baseline, stress, spike, endurance
Compare algorithms - Test round-robin vs least-connections
Document results - Track performance over time

Avoid These Mistakes

Testing from same machine as load balancer
Not accounting for connection limits
Testing without health checks enabled
Ignoring p95/p99 latencies
Not simulating realistic traffic patterns

Example Test Suite

test-suite.sh

#!/bin/bash

echo "UltraBalancer Load Test Suite"
echo "=============================="

# Configuration
LB_URL="http://localhost:8080"
RESULTS_DIR="./results"
mkdir -p $RESULTS_DIR

# Test 1: Baseline (100 concurrent, 10k requests)
echo "Test 1: Baseline performance..."
ab -n 10000 -c 100 $LB_URL/ > $RESULTS_DIR/test1-baseline.txt

# Test 2: High concurrency (500 concurrent)
echo "Test 2: High concurrency..."
ab -n 50000 -c 500 $LB_URL/ > $RESULTS_DIR/test2-high-concurrency.txt

# Test 3: Sustained load (60 seconds)
echo "Test 3: Sustained load..."
wrk -t4 -c200 -d60s --latency $LB_URL/ > $RESULTS_DIR/test3-sustained.txt

# Test 4: Spike test
echo "Test 4: Spike test..."
k6 run --out json=$RESULTS_DIR/test4-spike.json spike-test.js

echo "Tests complete. Results in $RESULTS_DIR/"

# Generate summary
echo "Summary:"
echo "--------"
grep "Requests per second" $RESULTS_DIR/*.txt

Local Testing

Test locally before load testing

Benchmarks

Performance benchmarks and comparisons

Performance Tuning

Optimize for maximum performance

Monitoring

Monitor during load tests

Getting Started

Concepts

Configuration

Deployment

Testing

API

Advanced

​Overview

Apache Bench

wrk

hey

k6

​Apache Bench (ab)

​Installation

​Basic Usage

​Output Analysis

​wrk

​Installation

​Basic Usage

​Lua Scripts

​hey

​Installation

​Usage

​k6

​Installation

​Load Test Script

​Advanced k6 Test

​Test Scenarios

​Baseline Performance

​Stress Testing

​Spike Testing

​Endurance Testing

​Monitoring During Tests

​Real-time Metrics

​Results Analysis

​Key Metrics to Track

​Best Practices

​Example Test Suite

​Related Topics

Local Testing

Benchmarks

Performance Tuning

Monitoring

Overview

Apache Bench (ab)

Installation

Basic Usage

Output Analysis

wrk

Installation

Basic Usage

Lua Scripts

hey

Installation

Usage

k6

Installation

Load Test Script

Advanced k6 Test

Test Scenarios

Baseline Performance

Stress Testing

Spike Testing

Endurance Testing

Monitoring During Tests

Real-time Metrics

Results Analysis

Key Metrics to Track

Best Practices

Example Test Suite

Related Topics