HAProxy Deep Intuition — Deep Tech Intuition

1. One-Sentence Essence

HAProxy is a single-threaded-per-core, event-driven TCP relay that learned to speak HTTP — every other feature is something layered onto that core, never grafted onto a different shape.

If you remember nothing else, remember that. HAProxy is not a web server that happens to balance load. It is not a service mesh. It is not an API gateway, even when people use it as one. At its core it is a process whose job is to take bytes off one socket and put them onto another socket as fast as possible, while occasionally inspecting and decorating them. Every feature — SSL termination, ACL routing, sticky sessions, rate limiting, health checks, HTTP/2 multiplexing — is bolted onto that core and respects its rules. When you understand HAProxy as “an obsessively optimized byte-mover that knows about HTTP,” everything from its configuration shape to its weird behaviors stops being arbitrary and starts being predictable.

2. The Problem It Solved

In 2000, Willy Tarreau was looking at a class of problem that ran out of good answers. You had a web application. The application server was the bottleneck — usually because of slow PHP or Perl, sometimes because of a database, sometimes because TLS handshakes were eating the CPU. The standard answer was to put more application servers behind something that distributed traffic across them. The available somethings were unpleasant: hardware load balancers from F5 or Cisco that cost five-figure-to-six-figure sums; round-robin DNS which spread load but couldn’t react to a dead server; or mod_proxy_balancer on Apache, which was a prefork-model web server pretending to be a proxy and falling over at a few thousand concurrent connections because each connection cost a whole process.

The specific frustrations were sharp. Hardware load balancers were opaque, expensive, and locked you into a vendor. Apache-based proxies could not handle slow clients — every slow modem dial-up holding a connection open consumed a full process. Round-robin DNS had no health checking, so when a server died, a percentage of users got broken pages until DNS TTLs expired. There was no good way to do “sticky sessions” outside the application. And nobody had a clean way to terminate SSL in front of a farm of plaintext app servers, which was becoming critical as HTTPS spread.

Tarreau’s design insight was structural: stop pretending the load balancer is a web server. Don’t fork a process per connection. Don’t allocate a thread per connection. Instead, use the operating system’s epoll/kqueue to handle tens of thousands of connections in a single process by reacting to socket events. Keep the data path obsessively short — when you can splice bytes from one socket to another inside the kernel, do that and never copy them into userspace at all. Make the configuration declarative and the behavior deterministic. Make it free. The result, HAProxy, has been the default answer to “how do I load-balance HTTP” for serious infrastructure ever since. Companies like Airbnb, Alibaba, GitHub, and Twitter use it for the same reason Tarreau wrote it: when you actually care about the throughput, latency, and operational behavior of your load balancer, nothing else has both the speed and the predictability.

The second-order insight is the one most people miss: by being narrowly focused on byte-moving and refusing to become a web server, HAProxy stayed fast forever. NGINX, which started life as a web server that grew proxy features, carries the weight of being a web server. HAProxy carries the weight of being a load balancer. That divergence in DNA is why, twenty-five years later, they still don’t quite do the same job.

3. The Concepts You Need

Before you can think clearly about HAProxy, you need its vocabulary. These are not optional jargon — they are the words the rest of the document uses, and they map directly onto the configuration file you’ll write.

The proxy structure

Frontend: the side of HAProxy that listens for client connections. A frontend binds to one or more (IP, port) pairs and defines the rules for what happens when a request arrives. “Frontend” means facing the client.
Backend: a pool of one or more servers that HAProxy can forward requests to, along with the policy for choosing among them (load balancing algorithm, health checks, session stickiness). “Backend” means facing the application.
Listen: a shorthand that combines a frontend and a backend into one block. Useful for simple setups; avoid for anything where you want to route across multiple backends.
Server: a single backend endpoint (an IP and port, or a DNS name). Each server line declares one of them. Backends contain a list of servers.
Defaults: a section whose settings are inherited by every frontend, backend, and listen below it. This is where you put timeouts and logging that apply to everything.
Global: process-wide settings — user, group, max number of connections, thread count, where to write logs. There is exactly one global section per config.

The request-shaping primitives

ACL (Access Control List): a named boolean expression that tests something about the request. acl is_api path_beg /api/ defines an ACL called is_api that is true when the request path starts with /api/. ACLs do not act — they only evaluate. You combine them with directives like use_backend and http-request deny to actually do something.
Fetch: the piece of the request or context that an ACL examines. path_beg, hdr(host), src, ssl_fc are fetches. There are hundreds of them — you can match on essentially any property of an HTTP request, the TCP connection, or HAProxy’s own state.
Match type: how the fetched value is compared. -i for case-insensitive, -m beg for “begins with”, -m end for “ends with”, -m reg for regex. Most fetches have a default match type (e.g., path_beg already does a “begins-with” match).
use_backend / default_backend: the directives that actually pick which backend handles the request. use_backend api_servers if is_api says “send it to the api_servers backend if the is_api ACL is true.” default_backend is the fallback if no use_backend matches.
http-request / http-response: rule chains that fire on every request (or response). Used to add/remove headers, redirect, deny, rewrite paths. They run in order, and the first matching action takes effect.

The load balancing primitives

Balance algorithm: how the backend picks a server. roundrobin (rotate), leastconn (fewest active connections), source (hash of client IP), uri (hash of URL), hdr(name) (hash of a header). Each backend declares exactly one.
Weight: a multiplier on how much traffic a server gets. server web1 ... weight 3 gets three times the traffic of weight 1. Used to model uneven hardware or do gradual rollouts.
Health check (check): HAProxy periodically probes each server to see if it’s alive. With just check, it’s a TCP connect. Add option httpchk and it sends an HTTP request and expects a 2xx/3xx response.
rise / fall / inter: health check tuning. inter 5s = check every 5 seconds. fall 3 = three consecutive failures and the server is marked DOWN. rise 2 = two consecutive successes to come back UP. Defaults are inter 2s fall 3 rise 2.

The persistence primitives

Sticky session / session persistence / affinity: making sure a given client lands on the same backend server repeatedly. Important when the application stores session state in server memory. The three flavors people muddle: affinity (a soft preference, often by hashing IP), persistence (a hard guarantee, usually via cookie), stickiness (HAProxy’s umbrella term).
Cookie persistence: HAProxy inserts a small cookie identifying the chosen server; the client returns it; HAProxy reads it and routes accordingly. The most reliable form.
Stick table: an in-memory key-value store, sized at startup, used to remember things across requests — which server a session went to, how many requests an IP has made, whether a client is on a deny list. This is HAProxy’s general-purpose state machine.
Peer: another HAProxy instance that this one synchronizes stick tables with, so two load balancers behind a virtual IP can share session state without an external store.

The protocol-handling primitives

Mode http vs mode tcp: the most consequential setting. In http mode, HAProxy parses the HTTP protocol — it sees methods, paths, headers — and you can do ACL routing, header manipulation, cookies, HTTP/2 multiplexing. In tcp mode, HAProxy treats the connection as an opaque byte stream and just forwards it. TCP mode is faster but blind.
SSL/TLS termination (offloading): HAProxy decrypts the TLS connection at the frontend, processes plaintext HTTP, and (typically) talks plaintext HTTP to the backend. Centralizes certs and frees backends from cipher work.
PROXY protocol: a tiny header HAProxy can prepend to backend TCP connections that tells the backend “this connection actually came from client IP 1.2.3.4.” Solves the “I’m in TCP mode and the backend thinks every request comes from the load balancer’s IP” problem. Created by HAProxy Technologies and now supported by NGINX, AWS NLB, Postfix, and many others.
X-Forwarded-For (XFF): the HTTP-mode equivalent of PROXY protocol. HAProxy adds a header containing the real client IP so the backend can log it.
Connection reuse / http-reuse: HAProxy maintains a pool of open TCP connections to backend servers and reuses them across different client requests. Saves the cost of repeated TCP and (when re-encrypting) TLS handshakes.

The operational primitives

Hitless reload: applying a new config or upgrading the binary without dropping a single in-flight connection. Done by handing off listening sockets from the old process to the new one.
Stats socket / Runtime API: a Unix or TCP socket where you can send commands to a running HAProxy process — disable a server, drain it, dump the current state, update an ACL file, change a weight. The dynamic complement to the static config file.
Stats page: a built-in HTML page showing every frontend, backend, and server with live counters. The first place you look when something is wrong.

These concepts compose. A frontend uses ACLs to choose a backend. The backend uses a balance algorithm over its servers, which it monitors with health checks, while a stick table (perhaps replicated to peers) remembers which server a session belongs to. A hitless reload swaps the whole config without dropping connections. You now have the vocabulary; the rest of the document fills in how to use it well.

4. The Distilled Introduction

This section replaces the 10-hour tutorial. Walk through it once and you will be able to install HAProxy, write a real config, run it, and reason about what it’s doing.

Install and run

On Debian/Ubuntu:

sudo apt update && sudo apt install -y haproxy
haproxy -vv     # confirms version and shows which features (SSL, Lua, HTTP/2) are compiled in

The relevant files:

/etc/haproxy/haproxy.cfg — the one config file. There is no conf.d directory by default; everything lives here.
/var/log/haproxy.log — logs (usually via syslog/journald to this file).
/run/haproxy/admin.sock — the runtime stats socket.
systemctl reload haproxy — apply a new config without dropping connections.
systemctl restart haproxy — drop everything and start over. Use sparingly.

Always validate before reloading: haproxy -c -f /etc/haproxy/haproxy.cfg. A bad config refused at parse time is dramatically better than a bad config that broke production at runtime.

The shape of every config

Every HAProxy config has the same skeleton: global, defaults, then any number of frontend/backend/listen blocks. Indentation is conventional but not required. Order matters within a section (rules fire top-down).

global
    log /dev/log local0
    maxconn 50000
    user haproxy
    group haproxy
    daemon

defaults
    mode    http
    log     global
    option  httplog
    timeout connect 5s
    timeout client  60s
    timeout server  60s
    timeout http-request 10s

The global section sets process-wide things. maxconn is the absolute cap on concurrent connections — it bounds your file descriptor usage. Pick a number well above your peak; HAProxy is happy at 100,000+ on modern hardware.

The defaults section sets inherited settings. The four timeouts above are not optional — without them, HAProxy refuses to start, because there is no sane default for “how long should I wait for a slow client.” option httplog enables HAProxy’s information-dense one-line-per-request log format, which you will read more than any other log in your career.

First config: a single HTTP service across three servers

frontend web_front
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    server web1 10.0.1.10:8080 check
    server web2 10.0.1.11:8080 check
    server web3 10.0.1.12:8080 check

That’s a real, deployable load balancer. bind *:80 listens on all interfaces, port 80. balance roundrobin rotates among the three servers. check on each server enables HAProxy’s default 2-second TCP-level health check. option httpchk GET /healthz upgrades it to an HTTP-level check: HAProxy sends GET /healthz HTTP/1.0 and http-check expect status 200 says the response must be a 200 to count as healthy.

Save it, run haproxy -c -f /etc/haproxy/haproxy.cfg, fix any errors, then systemctl reload haproxy. Hit http://your-haproxy/ a few times; watch the logs; you should see requests rotating across the three servers.

Adding HTTPS — SSL termination

You almost never want to deploy HAProxy without terminating TLS. Get a cert (Let’s Encrypt, your CA, whatever), concatenate the certificate, intermediates, and private key into one PEM file (cat fullchain.pem privkey.pem > /etc/haproxy/certs/site.pem — yes, in that order), and:

global
    # ...
    ssl-default-bind-options ssl-min-ver TLSv1.2
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    tune.ssl.default-dh-param 2048

frontend web_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/site.pem alpn h2,http/1.1
    http-request redirect scheme https code 301 unless { ssl_fc }
    default_backend web_servers

ssl crt /etc/haproxy/certs/site.pem makes the bind line terminate TLS using that cert. alpn h2,http/1.1 advertises HTTP/2 so modern clients use it. The redirect line forces plain HTTP to HTTPS — ssl_fc is the fetch that says “this connection is TLS”; unless { ssl_fc } runs the redirect only if it isn’t. You can drop multiple certs in a directory and bind *:443 ssl crt /etc/haproxy/certs/ will load all of them and route on SNI automatically.

Routing — ACLs and use_backend

The moment you have more than one app, you need to route. ACLs are how.

frontend web_front
    bind *:443 ssl crt /etc/haproxy/certs/

    acl is_api      path_beg /api/
    acl is_admin    path_beg /admin
    acl is_internal src 10.0.0.0/8

    http-request deny if is_admin !is_internal

    use_backend api_servers   if is_api
    use_backend admin_servers if is_admin
    default_backend web_servers

backend api_servers
    balance leastconn
    server api1 10.0.2.10:8080 check
    server api2 10.0.2.11:8080 check

backend admin_servers
    server admin1 10.0.3.10:8080 check

Read top-down: define ACLs (none fire on their own), then write rules that use them. The http-request deny if is_admin !is_internal rule kicks external IPs out of /admin before they reach any backend. Then use_backend lines route based on path, in order — first match wins. default_backend catches everything else. We’ll see in the Mental Model section why this top-down evaluation matters more than it might appear: HAProxy doesn’t look ahead, doesn’t optimize the chain, and runs every rule in order. It’s predictable, and it’s your job to put the cheap and selective ACLs first.

When the application keeps session state in a server’s memory, you must route the same user to the same server. The cleanest approach:

backend web_servers
    balance roundrobin
    cookie SERVERID insert indirect nocache httponly secure
    server web1 10.0.1.10:8080 check cookie s1
    server web2 10.0.1.11:8080 check cookie s2
    server web3 10.0.1.12:8080 check cookie s3

The cookie SERVERID insert indirect nocache line tells HAProxy: when a response goes out without our cookie, attach Set-Cookie: SERVERID=<server-id>. On the next request, read that cookie and route to the matching server. The cookie sN on each server line is the value HAProxy will use; indirect means HAProxy strips the cookie before forwarding to the backend (so the application never sees it); nocache ensures CDNs don’t cache the Set-Cookie response and hand it to other users.

Add option redispatch to the defaults so that if the user’s sticky server is down, they get sent to a different one instead of getting an error.

Rate limiting — your first stick table

frontend web_front
    bind *:443 ssl crt /etc/haproxy/certs/

    stick-table type ip size 1m expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

    default_backend web_servers

stick-table type ip size 1m expire 30s store http_req_rate(10s) declares an in-memory table keyed by IPv4 address, sized for 1 million entries, with 30-second idle expiration, that tracks the rate of HTTP requests over a 10-second sliding window. http-request track-sc0 src says: for each request, use the source IP as the key into stick-counter slot 0 (HAProxy has slots sc0 through sc2 by default for tracking). http-request deny ... if { sc_http_req_rate(0) gt 100 } then denies any request from an IP that has averaged more than 100 requests per 10 seconds — i.e., 10 RPS sustained. The same stick table can track connection rate, error rate, bandwidth, almost anything; this is HAProxy’s general-purpose stateful machinery.

The stats page — your first observability tool

Add this anywhere:

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

Now hit http://your-haproxy:8404/stats. You’ll see every frontend, backend, and server with live counters: current sessions, total sessions, queue depth, bytes in/out, HTTP response codes, server status (UP/DOWN/MAINT/DRAIN), how long each server has been in its current state, time of last status change. stats admin if LOCALHOST enables clickable buttons to drain, disable, or re-enable servers from localhost. The first thing you do when something is wrong: open the stats page.

Reading the logs

HAProxy logs are dense and brilliant once you know how to read them:

Oct 15 14:32:11 lb1 haproxy[1234]: 203.0.113.45:51234 [15/Oct/2025:14:32:10.456] web_front~ web_servers/web2 0/0/1/23/24 200 4521 - - ---- 50/45/12/4/0 0/0 "GET /api/users/42 HTTP/1.1"

The fields you care about, in order:

203.0.113.45:51234 — client IP and port
web_front~ — frontend that received the request (the ~ means TLS)
web_servers/web2 — backend and specific server it was routed to
0/0/1/23/24 — the timing fields, in milliseconds: Tq/Tw/Tc/Tr/Tt = request wait / queue wait / connect / response / total. If any are -1, that stage timed out. This single field tells you whether a slow response is the client’s fault, the queue’s fault, the backend’s fault, or the network’s fault.
200 — HTTP status code
4521 — bytes sent to client
---- — termination state flags; uppercase letters here mean something went wrong (e.g., sH-- means session ended on a server timeout while waiting for headers; SL-- means we had to close a Layer-7 retry)
50/45/12/4/0 — connection counts: process / frontend / backend / server / retries
"GET /api/users/42 HTTP/1.1" — the request line

Learn the timing field. Most production debugging starts there.

Commands you’ll actually use

# Validate config (run this every time before reload)
haproxy -c -f /etc/haproxy/haproxy.cfg

# Reload (hitless — no dropped connections)
systemctl reload haproxy

# Inspect runtime state via the stats socket
echo "show info"           | sudo socat stdio /run/haproxy/admin.sock
echo "show stat"           | sudo socat stdio /run/haproxy/admin.sock
echo "show servers state"  | sudo socat stdio /run/haproxy/admin.sock
echo "show table web_servers" | sudo socat stdio /run/haproxy/admin.sock

# Drain a server for maintenance (stops new connections, lets existing finish)
echo "set server web_servers/web2 state drain" | sudo socat stdio /run/haproxy/admin.sock

# Then mark fully down once drained
echo "set server web_servers/web2 state maint" | sudo socat stdio /run/haproxy/admin.sock

# Bring it back
echo "set server web_servers/web2 state ready" | sudo socat stdio /run/haproxy/admin.sock

socat is the standard way to talk to the socket; you can also use nc -U if you have it. The Runtime API has dozens of commands — help over the socket lists them.

You now have working knowledge. You can stand up HAProxy in front of a real service, terminate TLS, route on path, do session persistence, rate-limit by IP, observe what’s happening, and drain servers for maintenance. That’s the 80%. The rest of this document is what separates working from good.

5. The Mental Model

Four ideas. Internalize these and most HAProxy behavior becomes predictable without consulting the docs.

Core Idea 1: HAProxy is an event loop, not a process pool

HAProxy runs a small number of threads (one per CPU core, typically), and each thread runs a tight event loop on top of epoll (Linux) or kqueue (BSDs). When a socket has data, the event loop wakes up, reads what it can, processes it, writes whatever it can to the other side, and goes back to sleep. A single thread can hold tens of thousands of idle connections at near-zero cost, because idle connections are just file descriptors sitting in an epoll set, not threads or processes consuming memory and CPU.

This predicts a long list of behaviors:

Slow clients don’t degrade you the way they do Apache. A client on a 3G connection holding a connection open for 30 seconds costs HAProxy almost nothing. This is why HAProxy is a popular choice for fronting application servers that have a process-per-connection model — HAProxy absorbs the slow clients and only opens fast, well-behaved connections to the backend.
Adding more memory rarely helps performance; adding more CPU cores does. HAProxy is CPU-bound when it’s bound at all. On a single core, you can typically get tens of thousands of plaintext HTTP requests per second.
Configuration changes need a process restart, not a thread restart. The event loop’s data structures are bound to the process. This is why hitless reload exists as a feature — it’s not optional polish, it’s the only way to apply changes without killing connections.
Long-running operations inside HAProxy block everything on that thread. This is why Lua scripts in HAProxy must be fast: a slow Lua function stops the event loop, and every connection on that thread waits. It’s also why HAProxy avoids doing things like file I/O in the hot path.
CPU pinning matters more than you’d think. HAProxy will use CPU caches efficiently if you let it stick threads to cores (cpu-map). Bouncing threads across cores invalidates L1/L2 cache and costs measurable performance at high RPS.

Core Idea 2: There are two pipes — frontend-side and backend-side — and they meet in the middle

When a client connects, HAProxy is not a router that picks where the packet goes. It is two TCP connections glued together: client→HAProxy and HAProxy→backend. HAProxy reads from one, possibly transforms it, writes to the other, and vice versa. Each side has its own state, its own timeouts, its own buffer.

This is why HAProxy has separate timeouts for client and server: timeout client is how long it tolerates an idle client connection; timeout server is how long it tolerates an idle backend connection. Confusing them is one of the most common production mistakes — set timeout server to 5 seconds and your slow upstream API requests will start mysteriously failing.

This split also predicts:

You can do things in HAProxy that no L4 device can do. Because HAProxy has terminated the client connection and originated its own backend connection, it has full control of both. It can retry the backend without the client knowing. It can rewrite request headers. It can return a cached response with no backend at all (via errorfile or http-request return).
Client and backend can speak different protocols. HTTP/2 on the client side, HTTP/1.1 on the backend side. HTTPS on the client side, plaintext on the backend. HTTP/3 inbound, HTTP/2 outbound. This is the “HAProxy as protocol adapter” pattern, and it works because the two pipes are independent.
Connection reuse is asymmetric. HAProxy can hold one TCP connection to a backend open and pipe many different clients’ requests through it (http-reuse always). The client-side and backend-side connection lifecycles are decoupled by design.
Errors on one side don’t always reach the other. A backend that closes its connection mid-response causes HAProxy to close the client connection too, but the client gets a truncated response — there’s no way to “undo” the bytes already sent. The logs will show SH or SD flags. Understanding the two-pipe model tells you why.

Core Idea 3: Everything is a chain of declarative rules evaluated in order

HAProxy’s configuration is not procedural. You don’t write “do X, then check Y, then do Z.” You declare ACLs, then write rules of the form do <action> [if <acl>]. Rules are evaluated top-to-bottom; the first matching rule wins (or all matching rules apply, depending on the directive). HAProxy makes no attempt to optimize the order or cache the evaluation.

This shapes everything:

Order of use_backend and http-request lines is significant. use_backend api if is_api followed by use_backend admin if is_admin will never route /admin to the admin backend if you wrote them in the wrong order with overlapping ACLs.
Put cheap, selective ACLs first. acl is_internal src 10.0.0.0/8 is essentially free. acl bad_user_agent hdr(user-agent) -m reg .*evil.* does a regex evaluation. Order matters under load.
There’s no early exit unless you write one. If you want a deny to short-circuit, put it before the use_backend lines. HAProxy doesn’t deduce that a denied request shouldn’t be routed.
Rule chains are debugger-friendly. You can read top-down what will happen to a request — there’s no hidden middleware stack, no plugin ordering surprise. This is one of the reasons HAProxy configs scale to thousands of lines without becoming incomprehensible. The cost is verbosity.
Multiple acl lines with the same name OR together. Multiple ACLs after one if are ANDed. if is_api is_authenticated means both must be true. acl trusted src 10.0.0.0/8 followed by acl trusted src 192.168.0.0/16 makes trusted true if either matches.

Core Idea 4: State lives in three places — config, stick tables, and the kernel

To predict HAProxy’s behavior under failure, you need to know where state is.

The config file holds the structure: which frontends exist, which backends, which servers, which rules. This state is static within a process — to change it, you reload (which starts a new process).
Stick tables hold the runtime, per-key state: which session is sticky to which server, what a given IP’s request rate is, whether a client is on a soft-banlist. Stick tables can be replicated across HAProxy instances via the peers mechanism, which gives you cluster-aware state without an external Redis.
The kernel holds the connections themselves — TCP socket state, the listening sockets, the established connection table. This is why hitless reload works: the new process inherits listening sockets from the old one via SO_REUSEPORT and the runtime-API expose-fd listeners mechanism, and the old process keeps draining its connections from kernel state until they finish.

Implications:

A reload is not a restart. Hitless reload spawns a fresh process with the new config. The old process stops accepting new connections but continues serving existing ones until they finish (or hard-stop-after kicks them out). Stick-table state in the old process is not automatically transferred to the new one unless you’ve configured peers (with the old process as a peer of the new one); this is a real gotcha if you rely on stickiness across reloads.
Restarts (not reloads) drop everything. systemctl restart is a hard SIGTERM. You almost never want it in production.
Cluster state requires explicit configuration. If you run two HAProxy nodes behind a VRRP/keepalived virtual IP, they do not share stick tables by default. You need a peers section linking them. Otherwise, a failover loses every session’s stickiness.

These four ideas — event loop, two-pipe model, ordered rule chain, three places of state — are the entire mental scaffold. Everything that follows is either an elaboration of one of them or a real-world consequence of all of them interacting.

6. The Architecture in Plain English

Let’s narrate a single HTTPS request from arrival to response. This is the most important thing to internalize, because once you can predict the path, you can predict where things go wrong.

A client opens a TCP connection to HAProxy on port 443. The kernel accepts it on the listening socket (created by the bind directive in some frontend) and hands the file descriptor to whichever thread happens to be free — by default HAProxy’s threads share the listening socket via SO_REUSEPORT-like distribution. That thread’s event loop picks up the new connection and begins the TLS handshake. This is a CPU-intensive step: the RSA decryption (or ECDHE key agreement) happens here, and on a single core, HAProxy can do roughly 1,500 fresh RSA-2048 TLS handshakes per second, or ~14,000 TLS session resumptions per second (when clients have a cached session). This is why TLS session caching and OCSP stapling matter so much — they convert expensive operations into cheap ones.

Once the TLS handshake completes, HAProxy now sees plaintext bytes. It reads the request line and headers. While doing so, it consults the frontend’s rule chain in order: tcp-request rules (which can fire before HTTP is parsed, useful for things like denying connections from blacklisted IPs without even doing the TLS negotiation in some cases), then http-request rules. Each rule may match an ACL — every ACL is evaluated only when first referenced for this request, and the result is cached for the rest of the request. The rule chain runs to completion or until a terminal action (like deny, redirect, or return) fires.

If no terminal action fires, the frontend reaches its use_backend lines and picks a backend. Now the backend’s logic runs. The load balancing algorithm picks a server — if roundrobin, it advances its counter; if leastconn, it scans its servers and picks the one with the fewest active connections; if source, it hashes the client IP and uses that hash modulo the running servers (or, with hash-type consistent, a consistent-hash ring) to pick. If session stickiness is configured, the stick table is consulted first, and only if the table doesn’t have a match (or matches a server that’s down) does the algorithm choose freshly.

With a server chosen, HAProxy checks its connection pool. If http-reuse is enabled and a suitable idle connection to that server exists, HAProxy attaches the request to that connection. Otherwise, it opens a new TCP connection to the backend. If that connection is also TLS (server ... ssl), another handshake happens here. The request is then written to the backend, possibly with modifications — X-Forwarded-For added, Host rewritten, cookies stripped, whatever the rule chain said to do.

While waiting for the backend response, HAProxy doesn’t block. The thread goes back to its event loop and handles whatever other connections need attention. When the backend response arrives, the event loop wakes up for that file descriptor. HAProxy reads the response headers, runs the http-response rule chain (which can add headers, rewrite status codes, strip the sticky cookie, capture data into a stick table), and then begins streaming the response body to the client. On Linux, when conditions allow, HAProxy uses the splice() system call to move bytes from the backend socket to the client socket entirely inside the kernel, never copying through userspace at all. This is part of why HAProxy is so fast: in the optimal case, the data path for body bytes is a few kernel-space operations with no memory copies.

When the response finishes, the request is logged (a single dense line — the one we dissected earlier), counters update, and the connection either goes idle waiting for the next request (HTTP keep-alive) or closes. The backend connection, if eligible for reuse, goes back into the pool.

Where state lives across this flow: the listening socket and the established connections live in the kernel. The rule chains and routing decisions are immutable config in the HAProxy process. Per-connection state (current parser position, buffer contents, chosen backend, chosen server) lives in HAProxy memory associated with the connection’s file descriptor. Cross-request state (sticky session mappings, rate counters, soft-bans) lives in stick tables. Logs go out via syslog/journald or to stdout in containerized deployments. There is no database, no external state store, no dependency on anything other than the process and the kernel — which is precisely why HAProxy is so operationally cheap to run.

The threading model deserves a note. HAProxy used to be strictly single-process / single-threaded; it now runs multi-threaded by default (nbthread defaults to one thread per available core). Threads share most data structures with careful locking, and connections are pinned to the thread that accepted them so they don’t bounce. There is also a deprecated nbproc (multi-process) mode that is best avoided — it makes shared state across stick tables, stats, and SSL caches a coordination nightmare. For everything after HAProxy 2.0, use threads, not processes.

7. The Things That Bite You

These are the gotchas that take six to twelve months of production use to develop intuition for. Each connects back to the mental model.

Gotcha 1: A missing `default_backend` returns 503, not an error you’d notice in testing

If your frontend’s use_backend rules don’t match a request and there’s no default_backend, HAProxy returns 503 Service Unavailable. Worse, the log line will show the frontend name as the “backend,” which is confusing — your stats page will look fine, all backends green, but users get 503s. The root cause is in the rule chain ordering idea: rules are evaluated top-down and there’s no “what if nothing matches” pseudo-state — you must explicitly handle the fallthrough. Always specify a default_backend, even if it’s a single-server “everything else” backend that serves a styled error page.

Gotcha 2: `timeout server` is not “max request time”

timeout server is how long HAProxy waits between bytes from the backend after the request has been sent. If your backend streams a response over 30 seconds but never goes idle for more than timeout server worth of time, the timeout never fires. Conversely, a backend that takes 4 seconds of pure processing before sending the first byte will time out under the default timeout server 5s. This is the two-pipe model showing through — the timeout applies to socket inactivity on the backend pipe, not to total request duration. For long-tail APIs, set timeout server to something like 60–120 seconds, not the optimistic 30s default people copy from blog posts.

Gotcha 3: `mode tcp` strips your ability to see the request

In TCP mode, HAProxy has no idea what’s flowing through the connection — it’s bytes. This means: no path_beg ACLs (the bytes haven’t been parsed as HTTP), no X-Forwarded-For insertion, no HTTP keep-alive multiplexing, no HTTP response code logging. People configure mode tcp thinking “it’s just faster” and then can’t figure out why their HTTP routing rules silently do nothing. The fetches you have in TCP mode are limited to TCP-level things: source IP, destination port, SNI hostname (for TLS), connection age. Use TCP mode when you’re load-balancing a non-HTTP protocol (MySQL, Postgres, Redis, raw WebSocket) or when you want to pass TLS through to the backend without terminating it. Otherwise, use HTTP mode.

Gotcha 4: Stick tables are not persistent across reloads (unless you do specific things)

A stick table is in-memory in the HAProxy process. When you reload (hitless or not), a new process is started with empty stick tables. The session-stickiness that users were relying on is gone. There are two ways out: either use cookie-based persistence (the cookie lives in the client’s browser, so reloads don’t affect it), or configure the old process as a peer of the new one so it can transfer table contents over. The cookie path is simpler; the peer path is more general (it also handles HA across machines). This catches almost everyone the first time they do “I’ll just reload to pick up the new config.” If your monitoring shows a small spike of session loss after reloads, this is why.

Gotcha 5: `option redispatch` is off by default and you almost always want it on

If a server is marked DOWN between when HAProxy chose it and when it actually tried to connect, the default behavior is to try the same dead server retries times and then return 503 to the client. With option redispatch, HAProxy will pick a different server after the first failed attempt. The cost is that you lose strict sticky-session adherence under failure — but failure is exactly when you want HAProxy to be flexible. Put option redispatch in your defaults and forget about it.

Gotcha 6: Health check failures look fine until they don’t

Default health checks are 2-second TCP connects every 2 seconds. They tell you nothing about whether the application can actually serve a request — only that something is listening on the port. The classic disaster: backend’s web server is up, but the database connection pool is exhausted, and every real request returns 500. HAProxy keeps every server green. Always use option httpchk with http-check expect status 200 against a real /healthz endpoint that exercises the app’s dependencies. And keep that endpoint genuinely cheap — a slow health check that takes 500ms can starve HAProxy’s CPU on a busy box (200 backends × every 2s = 100 checks/s, each taking 500ms…).

Gotcha 7: HAProxy reads logs at startup, but log writes don’t fail when the destination dies

If your log directive points at syslog and syslog crashes, HAProxy doesn’t error — it just stops logging. You will not be told. Logs are critical for HAProxy debugging because there’s almost no other observability surface for individual requests. Always have an alert on “no HAProxy logs received in the last 60 seconds” in production. The dual-target pattern (log /dev/log local0 AND log 10.0.1.5:514 local0 for a remote logger) helps but doesn’t fully fix it.

Gotcha 8: `X-Forwarded-For` is forge-able if you don’t strip it at the edge

option forwardfor appends an XFF header with the client’s source IP. But it appends — it doesn’t replace. If a malicious client sends X-Forwarded-For: 127.0.0.1, that header still goes to your backend, and your backend may trust it. The pattern is: at the public edge HAProxy, http-request del-header X-Forwarded-For before option forwardfor runs. Only set option forwardfor if-none when you genuinely trust the upstream proxy (CDN, another HAProxy). The same logic applies to X-Real-IP and any other header your backend uses for client identification.

Gotcha 9: Connection reuse to backends with `mode http` can cross user boundaries

With http-reuse always, two different clients’ requests can be multiplexed over the same TCP connection to a backend. This is a feature — it’s why latency drops — but it has surprising consequences. If your backend application has any IP-based logic that looks at the connection’s source address (not the XFF header), it will see HAProxy’s IP, not the client’s. More subtly, if your backend code accidentally leaks state in connection-local variables (a bug, but a common one in some frameworks), data can bleed across requests. The safe setting for sensitive backends is http-reuse safe (the default): only subsequent requests of a session reuse the connection, never the first request of a new session.

Gotcha 10: Cookie persistence breaks if the backend modifies the `Set-Cookie` header

cookie SERVERID insert mode inserts HAProxy’s tracking cookie into the response. If your backend application also sets cookies in the response and your reverse proxy or CDN coalesces, reorders, or rewrites them, the SERVERID cookie can be lost. The symptom is: stickiness “mostly” works but a few percent of users keep getting balanced across servers. The diagnosis is to capture the response in tcpdump or browser dev tools and look at the actual cookies arriving. The fix is usually either to use cookie SERVERID prefix (which appends the server ID to an existing application cookie rather than creating a new one) or to stop whatever’s rewriting cookies.

8. The Judgment Calls

Each of these is a decision an experienced HAProxy operator has made many times. The right answer changes with context — but there is almost always a right answer for a given context.

Call 1: HTTP mode or TCP mode?

The decision rule is sharper than people make it. Use HTTP mode unless you’re load-balancing a protocol that isn’t HTTP, or you want to pass TLS through without terminating it. HTTP mode costs you a small amount of CPU for parsing, but gives you ACL routing, header manipulation, HTTP/2 demultiplexing, accurate per-request logging, and most of HAProxy’s features. The “TCP is faster” claim is true but irrelevant — the overhead of HTTP parsing is negligible compared to TLS, and you almost certainly need at least one HTTP feature (XFF, ACLs, response-code logging) for any real web traffic. Use TCP mode for: databases (MySQL, Postgres, Redis), SMTP, Minecraft servers, raw WebSocket where TLS is passed through, and when you specifically need TLS to terminate at the backend.

Cookie persistence is the right answer for almost all web applications. It’s per-user, it survives NAT and dynamic IPs, it doesn’t care about CG-NAT (where a hundred users behind a corporate firewall share one IP and would all stick to the same server). The exception: when you can’t insert cookies (because the protocol isn’t HTTP, or the client is a non-browser API consumer that doesn’t return them). Then use stick-table type ip — but be aware that mobile clients and corporate users will be misbalanced. Never use balance source as your primary persistence mechanism for public web traffic — it’s a fallback algorithm, not a persistence strategy. The whole reason cookies exist is because IP-based persistence doesn’t work on the public internet.

Call 3: `roundrobin`, `leastconn`, or hash-based?

Default to roundrobin for most web traffic. Servers are identical, request times are short, and round-robin’s predictability is a feature when you’re debugging. Switch to leastconn when request processing times vary significantly — long polling, streaming, heavy admin operations mixed with light reads. leastconn naturally directs new requests away from servers that are bogged down on slow ones. Use balance uri or balance hdr(Host) when there’s a caching benefit: if your backends are caches and the same URL should hit the same cache, hashing on URL maximizes hit rates. Use hash-type consistent if your backend pool is dynamic (autoscaling); without it, every server change reshuffles every hash assignment, which is catastrophic for cache hit rates.

Call 4: Terminate TLS at HAProxy, or pass it through?

Terminate at HAProxy unless you have a specific reason not to. The reasons not to are: regulatory (some payments compliance contexts forbid having plaintext anywhere outside the application), mTLS where the backend must verify the client cert directly, or end-to-end encryption requirements. Terminating at HAProxy gives you: centralized certificate management, ACL routing on path (impossible without decryption), HTTP/2 multiplexing, response-code logging, header manipulation, OCSP stapling, and SSL session caching that’s shared across all your backends. If your concern is “plaintext on the internal network,” the answer is re-encrypt to the backends (server ... ssl), not pass-through. You get the best of both — HAProxy can still inspect — and the additional cost is small if you use connection reuse on the backend side.

Call 5: How many threads? `nbthread` setting.

Set nbthread to the number of physical cores available, not the number of hyperthreads. HAProxy benefits from cache locality, and hyperthreading often hurts more than it helps for HAProxy’s workload. Use cpu-map to pin threads to specific cores for highest predictability:

global
    nbthread 4
    cpu-map 1 0
    cpu-map 2 1
    cpu-map 3 2
    cpu-map 4 3

In containerized environments, set nbthread to match the CPU limit on the container, not the host’s core count. Setting it too high causes context-switch overhead; too low leaves performance on the table.

Call 6: How aggressive should health checks be?

For most services: inter 5s fall 3 rise 2. That’s “check every 5 seconds, mark down after 3 failures (15 seconds of detection time), mark up after 2 successes.” More aggressive (inter 1s fall 1) detects failures fast but flap-flaps a marginal server in and out, causing connection storms. Less aggressive (inter 30s) leaves dead servers in rotation for half a minute. For ultra-critical paths, use fastinter: when a server is in a transitional state (some checks failed, but not enough to mark down), check it every fastinter instead of inter. inter 5s fastinter 1s gives you slow steady-state polling but fast detection during a problem.

Call 7: HAProxy in front of, instead of, or alongside NGINX?

These tools do not do the same job, despite overlap. HAProxy is better at load balancing; NGINX is better at being a web server. If your edge job is “terminate TLS, route to backends, do session stickiness, rate-limit, handle huge concurrency,” that’s HAProxy. If your edge job is “serve static files, do PHP-FPM via fastcgi, gzip on the fly, handle a small backend pool with simple routing,” that’s NGINX. Many serious stacks run both: HAProxy at the L4/L7 edge for raw traffic management, NGINX as the application-tier reverse proxy where the application lives. The “HAProxy or NGINX, pick one” framing is wrong; the question is which job each is doing.

Call 8: Stats endpoint exposure — what risk is acceptable?

The stats page is incredibly useful and incredibly leaky. It shows your entire topology, every backend, every server, and (in admin mode) lets you drain or disable servers. Default rule: never expose stats on a publicly-bindable IP. Either bind to an internal-only IP (bind 10.0.0.5:8404), require basic auth (stats auth admin:supersecret), or proxy access through your internal monitoring system. If you must expose it externally for a SaaS dashboard, use an unguessable URL path and basic auth and IP allowlisting — defense in depth.

Call 9: When does `http-reuse always` make sense vs. `safe` vs. `never`?

safe (the default) is right for most setups. Subsequent requests in a keep-alive session reuse the connection, but the first request of a fresh session goes on a new connection. This avoids a rare class of bug where the backend closes a connection mid-request and the client can’t safely retry. Use always when your backend is rock-solid about connection handling (in-house services you control, modern frameworks) and your latency-sensitivity is high — backend connection establishment can cost real milliseconds, especially with re-encryption. Use never if you’re connecting to backends that have buggy keep-alive (legacy systems, some custom protocols pretending to be HTTP).

Call 10: How do you handle a cluster of HAProxy instances?

The standard pattern is two HAProxy nodes in active/passive with keepalived running VRRP, sharing a virtual IP. Sticky-session state is shared between them via peers. This is enough for almost all use cases. The reasons to go beyond this: throughput exceeding what a single node can handle (rare — a modern HAProxy node does well into the hundreds of thousands of RPS), or you need multi-region active/active. For active/active, you generally put a layer of DNS-based or anycast routing in front of multiple HAProxy clusters, and accept that stick-session state is per-cluster (so users may lose stickiness across regions on a regional failover; usually acceptable). Avoid the temptation to use nbproc > 1 to “scale horizontally on one box” — it makes shared state painful. Scale vertically with threads first, then horizontally with multiple boxes.

Call 11: Lua, the SPOE, or just rewrite the config?

HAProxy supports embedded Lua and a separate “Stream Processing Offload Engine” (SPOE) that lets you call out to an external service for per-request logic. The strong default is neither. HAProxy’s ACL language is more powerful than people realize — most “I need a Lua script” turns out to be expressible in ACLs and http-request rules. Use Lua when you genuinely need procedural logic (cryptographic operations, complex string manipulation, external lookups), and remember that Lua runs in the event loop and blocks every connection on that thread while it executes — keep it microseconds-fast. Use SPOE when you need to call out to a service for each request and you can tolerate the round-trip — typical use cases are bot-detection scoring or expensive auth checks where the latency tradeoff is acceptable.

9. The Commands/APIs That Actually Matter

Grouped by what you’re trying to accomplish.

Config inspection and reload

haproxy -c -f /etc/haproxy/haproxy.cfg          # validate config, exit 0 if valid
haproxy -vv                                     # detailed build info: OpenSSL version, features
haproxy -dKall -q -c -f /dev/null               # dump all default keywords (for autocomplete tools)
systemctl reload haproxy                        # hitless reload — the daily-use command
systemctl restart haproxy                       # hard restart — drops connections, avoid

The reload-vs-restart distinction is the single most important operational concept. Always reload for config changes. restart only when you’ve changed something the reload can’t pick up — major version upgrades occasionally need it, certain syscall-level changes need it.

Runtime control via stats socket

The socket lives at /run/haproxy/admin.sock if you set it up:

global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

(The expose-fd listeners part is what enables hitless reload.)

# Information
echo "show info"            | sudo socat stdio /run/haproxy/admin.sock     # process info: uptime, version, current sessions, max sessions
echo "show stat"            | sudo socat stdio /run/haproxy/admin.sock     # CSV stats for every frontend/backend/server
echo "show servers state"   | sudo socat stdio /run/haproxy/admin.sock     # server states (UP/DOWN/MAINT/DRAIN)
echo "show pools"           | sudo socat stdio /run/haproxy/admin.sock     # memory pool usage
echo "show errors"          | sudo socat stdio /run/haproxy/admin.sock     # captured request/response errors
echo "show sess"            | sudo socat stdio /run/haproxy/admin.sock     # list every active session (verbose!)

# Stick tables
echo "show table http_front"                              | sudo socat stdio /run/haproxy/admin.sock
echo "show table http_front data.http_req_rate gt 50"     | sudo socat stdio /run/haproxy/admin.sock
echo "clear table http_front key 203.0.113.45"            | sudo socat stdio /run/haproxy/admin.sock

# Server state changes
echo "set server web_servers/web2 state drain"    | sudo socat stdio /run/haproxy/admin.sock  # stop new connections
echo "set server web_servers/web2 state maint"    | sudo socat stdio /run/haproxy/admin.sock  # mark fully down
echo "set server web_servers/web2 state ready"    | sudo socat stdio /run/haproxy/admin.sock  # back to service
echo "set server web_servers/web2 weight 50%"     | sudo socat stdio /run/haproxy/admin.sock  # gradual rollout
echo "set server web_servers/web2 addr 10.0.1.20" | sudo socat stdio /run/haproxy/admin.sock  # change target IP

# ACL/map file updates without reload
echo "add acl /etc/haproxy/blacklist.lst 203.0.113.45"    | sudo socat stdio /run/haproxy/admin.sock
echo "del acl /etc/haproxy/blacklist.lst 203.0.113.45"    | sudo socat stdio /run/haproxy/admin.sock

The drain → wait → maint pattern is how you do graceful server removal without dropping any in-flight requests. Drain stops new connections, existing ones finish naturally, then you mark maintenance fully.

Configuration anchors you’ll write in every file

# In global:
maxconn 50000                                       # hard cap; size for peak + headroom
nbthread 4                                          # match physical cores
log /dev/log local0                                 # logging target
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
tune.ssl.default-dh-param 2048

# In defaults:
mode http
log global
option httplog                                      # the dense one-line-per-request format
option dontlognull                                  # skip logging health-check-only connections
option redispatch                                   # retry on a different server when sticky one is down
option http-server-close                            # don't keep alive to dead-end servers
retries 3
timeout connect 5s
timeout client 60s
timeout server 60s
timeout http-request 10s                            # slowloris protection
timeout http-keep-alive 30s
timeout queue 30s

# Typical frontend pattern:
frontend https_in
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    http-request set-header X-Forwarded-Proto https
    http-request del-header X-Forwarded-For
    option forwardfor

# Typical backend pattern:
backend web_servers
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    cookie SERVERID insert indirect nocache httponly secure
    server web1 10.0.1.10:8080 check inter 5s fall 3 rise 2 maxconn 1000 cookie s1
    server web2 10.0.1.11:8080 check inter 5s fall 3 rise 2 maxconn 1000 cookie s2

That’s the production-quality boilerplate. Almost every real config looks like a variation on this.

Diagnostic shell commands

# Watch the log live, filtered for non-2xx responses
sudo tail -f /var/log/haproxy.log | grep -v ' 200 '

# Look for backend errors
sudo grep -E ' 5[0-9]{2} ' /var/log/haproxy.log | tail -50

# Find the slowest requests (sort by Tt - total time, in the 8th field)
sudo awk '{print $11, $0}' /var/log/haproxy.log | sort -rn | head -20

# Check kernel-level connection limits
sysctl net.ipv4.ip_local_port_range          # outbound port range to backends
sysctl net.ipv4.tcp_max_syn_backlog          # SYN queue depth
ss -s                                        # socket summary

10. How It Breaks

Failure modes, mapped to the mental model.

503 with no clear backend

Symptoms: clients see 503 Service Unavailable. The HAProxy log shows the request being processed. Root cause: either no backend matched a use_backend rule and there’s no default_backend, or every server in the chosen backend is DOWN. The backend field in the log will be empty (just the frontend name with no /) for the no-backend-matched case. Diagnose: check the stats page — look for backends where every server is red. If the stats page shows everything green but you’re still getting 503s, look at the request’s path against your use_backend rules and confirm one of them should have matched. Fix: add default_backend for the fallthrough case; if everything’s down, check the health check configuration — often the check is too aggressive or pointing at a /health endpoint that’s broken.

Backend connection failures spike during peak traffic

Symptoms: 503s during peak load, sometimes 502s. Log termination flags show sC (server connection failed). Root cause: either you’ve hit maxconn on the backend servers, or you’ve exhausted the local source ports on the HAProxy box, or the backend’s listen queue is full. Diagnose: stats page → check cur vs max for the affected backend and individual server connection counts. Run sysctl net.ipv4.ip_local_port_range to see the port range, and ss -s to see how many connections are open. Fix: enable http-reuse to amortize backend connections; increase maxconn on the server lines if the backends can handle more; widen the local port range (net.ipv4.ip_local_port_range = 1024 65535); or scale out the backend.

Slow responses, but no errors

Symptoms: clients report slowness; HAProxy logs show 200s with high Tr (response wait) values. Root cause: backend is slow but not broken. The two-pipe model is honestly telling you where time is being spent — Tr is exactly “time waiting for the backend’s first byte after we wrote the request.” Diagnose: parse out the timing fields from logs. If Tw (queue wait) is high, the backend’s maxconn is too low and requests are queueing in HAProxy. If Tr is high, the backend itself is slow. If Tt (total) is far higher than Tr, response generation is slow on the backend. Fix: depends on where the time is. Slow Tr = look at the backend application. High Tw = increase server-level maxconn or add servers. None of these are HAProxy’s fault — it’s just measuring.

Periodic connection drops at exactly the reload interval

Symptoms: a small percentage of connections fail right around when you reload HAProxy. Root cause: under high load (thousands of new connections per second), there’s a few-millisecond window during reload where socket handoff can drop connections. Typical rate observed is 1 failed connection per 10,000 new connections per second per reload. Diagnose: correlate failure spikes with systemctl reload timestamps. If they line up, this is your problem. Fix: make sure expose-fd listeners is set on your stats socket (it’s required for proper hitless reload); set hard-stop-after to a reasonable value (10 minutes) so old processes don’t pile up; if reload frequency is high, consider whether you can batch config changes.

Memory grows over time

Symptoms: HAProxy memory usage climbs slowly across hours/days. Root cause: usually one of three things. (1) Old HAProxy processes from previous reloads still serving long-lived connections — check with ps -ef | grep haproxy for multiple PIDs. (2) Stick tables growing — sized too large, or your expire is too long. (3) The SSL session cache (tune.ssl.cachesize) sized too large. Diagnose: echo "show pools" | socat stdio /run/haproxy/admin.sock shows pool memory; echo "show table" shows stick table sizes; check for old processes. Fix: set hard-stop-after 10m so old processes drain in bounded time; right-size stick tables (a size 1m table can use hundreds of MB); reduce SSL cache size if you’re not benefiting from it.

TLS handshake failures or slowness

Symptoms: clients see “SSL_ERROR” in browsers, or handshakes take seconds. Root cause: usually CPU saturation. TLS handshakes are the most expensive operation HAProxy does. A single core does ~1500 RSA-2048 fresh handshakes per second. If you’re handling thousands of new HTTPS connections per second with no session resumption, you’ll saturate. Diagnose: top will show HAProxy threads at 100% CPU. Stats page may show acceptable RPS but extremely high CPU. Check tune.ssl.cachesize and tune.ssl.lifetime — if session caching isn’t enabled or expires too fast, every connection is a fresh handshake. Fix: ensure TLS session caching is enabled (tune.ssl.cachesize 100000 typical); use ECDSA certificates if possible (faster than RSA); make sure clients are reusing connections (HTTP/2 helps enormously); scale to more cores.

”Server is up but no traffic”

Symptoms: a backend server shows UP in stats but receives no requests. Root cause: usually weight 0, or the server is in drain state, or the load balancing algorithm + sticky session combo is unintentionally pinning everyone elsewhere. Diagnose: stats page → check the server’s weight and state. Check the stick table contents. Fix: set server backend/srv weight 100 via the socket, or set server backend/srv state ready. If it’s a sticky-session issue, clear the relevant stick table entries.

The general debugging workflow

When something is wrong and you don’t know what:

Open the stats page first. Are all servers green? Where are the queues backed up? What’s the connection count vs maxconn?
tail -f the HAProxy log. What does the termination state field say (the four characters like ---- or sH-- or SC--)? Uppercase letters here tell you which side gave up and why.
Check the timing fields. Tw/Tc/Tr/Tt tell you exactly where time is going.
Probe directly. curl your backend servers from the HAProxy box. Is the issue HAProxy, the backend, or the network in between?
show errors from the socket. HAProxy captures the most recent erroring request and response — invaluable for debugging weird protocol issues.
Look at the kernel state. ss -s, dmesg, sysctl for the relevant TCP parameters. HAProxy is often blamed for what is actually a kernel limit.

That’s roughly the order. Almost every HAProxy incident I’ve ever debugged has been answerable from those six steps.

11. The Downsides / Disadvantages

The honest accounting. HAProxy is genuinely one of the best load balancers ever written — and these are the structural costs you pay for choosing it.

Downside 1: The config language ages, hard

The HAProxy config file format is a strange little language. It’s not YAML, not JSON, not a real DSL — it’s an unstructured sequence of directives with positional arguments, terse keywords, and a syntax that has accreted across a quarter-century. Twenty-five years of additions have left it with three different ways to do almost everything, multiple deprecated-but-still-supported forms, and fetches/match-types/ACL syntax that read like APL. http-request set-header X-Foo %[req.hdr(X-Bar),lower,regsub(\.,_,g)] is normal. There is no formatter, no linter that catches semantic mistakes, no language server worth its name. Reading someone else’s HAProxy config is genuinely hard, and writing a large one without a style guide produces an unreadable artifact.

Where it comes from: HAProxy was a side project that grew, never had a config-language redesign, and prioritizes backwards compatibility absolutely. Every old config still works on new versions, and that’s a wonderful operational property — but it freezes the language.

What it costs you: hours of grep’ing through docs for the right fetch name; mistakes that pass syntax check but do the wrong thing semantically; configs that the original author can’t read six months later. Newer tools (Traefik, Caddy) have far gentler config experiences.

When this is a dealbreaker: small teams that need many engineers to be able to read and change the config quickly. Pick Traefik or Caddy. When it’s livable: dedicated infra/SRE teams who write HAProxy configs as a primary skill.

Downside 2: Dynamic configuration is bolted-on, not native

HAProxy was designed around a static config file, parsed at startup, immutable for the life of the process. The Runtime API and Data Plane API exist to give you some dynamic control — you can change weights, drain servers, update ACL files, add/remove entries from maps — but you cannot add a new backend, add a new frontend, change a bind, or restructure your routing without writing a new config and reloading.

Where it comes from: the event-loop architecture and the assumption that config is immutable. Dynamic config would require either restarts (costly) or a complex coordination protocol within the running process (architecturally invasive).

What it costs you: every meaningful topology change is a config-file edit, a validate, and a reload. In a containerized world where backends come and go every minute, this becomes painful — HAProxy Kubernetes Ingress Controller and Data Plane API are workarounds, but they’re sitting on top of an architecture that fundamentally wants to be static. Tools designed for dynamic environments (Traefik, Envoy) have a much more natural fit.

When this is a dealbreaker: heavy Kubernetes or service-mesh use, where the topology genuinely changes every minute. When it’s livable: traditional infrastructure where backend pools change a few times a day at most.

Downside 3: TLS performance is good but not best-in-class

HAProxy’s TLS handling is via OpenSSL, and while it’s optimized about as well as anyone can optimize OpenSSL, there are competitors that are faster. NGINX, in some benchmarks, terminates SSL at significantly higher rates than HAProxy at equivalent CPU. The historical reason is that HAProxy does TLS handshake work in its main event loop; NGINX has historically had a more efficient SSL state-machine integration. HAProxy Technologies is transitioning to AWS-LC (a forked BoringSSL) for performance reasons.

Where it comes from: HAProxy’s general-purpose event-loop architecture vs. NGINX’s more specialized SSL handling code.

What it costs you: in pure HTTPS-termination workloads at massive scale, you may pay 20–50% more CPU than NGINX for the same handshake throughput. For most workloads, this is well-amortized by session caching and HTTP/2 — and HAProxy’s overall request throughput remains higher than NGINX once you factor in HAProxy’s faster non-SSL request path. But if your entire job is “terminate millions of fresh TLS handshakes per second,” HAProxy is not the unambiguous winner it is for load balancing.

When this is a dealbreaker: pure TLS-termination workloads at the absolute scale where CPU cost dominates everything else (CDN origin layers, etc.). When it’s livable: virtually everywhere else.

Downside 4: Observability is mostly logs and the stats page

For a tool that handles your most production-critical traffic, HAProxy gives you remarkably few introspection surfaces. You get: logs (excellent — once you learn to read them), the stats page (good for live status, weak for historical), the Runtime API (control + a few queries), and a Prometheus exporter (good but not deep). What you don’t get: built-in distributed tracing, request body inspection, per-request span emission, or rich metrics about ACL evaluation cost.

Where it comes from: HAProxy is performance-obsessed and refuses to instrument hot paths where it would cost cycles. Observability that exists has to be near-free.

What it costs you: when you need to understand “why is this one user’s request slow,” you reach for HAProxy logs and find them — but you do not get the rich middleware visibility that Envoy provides natively. Modern observability stacks (OpenTelemetry, Datadog APM) integrate with HAProxy more grudgingly than with Envoy or Istio.

When this is a dealbreaker: service-mesh environments where every hop must emit distributed-tracing spans. When it’s livable: edge load balancing where logs and metrics are enough.

Downside 5: Multi-tenancy is poor

If you want to give different teams ownership of different parts of an HAProxy config — Team A manages routing to their services, Team B manages theirs, neither can see or break the other — you can’t. The config is one file (or a set of included files, which is better but still globally validated). There’s no namespace, no RBAC at the config level, no “Team A can only edit their backend definitions.” Either everyone has full edit access or no one does.

Where it comes from: HAProxy was designed for a single operator (or operator team) managing a load balancer for a fleet. The Kubernetes-era expectation of self-service is just not in the design.

What it costs you: organizationally, HAProxy becomes a bottleneck — every team wanting a routing change opens a ticket with the infra team. The alternative is to give every team root on HAProxy, which is its own problem. Tools designed for the multi-tenant world (Ingress controllers backed by Envoy, service meshes) handle this natively.

When this is a dealbreaker: medium-to-large organizations with many product teams wanting independent control over their traffic. When it’s livable: small orgs, or larger ones with a dedicated SRE/platform team that owns the load balancer.

Downside 6: HTTP/3 and QUIC support is recent, less battle-tested

HAProxy added QUIC and HTTP/3 support relatively late (2.6+ era) and it’s solid but newer than the rest of the codebase. NGINX similarly was late. The places that have been doing QUIC longest in production are CDNs running custom stacks and Envoy. HAProxy’s QUIC implementation works, but if you’re an early adopter at the bleeding edge of HTTP/3 traffic, you’re more likely to find quirks in HAProxy than in tools that grew up with it.

Where it comes from: HAProxy is conservative about new protocols and prefers to add them once specifications stabilize.

What it costs you: occasional edge-case bugs in QUIC handling; less community wisdom on tuning HAProxy specifically for HTTP/3.

When this is a dealbreaker: streaming-media or mobile-heavy workloads where HTTP/3 is critical and you need a battle-hardened implementation. When it’s livable: anywhere HTTP/2 over TCP is still acceptable.

Downside 7: Documentation is comprehensive but daunting

The HAProxy configuration manual is genuinely complete — every directive, every fetch, every match type, every option. It is also several hundred dense pages, organized by feature rather than by task, with limited examples per directive. The community-maintained guides are good but partial; HAProxy Technologies’ blog is excellent but advertorial. There is no “Programming HAProxy” O’Reilly book that walks a beginner through the full landscape with taste. You learn HAProxy by writing it, breaking it, and grepping the manual.

Where it comes from: HAProxy is maintained by a small core team focused on code; documentation gets the time that’s left.

What it costs you: new engineers take 6–12 months to become genuinely fluent. The learning curve is real and steeper than for tools with rich tutorial ecosystems.

When this is a dealbreaker: short-staffed teams that can’t invest in deep HAProxy expertise. When it’s livable: teams that have HAProxy as a primary technology and treat it as a multi-year skill investment.

Stick tables are in-memory in the HAProxy process, replicated to direct peers. This works perfectly for an active/passive pair (or a small cluster). It does not scale to “I have 20 HAProxy nodes across three regions and they all need to share session state and rate-limit state.” There’s no central store; there’s no consistent hashing across N peers.

Where it comes from: stick tables were designed as a fast in-memory store. Their replication is push-based gossip between known peers, not a distributed consensus.

What it costs you: large multi-region deployments need either a different state store (Redis, etc.) for cross-region state, or to accept regional isolation of rate limits and stickiness. There’s no HAProxy-native answer.

When this is a dealbreaker: global active/active deployments needing globally-consistent rate limiting or session state. When it’s livable: regional or single-cluster setups, or those where cross-region inconsistency is acceptable.

Downside 9: The “free” operational cost is real

HAProxy is free as in beer. Running it well is not free as in time. You need: a person who actually understands the config language. A monitoring setup that watches the stats page. Logging infrastructure that ingests HAProxy’s syslog firehose. Health-check endpoints in your backends that don’t suck. A reload mechanism that ties into your deployment system. Certificate automation. Failover via keepalived or similar.

In an alternate universe, your team picks an AWS ALB, pays Amazon $20/month, and gets a managed load balancer with zero operational burden. The ALB is less featureful, less fast, less customizable — but it is genuinely zero ops.

When this is a dealbreaker: small teams with no dedicated infra/SRE. When it’s livable: organizations with capable infra teams and traffic volumes where the cost savings of self-managed HAProxy justify the operational overhead.

Downside 10: It is, and will remain, a pure proxy

HAProxy is not a web server, not an API gateway in the strong sense (no JWT validation primitives, no API quota management, no plug-and-play OAuth flows), not a service mesh, not a static file server. It can be made to do API-gateway-like things with enough rules, Lua, and SPOE — but you’re stacking features onto a thing that wasn’t built for them. If your actual need is “API gateway with rate limiting, JWT validation, transformation, and multiple authentication strategies,” tools like Kong or Envoy-based gateways are built for that and HAProxy will feel like fighting upstream.

Where it comes from: HAProxy is fanatically focused on its core job. Refusing to expand scope is what’s kept it fast.

What it costs you: when your needs grow into the “we need API-gateway features,” you’ll likely add another layer in front of or behind HAProxy. HAProxy will be one piece of a stack, not the whole thing.

When this is a dealbreaker: when you need a one-stop API gateway. When it’s livable: when you accept HAProxy’s narrow job and pair it with other tools for the rest.

12. The Taste Test

What does good HAProxy usage look like? What signals do experienced operators look for in a config?

Good signs

One file, organized top-down: global → defaults → frontend(s) → backend(s) → stats endpoint. Sections clearly separated by comments. ACLs grouped at the top of each frontend.
defaults section sets all four timeouts explicitly. No magic numbers in individual frontends or backends.
Health checks are HTTP-level, hitting a real endpoint, with http-check expect status 200 or similar. Not just bare check.
Sticky sessions use cookies (with insert indirect nocache), not source-IP hash. If source-IP, only with explicit justification.
option redispatch in defaults. Always. Without exception, in production.
TLS configured with explicit cipher suite list, ssl-min-ver TLSv1.2, multiple certs in a directory for SNI.
X-Forwarded-For is deleted first at the edge, then added via option forwardfor. The author thought about XFF spoofing.
Stats endpoint exists, bound to an internal IP or behind auth, with admin enabled only from LOCALHOST.
maxconn set on individual server lines, not just globally — protects each backend from being overwhelmed.
Stick tables explicitly sized and expired: stick-table type ip size 1m expire 30m, with clear comments about what they store and why.
Reload integration with the deployment system: someone has thought about expose-fd listeners, hard-stop-after, and the runtime socket.
Comments. Real comments. Explaining why a particular setting was chosen, not just what it does. (# 60s timeout — slow analytics endpoint can take ~45s p99)

Bad signs

listen blocks everywhere instead of frontend/backend. Fine for tiny configs, but a smell when there are multiple backends — usually means someone wanted simple syntax and accepted lossy structure.
No health checks at all (server web1 10.0.0.1:80 with no check). HAProxy is treating dead servers as alive. Disaster waiting.
Default 2-second inter health checks on a fleet of 100 servers. That’s 50 health checks per second, hitting / (the default URL), often hitting the application’s heaviest endpoint. Common, devastating.
balance source for public web traffic. Almost always a sign of someone who didn’t understand cookie persistence.
No default_backend and overlapping use_backend ACLs. Eventually a request will fall through and silently get a 503.
Timeouts that are wildly miscalibrated: timeout server 5s for a backend that occasionally takes 30 seconds. Or timeout client 30m “to be safe” — which means HAProxy holds idle slowloris connections forever.
option httpclose in modern configs. It was a 2009-era recommendation. With modern HTTP, you want option http-keep-alive (which is the default).
No nbthread configured on a multi-core machine. Recent HAProxy versions default sanely, but older configs still run on one thread by default.
A massive single file with hundreds of backends and no comments. Cognitive debt accumulating.
Stats endpoint on *:8404 with no auth, no IP restriction. Anyone on the network gets a free topology map.
http-reuse always to backends that the author didn’t write. Trusting third-party backends’ keep-alive handling is a bet.
Hardcoded IPs in server lines for hosts that have DNS names. No service discovery, no failover beyond check.

Side-by-side: beginner vs experienced

A beginner’s config might look like:

frontend front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/cert.pem
    default_backend back

backend back
    server srv1 192.168.1.1:80
    server srv2 192.168.1.2:80
    server srv3 192.168.1.3:80

It works! It load-balances three servers. It also: has no timeouts (will refuse to start in newer versions), no health checks (dead servers stay in rotation), no http-request redirect to HTTPS, no XFF, no defaults section, no stats. Every problem is invisible until production breaks.

An experienced operator’s version:

global
    log /dev/log local0
    maxconn 50000
    nbthread 4
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    tune.ssl.default-dh-param 2048
    hard-stop-after 10m

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option redispatch
    option forwardfor                                 # adds XFF; del-header strips spoofed ones below
    retries 3
    timeout connect 5s
    timeout client 60s
    timeout server 60s
    timeout http-request 10s
    timeout http-keep-alive 30s
    timeout queue 30s
    timeout check 5s

frontend public_https
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    http-request redirect scheme https code 301 unless { ssl_fc }
    http-request del-header X-Forwarded-For          # strip client-supplied XFF
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

    # Rate limit by IP — 100 req per 10s = 10 RPS sustained
    stick-table type ip size 1m expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

    acl is_api path_beg /api/
    use_backend api_servers if is_api
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    cookie SERVERID insert indirect nocache httponly secure
    # 5s checks; fall-3 = 15s detection; rise-2 = 10s recovery; per-server maxconn protects from overload
    server web1 10.0.1.10:8080 check inter 5s fall 3 rise 2 maxconn 1000 cookie s1
    server web2 10.0.1.11:8080 check inter 5s fall 3 rise 2 maxconn 1000 cookie s2

backend api_servers
    balance leastconn                                  # API request times vary; leastconn balances better
    option httpchk GET /api/health
    http-check expect status 200
    server api1 10.0.2.10:8080 check inter 5s fall 3 rise 2 maxconn 500
    server api2 10.0.2.11:8080 check inter 5s fall 3 rise 2 maxconn 500

frontend stats
    bind 127.0.0.1:8404                                # localhost only
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

Same load balancer. Profoundly different posture. The second one has anticipated failure, attack, observability, and operational maintenance. Reading them side-by-side teaches more about HAProxy than any tutorial.

13. Where to Go Deeper

Curated, opinionated.

The HAProxy Configuration Manual (https://docs.haproxy.org/) — the canonical reference. Use the version-specific URL for your version. Not designed for cover-to-cover reading; designed for “I need the exact semantics of this directive.” Bookmark the fetches and converters reference; you’ll come back to it constantly.
HAProxy Technologies’ blog (https://www.haproxy.com/blog/) — the company behind HAProxy publishes deeply technical, well-written articles. The pieces on stick tables, hitless reloads, http-reuse, and the PROXY protocol are essentially required reading. Filter to authors like Nick Ramirez and Willy Tarreau directly.
“Truly Seamless Reloads with HAProxy” by Willy Tarreau (HAProxy blog, 2017) — the creator’s own history of how HAProxy got zero-downtime reload right, with the false starts and dead ends. Reads like a war story. Teaches you how the architectural decisions actually got made.
The HAProxy GitHub Issues (https://github.com/haproxy/haproxy/issues) — a working operator’s lab notebook. Real production failures, real fixes. Searching here for symptoms you’re encountering often saves hours.
“HAProxy Cookbook” — there isn’t an official one as of writing, but the HAProxy Tech blog’s tutorial series functions as one. Walk through their “Using HAProxy as an API Gateway” multi-part series for a worked example of advanced features.
Willy Tarreau’s conference talks — searching YouTube for his name surfaces talks from HAProxyConf, USENIX, and Linux Plumbers Conference. He explains the internals with first-principles clarity. The talks on multi-threading and the architecture evolution are especially illuminating.
Operational guides for keepalived + VRRP — once you have a single HAProxy working, the next mountain is HA. The keepalived documentation and a few production-grade tutorials on the active/passive pattern are the natural next step.
Build a hands-on project: spin up three small VMs (or containers); put a basic web service on each; install HAProxy on a fourth; configure HTTPS termination, ACL routing on path, cookie-based stickiness, rate limiting, the stats page; intentionally break things — kill a backend, edit the config wrong, send a SIGKILL — and watch what happens. You’ll learn more in a week of doing this than in a month of reading.

14. The Final Verdict

HAProxy is the most quietly excellent piece of infrastructure software in widespread use. It is what you reach for when you actually care about the load balancer — when “good enough” isn’t, when the cost of dropping connections is real, when you need to be able to predict latency to the millisecond under load. It has been used in the highest-stakes production environments in the world for over two decades, and the reason it survives generation after generation of “load balancer killer” projects is that none of them have actually been better at the job HAProxy does. Faster on some axis, easier on some axis, more dynamic on some axis — yes. Better at being a load balancer? Not really.

What it gets profoundly right: the architecture is honest about what a load balancer is, and that honesty pays dividends everywhere. The event-loop model means it handles slow clients gratis. The two-pipe model means client-side and backend-side concerns are properly separated. The declarative rule-chain configuration means a config that worked in 2010 still works in 2026 and the behavior is auditable. The PROXY protocol — invented by HAProxy Technologies — solved a real problem so well that it became an industry standard. And critically, HAProxy refuses to grow features that would compromise its core: there is no “easier” config language, no API-gateway sprawl, no service-mesh ambition. The narrowness is the point.

What it costs you: the config language is a 25-year-old artifact, dynamic reconfiguration is bolted on, multi-tenancy is essentially absent, and the operational learning curve is real. None of these are fatal, but they are real, and they show up most painfully in modern cloud-native workflows where everything is supposed to be self-service and dynamic. If your worldview is “infrastructure should be invisible and autoconfigured,” HAProxy will frustrate you. It is infrastructure that demands an operator who takes it seriously.

Who should reach for it: SRE/platform teams running serious production traffic at scale (thousands of RPS and up); anyone whose load balancer is in a regulated or latency-critical path; teams running on bare metal or VMs (not pure Kubernetes); anyone replacing a hardware load balancer (F5, A10, NetScaler) where feature parity matters; anyone whose existing NGINX-as-load-balancer setup is creaking and needs more sophisticated routing or stickiness.

Who shouldn’t: small teams with no dedicated infra/SRE function (use a managed load balancer); pure-Kubernetes shops where Ingress controllers and service meshes give you better ergonomics for free (Traefik or Envoy-based options will feel more natural); anyone whose actual need is “API gateway” with JWT validation, quotas, and transformation pipelines (use Kong, Envoy, or a real API gateway product); anyone whose traffic is so dynamic that the static-config model would mean reloading every minute.

What to believe after all this:

Believe that HAProxy is, in the narrow domain of “L4/L7 load balancing,” the gold standard. The benchmarks back this up, the production track record backs this up.
Don’t believe that “HAProxy or NGINX” is a meaningful question. They do overlapping but distinct jobs; a serious stack often uses both.
Don’t believe that switching to a newer tool buys you HAProxy’s performance “for free.” Traefik, Caddy, even Envoy in many configurations are slower, sometimes by factors of two or three. They buy you other things — dynamic config, multi-tenancy, observability — but performance is HAProxy’s home turf.
Believe that when someone says “HAProxy is hard to learn,” they mean it took them six months, and they probably stopped just before the moment it would have started feeling easy.

The hard-won line: HAProxy is not the tool you reach for when you want load balancing to be easy. It is the tool you reach for when you have decided load balancing is important enough to be done right, and you are willing to pay the cost in operator skill to make that decision pay off. Whether that trade is worth it is a real question. But once you’ve made it, you’ll never wonder if your load balancer is the problem — and that, after a few 3 AM incidents pointed at the right backend instead of the wrong one, is a kind of operational peace that’s hard to put a price on.

The ideas are mine. The writing is AI assisted