A client reports intermittent timeouts. You check the box: load average 0.4, CPU 3%, memory fine, application logs full of successful requests, zero errors. The reverse proxy says upstream responded in 12ms. And yet a small, consistent slice of connections never completes — no 502, no 503, just nothing until the client gives up.
If that describes your incident, the culprit is almost never resource starvation. It is one of two kernel queues that sit between the network card and your accept() call, and both are silent by default: when the accept queue overflows, Linux drops the final ACK of the handshake and says nothing to the client. The client retransmits, waits, and eventually times out — while your server reports that it is perfectly idle. It is, in a sense, telling the truth.
Here is what the queues look like on the host that produced this article:
| |
Every one of those Send-Q values is a different application’s idea of how many pending connections it can tolerate — 200 for the database, 2,048 for a Python service, 4,096 for the Caddy listener that simply inherited somaxconn. Nothing on this host is tuned consistently, which is the normal state of the world.
TL;DR — The Two Queues and the One Column That Matters
Send-Qon a LISTEN socket is not a queue depth. It is the backlog ceiling — the value passed tolisten(), clamped bynet.core.somaxconn.Recv-Qon a LISTEN socket is the live accept-queue depth — fully handshaked connections your application has notaccept()ed yet. IfRecv-Qsits atSend-Q, your accept queue is full and clients are being timed out right now.- Two overflow counters prove it after the fact:
TcpExtListenOverflowsandTcpExtListenDrops(visible vianstat -azornetstat -s). - Fix the accept rate first, the backlog second. Raising
somaxconnon a server whose app accepts 50 connections/second under a 5,000/second flood just makes the queue a bigger coffin. - Set
net.ipv4.tcp_abort_on_overflow=1in staging only — it converts silent timeouts into immediate resets, which makes the problem visible without waiting for a client complaint.
The Two Queues, Visualized
Most guides blur these together. They are separate, they have separate limits, and they fail with separate counters.
| Queue | What sits in it | Bound by | Overflow counter | Client experience |
|---|---|---|---|---|
| SYN queue (incomplete) | Half-open connections: SYN received, final ACK not yet | net.ipv4.tcp_max_syn_backlog | TcpExtTCPReqQFullDrop (syncookies off) or TcpExtSyncookiesSent | Slower connect under SYN flood |
| Accept queue (completed) | Handshake finished, waiting for accept() | min(app_backlog, net.core.somaxconn) | TcpExtListenOverflows, TcpExtListenDrops | Silent timeout, no error either side |
| NIC ring | Frames not yet picked up by the driver | ethtool -G rx | rx_dropped / rx_fifo_errors | Retransmits, jitter |
| Softnet backlog | Frames queued for the network stack | net.core.netdev_max_backlog | softnet_stat column 2 | Retransmits, uneven latency |
Only the accept queue produces the classic “idle server, timing-out client” signature — because the kernel’s response to a full accept queue (with tcp_abort_on_overflow=0) is to drop the ACK and rely on the client’s retransmission timer. That is not a bug; it is a deliberate design that assumes the application will catch up within a retransmit interval or two. When it does not, the client sees a 1s/3s/7s backoff and then a timeout.
Proving It Happened: The Counters
Run this first. It answers the question in one shot, without touching production config:
| |
Real output from this host:
| |
Interpretation matters more than the numbers:
ListenOverflows = 0means the accept queue never filled on this host. Good — but this is a quiet box. On a busy one, any non-zero value that keeps climbing with traffic is proof of overflow.TCPDeferAcceptDrop = 81,009is the interesting one.TCP_DEFER_ACCEPTmakes the kernel wait for the first byte of data before waking the application; connections that handshake but never send data (health probes, port scanners, load-balancer checks that connect and immediately close) get dropped and counted here. It looks alarming and is usually benign — but a sharp rise in this counter alongside risingListenOverflowsmeans your accept loop is being woken constantly by junk connections and starved of time for real ones.
Rate-based reading is what you want. nstat shows per-second deltas when run without -a:
| |
The Clamp That Confuses Everyone
The backlog you configure in your application is a request, not a guarantee. Since Linux 4.x the kernel uses:
| |
So with somaxconn = 4096 (this host), Caddy listening with a 4096 request gets 4096. But raise somaxconn to 65535 and Caddy keeps accepting at 4096 because it passes 4096 to listen() — only the somaxconn side of the min() changed. This is why “I raised somaxconn and nothing happened” is such a common report.
The table below shows the actual per-application defaults, which is what you are really living with:
| Service | How backlog is set | Typical effective value |
|---|---|---|
| Caddy | no knob; uses somaxconn | 4096 (as observed) |
| nginx | listen ... backlog= (default 511) | clamped by somaxconn |
| Gunicorn | --backlog (default 2048) | 2048 |
| Redis | tcp-backlog (default 511) | 511 |
Node.js http | server.listen({backlog}) (default 511) | 511 |
Go net/http | not configurable; reads somaxconn | 4096 |
| systemd socket activation | Backlog= (default 4096) | 4096 |
| PostgreSQL | derived internally from connection limits | 200 (as observed) |
Verify rather than assume — ss prints the resolved value, not the requested one:
| |
Reading Recv-Q Live (The Smoke Test)
If you only take one command from this article, take this one. It distinguishes “queue full” from “app slow” in a single sample:
| |
Rules of thumb:
Recv-Q = 0always → nothing is queuing; look elsewhere (softnet, qdisc, upstream).Recv-Qbriefly spikes → normal burst absorption; the app is keeping up.Recv-Qpinned atSend-Q→ accept queue saturated. Each additional connection is dropped after handshake. This is your incident.Recv-Qgrows whileSend-Qstays large → the ceiling is fine but the app is not accepting fast enough; the next step is profiling the event loop, not increasing the backlog.
The deeper per-socket view includes the drop counters that ss -lnt omits:
| |
Why the App Is Slow to Accept
Backlog overflow is a symptom. The causes, in rough order of frequency on self-hosted boxes:
- Blocking work inside the accept path. A connection handler that does a synchronous DNS lookup, a Redis
KEYS, or afsyncbefore returning toaccept()stalls the whole loop. Every millisecond of that stall is queue growth. accept()in a supervisor that spawns processes. Process-per-connection designs (PHP-FPM withpm=dynamic, some CGI setups) are bounded by process count, not by the socket backlog. The queue fills whilefork()work is in flight.- Keep-alive starvation. A high
keepalive_requestscombined with a small worker pool means workers are held by idle keep-alive connections while new connections queue. Loweringkeepalive_timeoutoften fixes “random” timeouts that no backlog change ever fixed. - Health-check storms. Load balancers probing every 2 seconds across many backends can consume a measurable share of the accept queue.
TCPDeferAcceptDropcounts the ones that connect and send nothing. - Single-threaded event loops doing CPU work. A JSON serialization of a large payload on the same thread as
epoll_waitis a queue-growth generator.
Configuration That Actually Fixes It
Step 1 — raise the ceiling (kernel side), sized to what the app can drain:
| |
| |
Step 2 — raise the application’s request to match, or the clamp eats the change. For nginx:
| |
For Gunicorn:
| |
For Redis:
| |
Step 3 — make overflow visible in staging, so you never again learn about it from a customer:
| |
With this set, clients get a fast connection reset instead of a 30-second timeout. That is a worse user experience and a much better diagnostic — which is exactly the trade you want while tuning, and exactly the trade you do not want in production.
Step 4 — alarm on the counters, not on latency. Scrape these and alert on any increase:
| |
A rate of zero is the only acceptable steady state for a healthy listener.
Pitfalls and Migration Notes
Send-Qis not “queue is backed up.” On LISTEN sockets it is the backlog value. Reading it as occupancy produces the wrong conclusion 100% of the time.somaxconnalone never fixes an app that accepts slowly. It converts a fast, visible failure into a slow, invisible one.- Inside containers, the listener’s backlog lives in the container’s network namespace, but
somaxconnis read from that namespace’s sysctls — which may differ from the host’s.docker run --sysctl net.core.somaxconn=8192is often required, and Kubernetes pods cannot always set it (it is namespaced, but some runtimes reject it) — in that case the host value wins. - Userspace proxies add a second queue. Docker’s
docker-proxyand many CNI plugins terminate connections in userspace and open a new connection to the backend with their own default backlog. You can tune the backend perfectly and still overflow at the proxy. - Syncookies mask SYN-queue overflow but degrade connections. With
tcp_syncookies=1, overflowed SYNs are answered with a cookie that cannot carry window-scaling and SACK options; those connections still work but are slower. WatchTcpExtSyncookiesSent— a steady non-zero rate means your SYN backlog is too small, not that you are under attack. systemdsocket activation has its own backlog (Backlog=, default 4096) and it is applied by systemd, not by your app. Tuning the app’slisten()call has no effect when systemd owns the socket.- Enabling
tcp_abort_on_overflow=1in production trades timeouts for resets. Some clients and mobile networks retry RSTs poorly; measure before standardizing on it.
FAQ
What is the difference between Recv-Q and Send-Q for a LISTEN socket?
For listening sockets, Recv-Q is the current number of fully-established connections waiting in the accept queue, and Send-Q is the maximum backlog configured for that socket. On an established (non-LISTEN) socket the meaning flips back to bytes buffered in each direction.
Which limit actually caps my backlog — somaxconn or what I pass to listen()?
Both, because the kernel takes the smaller of the two: min(backlog, net.core.somaxconn). Raising only somaxconn does nothing if the application still requests 511, and raising only the application’s value does nothing if somaxconn is 128.
Why do clients time out but the application log shows no errors at all?
Because with net.ipv4.tcp_abort_on_overflow=0 the kernel silently discards the handshake’s final ACK when the accept queue is full. The application never sees the connection, so it cannot log anything, and the client only notices when its retransmissions exhaust the timeout.
Which counters prove accept-queue overflow?
TcpExtListenOverflows and TcpExtListenDrops, read with nstat -az. TcpExtTCPReqQFullDrop indicates SYN-queue drops when syncookies are disabled, and TcpExtTCPDeferAcceptDrop counts connections dropped for sending no data after the handshake — benign for probes, suspicious when it tracks traffic.
Is a bigger backlog always better? No. A large backlog absorbs bursts, but it also converts an overload into high latency instead of a fast failure, and it hides an accept loop that cannot keep up. Size the backlog from measured peak accept rates, fix the accept path first, and alert on overflow counters so a bigger queue never becomes a silent substitute for capacity.
For the layer beneath this one, see our packet loss diagnosis walkthrough covering the NIC ring, softnet backlog and socket buffers. If your symptom is connect failures rather than slow handshakes, the TIME_WAIT and ephemeral port exhaustion guide covers the client side, and the broader sysctl and tuned tuning reference explains how these values survive reboots. Servers that run out of workers rather than sockets should read our PID and thread exhaustion field guide.
💰 想测试你的市场判断力?我用 Polymarket 做预测市场交易——这是全球最大的预测市场平台,从大选结果到技术监管时间线,什么都可以押注。和赌博不同,这是真正的信息市场:你懂的信息越多,胜率越高。我靠预测技术相关事件的走向已经赚了不少。用我的邀请链接注册:Polymarket.com