There is a specific class of container failure that every self-hoster meets exactly once and never forgets: the image starts, logs a bootstrap check, and exits with
| |
Nothing is out of memory. free -h shows gigabytes free. The CPU is idle. The container simply refuses to run — and if you have ever “fixed” it by adding a compose line and moved on, you never found out that the same kernel limit is also what kills thread-heavy services with a completely unrelated-looking error.
The reason this bites is that the same image gets wildly different ceilings depending on the host. On the box that produced this article:
| |
That is 1,048,576 — sixteen times the 65,530 that Debian 12, RHEL 8/9 and most container base images still ship. Ubuntu 24.04 raised the default in 2024 precisely because of this class of bug. So a compose file that works on one host fails on another with byte-identical configuration, and the error message blames your application.
TL;DR — What To Do in 60 Seconds
- If a search, cache or database container refuses to start with a
max_map_counterror: raise the sysctl on the host (it is not namespaced), then restart the container.vm.max_map_countis per-process and global to the kernel, sodocker run --sysctlonly works where the runtime allows it and Kubernetes needs an explicit allowlist. - If a thread-heavy service dies with
pthread_create: EAGAINwhilepsshows few processes: it is not the process table — it is mapping exhaustion from thread stacks and allocator arenas. Same sysctl, different symptom. - Do not “fix” it with a huge number and walk away. 1,048,576 is a ceiling, not a target; the real work is finding which process is at 21,000 mappings and why.
- Leave
vm.overcommit_memory=0unless something specific requires otherwise. Mode 0 is heuristic overcommit and is what makesCommitted_ASlegally exceedCommitLimit— which is normal, not a bug, and switching to mode 2 on a production host is how you cause an outage instead of preventing one.
What a “Memory Area” Actually Is
Every line in /proc/<pid>/maps is one vm_area_struct in the kernel — one discrete mapping. They are created by far more than mmap() calls:
| Source of mappings | Approximate cost | Typical offender |
|---|---|---|
| Thread stacks | 1–2 mappings per thread | Java services (-Xss × threads), .NET, Rust thread pools |
| glibc malloc arenas | up to 8 per core × 64 MB | multi-threaded C/C++ services, Python with many threads |
mmaped files | 1 per file | LMDB/MDBX databases, memory-mapped indexes |
| Shared libraries | 4–6 mappings per .so | anything with heavy plugin loading |
| Anonymous allocations > 128 KB | 1 per allocation | jemalloc/tcmalloc workloads, columnar caches |
| Huge pages / GPU buffers | 1 per region | database buffer pools, compute nodes |
Two consequences follow, and they explain why the same sysctl produces such different failures:
- A thread is not free. Raising thread counts raises mapping counts, and a 64-core box with glibc’s default arena policy can burn thousands of mappings before the application allocates a single byte of useful data.
- Map count is orthogonal to memory usage. You can be at 90% of the mapping ceiling while using 2% of RAM. That is why every instinct trained on OOM killers is wrong here.
Memory-mapped databases are the extreme case. LMDB-based stores map the entire database file into the address space, and each reader thread may hold its own read transaction view — so a 200 GB LMDB environment with dozens of readers can consume tens of thousands of mappings while resident memory stays modest.
Real Mapping Counts From a Live Host
Here is why this matters in practice. Sampling the busiest processes on this host:
| |
A single userland process is holding 21,020 mappings. On this host that is 2% of the ceiling and utterly harmless. On a Debian 12 host with max_map_count=65530 it would be 32% of the entire budget for one process — and if that process spawns worker threads or opens a memory-mapped index, it will hit the wall, print Out of memory (misleadingly), and die.
That is the whole trap in one number. The limit is not about bytes; it is about the number of distinct regions, and modern runtimes create them profligately.
Diagnosing Which Process Is Eating Mappings
Start with a census, not with free:
| |
Then break one process down by mapping type, which usually reveals the cause immediately:
| |
Read the result like this:
anondominates → allocator arenas and large anonymous allocations. Check thread count and glibc arena settings.thread-stackis large and grows with load → thread-per-connection design; either cap threads or shrink stacks.filedominates → a memory-mapped index or LMDB environment. This is the classic search/cache case and it is the one that legitimately needs a higher ceiling.shared-libis enormous → hundreds of plugin.sofiles; containerizing that as one process is the design problem.
Two more views worth knowing:
| |
VmPeak in the tens of gigabytes with a VmRSS of hundreds of megabytes means you are mapping, not using — virtual address space is cheap and effectively unlimited on 64-bit, which is exactly why the kernel caps map count instead of map size.
The Overcommit Question (And Why Mode 0 Is Usually Right)
vm.max_map_count is only half of the address-space story. The other half decides whether an allocation is allowed at all:
vm.overcommit_memory | Behavior | What it means in practice |
|---|---|---|
0 (heuristic, default) | Kernel estimates; allows generous overcommit | Committed_AS may exceed CommitLimit; almost nothing is refused up front |
1 (always) | Never refuse an allocation | Required by some fork-heavy designs (Redis background save legacy advice) |
2 (strict) | Refuse anything beyond CommitLimit | Turns overcommit into hard accounting; can convert a benign spike into ENOMEM |
This host runs mode 0, and the numbers show exactly what that permits:
| |
Committed memory (30.8 GB) exceeds the commit limit (22.6 GB) — on a 28.4 GB machine whose overcommit_ratio is 50. In mode 0 this is expected: the limit is a heuristic input, not a wall. Systems that monitor Committed_AS against CommitLimit and page on the ratio will page constantly on a perfectly healthy host. If you inherit such an alert, fix the alert before you touch the sysctl.
Switching to mode 2 (overcommit_memory=2) on a host in this state is actively dangerous: the next allocation that pushes committed memory past CommitLimit is refused immediately, which is how you turn a routine traffic spike into application-level ENOMEM failures across every service at once. Mode 2 is for appliances with precisely modeled workloads, not for general-purpose self-hosted boxes.
Fixing It: Host, Container, Kubernetes
Because vm.max_map_count is a global, non-namespaced sysctl, the fix has to land on the host kernel — but how you deliver it depends on the platform.
Bare metal / VM host (persistent):
| |
| |
Docker Compose — set it per container where the runtime permits, but understand it is a host-level value:
| |
Kubernetes — vm.* sysctls are classified as unsafe, so a plain securityContext.sysctls entry is rejected unless the kubelet explicitly allows it:
| |
With the kubelet flag --allowed-unsafe-sysctls=vm.max_map_count, a pod-level securityContext.sysctls entry becomes legal instead — but that flag is cluster-wide and changes the threat model for every workload, so most operators prefer the DaemonSet or a node bootstrap template.
Sizing the Value Instead of Guessing
Do not pick a number because a blog post said so. Derive it:
| |
Then size to 4–8× the worst observed peak, with a floor of 262,144 for search/cache workloads. Rationale: mapping counts scale with thread count and shard/part count, both of which grow with traffic and data, and hitting the ceiling is a hard, immediate failure rather than a graceful degradation. A value of 1,048,576 gives ~50× headroom over the 21,020-mapping process measured above, which is roomy but not absurd.
Pitfalls and Migration Notes
vm.max_map_countis not namespaced. A container cannot have its own value;--sysctlworks only because the runtime writes the host value. Two containers wanting different values cannot both be satisfied.Out of memoryin a thread-creation error is not an OOM.pthread_createreturningENOMEM/EAGAINwith plenty of free RAM is the mapping-ceiling signature. Checkwc -l /proc/$PID/mapsbefore you add memory.- The distro lottery is real. Ubuntu 24.04 ships 1,048,576; Debian 12 and RHEL 8/9 ship 65,530. Container images cannot fix this because the limit comes from the host kernel. Document the required host value next to your compose file.
- Raising the ceiling can hide a map leak. Mappings from
dlopenloops or repeatedmmapwithoutmunmapgrow monotonically. Sample the count twice a minute apart; if it only climbs, you have a leak, not a capacity problem. ulimits.memlock=-1is a separate requirement for search engines that want to lock the heap. Without it,bootstrap.memory_lockfails and the process may swap its index pages.- Changing
overcommit_memoryat runtime does not retroactively validate existing mappings. Existing committed memory above the new limit stays allocated; the next allocation is the one that gets refused — which is why flipping to mode 2 during an incident makes it worse. - Huge pages change the arithmetic — a 2 MB huge-page-backed buffer consumes one mapping per page-pair rather than per allocation, which can actually reduce map counts while increasing committed memory. See our KSM and memory deduplication tuning guide for the adjacent trade-offs.
For the memory-accounting side of the same debugging session — what is resident versus shared versus mapped — read RSS vs PSS vs USS container memory accounting, and for the process-limit side that produces the identical EAGAIN error from a completely different cause, see the PID and thread exhaustion field guide. Our systemd, PAM and cgroup v2 resource limits guide covers the per-service knobs that sit above the kernel ones.
FAQ
What does vm.max_map_count actually limit?
The number of distinct memory mappings (vm_area_struct entries) a single process may create, as listed in /proc/<pid>/maps. It limits count, not bytes — a process can have 20,000 mappings and use only a few hundred megabytes of resident memory.
Why does the same container image work on Ubuntu and fail on Debian? Because the limit is a host kernel setting, not an image setting. Ubuntu 24.04 raised its default to 1,048,576 while Debian 12 and RHEL 8/9 still ship 65,530, so a search or cache container that maps thousands of index regions exhausts the ceiling on the older host and exits during bootstrap.
Should I set vm.overcommit_memory=1 to fix it?
No — those are different knobs. max_map_count limits mapping count; overcommit_memory controls whether the kernel refuses allocations up front. Mode 0 (heuristic) is correct for general-purpose hosts, and switching to strict mode 2 on a machine whose Committed_AS already exceeds CommitLimit will cause allocation failures across every service.
How do I find which process is close to the ceiling?
Count lines in /proc/<pid>/maps for every process and sort descending. Then break one process down by mapping type with awk over maps to distinguish thread stacks, anonymous arenas, memory-mapped files and shared libraries — the dominant category tells you whether to cap threads, fix a leak, or raise the ceiling.
Does docker run --sysctl vm.max_map_count=... really work?
On Docker it writes the host value because the sysctl is not namespaced, so it “works” while affecting every process on the machine. On Kubernetes the vm.* family is classified as unsafe and is rejected unless the kubelet is started with --allowed-unsafe-sysctls=vm.max_map_count — which is why a node-level DaemonSet or bootstrap template is the usual production fix.
💰 想测试你的市场判断力?我用 Polymarket 做预测市场交易——这是全球最大的预测市场平台,从大选结果到技术监管时间线,什么都可以押注。和赌博不同,这是真正的信息市场:你懂的信息越多,胜率越高。我靠预测技术相关事件的走向已经赚了不少。用我的邀请链接注册:Polymarket.com