A 16 GB node reports 41 GB of resident memory across its containers. It is not swapping, free -m says 9 GB is available, and yet the kernel OOM-killer just executed your API server. Every number on the dashboard is technically real and every one of them is answering a different question. The team then spends a week “fixing a memory leak” that does not exist.
The root cause is almost always the same: RSS double counts shared pages. Every process that maps the same 40 MB library, the same container image layer, or the same tmpfs page adds those 40 MB to its own RSS. Sum RSS across 20 containers built from one base image and you have invented gigabytes of memory that exist exactly once in physical RAM.
This guide is about measurement, not guessing: which metric to read, where the kernel exposes it, what each number includes, and which one to put on an alert. Every command here runs on a stock Linux box.
TL;DR — Quick Verdict
- For “how much RAM does this process really cost me?” use PSS from
/proc/PID/smaps_rollup(orsmem). It charges shared pages proportionally and never double counts. - For “how much memory is this container using right now?” use cgroup v2
memory.currentminusinactive_file, which is the working-set number Kubernetes already bases eviction on. - For “which process should I kill or shrink?” use USS (private memory). It is the only number that drops when that process dies.
- Never sum RSS across processes. Never set a container limit from
docker statswithout checking whether page cache is included. Never compare a cgroup number to afree -mnumber and expect them to add up. - If a Java or Go service is involved: RSS above heap is normal (metaspace, stacks, GC structures, arenas). Establish a baseline before calling it a leak.
The Five Numbers That All Call Themselves “Memory”
| Metric | What it counts | Shared pages | Double counts across processes | Where to read it | Best used for |
|---|---|---|---|---|---|
| RSS (resident set size) | Pages currently mapped and resident | Counted in full in every process | Yes — badly | /proc/PID/status (VmRSS), ps, top | Spotting a single process ballooning; never for totals |
| PSS (proportional set size) | Private pages + shared pages ÷ number of sharers | Split evenly among sharers | No | /proc/PID/smaps_rollup (Pss:), smem | Per-service cost attribution, capacity planning |
| USS (unique set size) | Private pages only | Excluded | No | /proc/PID/smaps_rollup (Private_*), smem -u | “What do I get back if this dies?” |
cgroup memory.current | Everything charged to the cgroup: anon + file cache + shmem + kernel | Charged once, to the owning cgroup | No — but includes reclaimable cache | /sys/fs/cgroup/.../memory.current | Container-level enforcement and limits |
| Working set | memory.current − inactive file cache | Charged once | No | Computed; exposed as container_memory_working_set_bytes in Kubernetes | Eviction decisions, “real” container footprint |
Read that table again with one fact in mind: memory.current includes page cache. A container that reads a 2 GB file once can show 2 GB of “memory usage” while holding almost no anonymous memory, because the kernel is caching the file on that cgroup’s tab. That cache is reclaimable under pressure, which is why the working-set number exists and why memory.current alone is a terrible alerting signal.
Why Summing RSS Is a Category Error
When a process forks, the child gets copy-on-write mappings of the parent’s pages. Both processes report the same physical pages in RSS. When 20 containers run the same 200 MB base image, the shared read-only layers are one copy in the page cache — charged to nobody individually, counted in everybody’s RSS. Add memory-mapped libraries, shared memory segments, and tmpfs pages and the arithmetic stops meaning anything.
| |
That command produces a number larger than physical RAM on a busy host and is the origin of half the “memory leak” tickets in the world. It is not measuring a leak; it is measuring sharing.
The kernel has known this for years. Since Linux 4.14, every process exposes a pre-aggregated summary so you do not need to parse a multi-megabyte smaps file:
| |
| |
Rss: 412 MB versus Pss: 115 MB for the same process. The 300 MB of Shared_Clean is mapped library and page-cache data that 5-30 other processes are also mapping. PSS divides it; RSS duplicates it. Private_Dirty + Private_Clean is USS — the memory that actually returns to the system when this process exits.
Two practical notes on smaps_rollup: it is much cheaper than smaps (one aggregated read instead of a page-table walk producing megabytes of text), so it is safe to poll every few seconds; and it is only available if your kernel has CONFIG_PROC_PAGE_MONITOR, which every mainstream distribution enables.
Tool 1: smem — PSS for Humans
smem reports USS, PSS and RSS side by side, which makes the difference visible in one screen instead of three tools. Install it from your distribution (apt install smem, dnf install smem).
| |
| |
The total line is the point: PSS 1.9 GB, RSS 6.4 GB on the same machine. If you had alerted on summed RSS you would be paging someone about a 6.4 GB footprint that is physically 1.9 GB.
A few flags that matter in practice:
| |
smem needs root (or CAP_SYS_PTRACE) to read other users’ smaps; without it, numbers silently drop processes and totals look reassuringly small. That is the most common way a smem report misleads — run it unprivileged and you measure only your own shell.
smem is a single-file Python script with no daemon and no database, which is exactly why it is still the default answer for “who is actually using the RAM”. Pair it with a real time series if you need history.
Tool 2: ps_mem — The Quick Second Opinion
ps_mem (1,647 stars, last commit 2022) is a 200-line Python script that computes private and shared memory per program and prints them separately:
| |
| |
The layout is the useful part: it makes the shared component explicit instead of hiding it. The caveat is maintenance — the project has not been pushed to since 2022, so on a kernel with unusual smaps formatting (or inside a container with a restricted /proc), verify its output against smaps_rollup before you trust it. It is a good interactive sanity check, not a monitoring dependency.
If you want live per-process memory history with PSS semantics rather than a one-shot snapshot, run a small exporter that reads smaps_rollup and ships the numbers — our cgroup v2 management guide covers systemd-cgtop and cgroup-tools, which give you the cgroup-level view without writing a parser.
The Container Number: cgroup v2 memory.current and Working Set
Inside a cgroup v2 hierarchy, memory accounting is per-cgroup and happens once — shared pages are charged to the first cgroup that touched them, then charged again to anyone else mapping them (the memory.stat field file counts mapped file pages charged to that cgroup). That is why container numbers do add up, and why they still do not match what you expect.
| |
A typical reading inside a container that “uses 1.8 GB” according to docker stats:
| |
The working-set estimate used by Kubernetes is exactly this subtraction:
| |
740 MB is the number that reflects what will actually be stolen from the page cache before the kernel starts reclaiming anonymous memory or invoking the OOM killer. It is also why a Redis or Elasticsearch pod can sit at 95% of its limit indefinitely and never get killed: the bulk of that 95% is reclaimable file cache, and memory.max only bites when reclaim cannot free enough.
Three consequences worth internalising:
memory.currentis not “memory in use”. It is “memory charged”, and file cache is charged. Alert on working set, or onanon, never onmemory.currentalone.memory.highis your friend. Settingmemory.highbelowmemory.maxthrottles the cgroup and forces reclaim before the hard limit triggers a kill — a graceful degradation path instead of an execution.shmemis not reclaimable without swap.tmpfsand shared memory segments show up asshmem, are accounted like file pages, and cannot be evicted to disk if there is no swap. A container writing 1 GB to/dev/shmwill OOM at its limit even though the number looked like “just cache”.
memory.events deserves a permanent place in your dashboards, because it distinguishes the expensive failure from the cheap one:
| |
If max is climbing but oom_kill is flat, reclaim is working and you have headroom problems, not a leak. If oom_kill is climbing while anon is flat, you are being killed by page cache you cannot reclaim in time — usually tmpfs, dirty pages, or a memory-mapped index.
Language Runtimes: Why RSS Above Heap Is Normal
Most “leak” investigations end here, with a runtime that is holding memory on purpose.
Java. Heap is one part of RSS. Add metaspace (class metadata), code cache (JIT), thread stacks (1 MB reserved per thread by default), GC bookkeeping (card tables, remembered sets), direct ByteBuffer allocations, and glibc malloc arenas. A 2 GB heap commonly produces 2.8-3.4 GB of RSS. Measure the split with native memory tracking rather than inferring it:
| |
Go. The runtime returns memory to the OS lazily and keeps a heap reserve. GOMEMLIMIT (Go 1.19+) sets a soft limit that makes the GC work harder near it; runtime/debug.SetMemoryLimit does the same in code. Without it, a Go service with a 512 MB live heap can hold 1.5 GB of RSS indefinitely because the allocator sees no reason to release pages.
| |
glibc-based services (C, C++, Python, Ruby). Per-thread arenas fragment and rarely return pages. MALLOC_ARENA_MAX=2 is the standard mitigation for multi-threaded services whose RSS grows with thread count, and malloc_trim(0) (or MALLOC_TRIM_THRESHOLD_) forces a release in some workloads.
Any runtime with transparent huge pages. THP collapses 4 KiB pages into 2 MiB pages, which inflates RSS granularity by up to 2 MB per mapping and makes smaps_rollup totals jump in steps. If RSS moves in 2 MB increments, you are watching khugepaged, not a leak.
When memory pressure is real rather than cosmetic, the next layer down is reclaim behaviour — kswapd, watermark tuning and dirty-page ratios — which our memory reclaim and drop_caches guide covers in detail, along with why drop_caches is not a fix.
The Five Measurement Traps
- Summing RSS across containers. Double counts every shared library and page-cache page. Use PSS for attribution and cgroup working set for enforcement.
- Alerting on
memory.current. Includes reclaimable page cache. A pod reading a large file once looks like it doubled its memory. Alert on working set oranon. - Comparing cgroup numbers to
free.free -mreports host-wide available memory; cgroup accounting charges shared pages once to the first toucher. The two live in different accounting universes and will never reconcile. - Running
smemwithout root. Silently omits processes you cannot read, producing a total that is wrong in the comforting direction. - Forgetting
shmemand hugepages.tmpfsis charged as page cache but cannot be reclaimed without swap, and THP rounds every mapping up to 2 MB. Both make “cache” numbers behave like anonymous memory under pressure.
FAQ
What is the difference between RSS, PSS and USS?
RSS is the total pages resident in a process’s address space and counts shared pages in full. PSS charges shared pages proportionally to the number of processes mapping them, so it never double counts. USS counts only private pages — the memory that is freed when the process exits. For a process mapping 300 MB of shared libraries, RSS might read 410 MB, PSS 115 MB, and USS 85 MB.
Which memory number should I use for a Kubernetes memory limit?
Set memory.max (the pod limit) from measured working set with headroom, not from memory.current at peak. Working set equals memory.current minus inactive_file, which is what Kubernetes uses for eviction ranking. Size the limit so that working set at peak load plus a safety margin fits, and set memory.high slightly below it so you get throttling and reclaim before the OOM killer.
Why does my container show high memory usage but never get OOM-killed?
Because most of the number is reclaimable page cache. The kernel only invokes the OOM killer when it cannot reclaim enough memory to satisfy an allocation, so a cgroup at 95% of memory.max full of clean file pages is comfortable. Check memory.stat: if inactive_file is large, the “usage” is cache, and memory.events will show max hits with few or no oom_kill events.
Is /proc/PID/smaps_rollup expensive to read?
No. It returns one aggregated structure per process rather than the full per-VMA table that /proc/PID/smaps produces, so it is cheap enough to poll every few seconds for a few hundred processes. Reading full smaps for the same set can cost tens of milliseconds each and is a common cause of monitoring-induced CPU load.
Why is my Java or Go process using far more RSS than its heap?
Both runtimes hold memory outside the heap: Java needs metaspace, JIT code cache, thread stacks, GC structures and direct buffers; Go keeps a heap reserve and returns pages to the OS lazily. Use jcmd VM.native_memory summary for Java and GOMEMLIMIT plus a heap profile for Go, and compare PSS over time instead of RSS at a single instant. RSS that is flat over hours is not a leak, regardless of how large it is.
Does tmpfs count against a container’s memory limit?
Yes. tmpfs and shared memory segments are accounted as shmem inside the cgroup and count toward memory.current and memory.max. Unlike clean file cache, shmem cannot be reclaimed without swap, so a container writing a large file to /dev/shm can be OOM-killed while its anonymous memory usage is small.
💰 想测试你的市场判断力?我用 Polymarket 做预测市场交易——这是全球最大的预测市场平台,从大选结果到技术监管时间线,什么都可以押注。和赌博不同,这是真正的信息市场:你懂的信息越多,胜率越高。我靠预测技术相关事件的走向已经赚了不少。用我的邀请链接注册:Polymarket.com