A nightly compression job starts at 02:00, the API’s p99 latency jumps from 40 ms to 900 ms, and uptime reports a load average of 38 on a 16-core box. Nothing is broken. Nothing is leaking. Two workloads are simply competing for the same cores, and the kernel is being perfectly fair — which is exactly the problem, because fairness is not the same as priority.
Linux offers at least six mechanisms for influencing who gets CPU time, and they are not interchangeable. nice adjusts a per-process weight. chrt changes the scheduling class. cgroup v2 cpu.weight sets a proportional share for a group. cpu.max imposes a hard bandwidth ceiling. cpuset decides which cores a group may run on at all. And systemd wraps four of them into unit directives that many administrators never touch.
Getting this wrong produces two failure modes: a background job that starves production traffic, or a quota so tight that a healthy service gets throttled into latency spikes. This guide covers the mechanisms, the arithmetic, and the verification commands.
TL;DR: the 60-second verdict
- A systemd service that must never starve → set
CPUWeight=1000on it and drop the noisy neighbour toCPUWeight=10. Proportional shares, no hard cap, no idle waste. - A batch job that must not exceed a fixed budget →
CPUQuota=200%(two full cores). Hard ceiling, predictable capacity planning. - A latency-critical process that must not be preempted →
chrt -f 50only with a dedicated cpuset (AllowedCPUs=), otherwise you risk stalling the whole box. - A container →
--cpu-sharesmaps tocpu.weight;--cpusmaps tocpu.max. In Kubernetes,requests.cpubecomes weight andlimits.cpubecomes quota — and the ratio between them is what protects your latency. nicealone is the weakest tool available. On a systemd host it is usually applied to the wrong process, in the wrong group, at the wrong time. Use it for ad-hoc shell work, not for service policy.
The six mechanisms, compared honestly
| Mechanism | Scope | Type | Range / units | Survives restart | Composable with others |
|---|---|---|---|---|---|
nice / renice | Single process (and children) | Proportional weight | −20 … +19 | No | Yes, but overridden by cgroup weights in v2 |
chrt | Single process | Scheduling class | FIFO 1–99, RR 1–99, BATCH, IDLE, DEADLINE | No | Yes — class is orthogonal to weight |
cgroup v2 cpu.weight | Group | Proportional share | 1 … 10000 (default 100) | Depends on manager | Yes, it is the base layer |
cgroup v2 cpu.max | Group | Hard bandwidth cap | quota period, e.g. 200000 100000 | Depends on manager | Yes — combines with weight |
cgroup v2 cpuset.cpus | Group | Core affinity (exclusive possible) | CPU list, e.g. 0-3,8-11 | Depends on manager | Yes — restricts where the group may run |
systemd CPUWeight / CPUQuota / AllowedCPUs | Unit or slice | Policy wrapper | Weight 1–10000, quota as % | Yes | Yes — maps onto the cgroup files above |
The key mental model: weight decides who wins when cores are oversubscribed; quota decides how much a group can ever use; cpuset decides where it may run. They answer three different questions, and production hosts usually need weight plus cpuset, with quota reserved for cost control.
Decision matrix
| Situation | Use | Why |
|---|---|---|
| API + nightly batch on the same host | CPUWeight=1000 on the API unit, CPUWeight=10 on the batch unit | The batch job still uses idle capacity, but yields instantly under load |
| Customer-facing service must be capacity-capped per tenant | CPUQuota= per slice | Hard, auditable ceiling; enables overcommit planning |
| Video transcoder that must never touch the API’s cores | AllowedCPUs=8-15 on the transcoder, AllowedCPUs=0-7 on the API | Physical isolation; removes contention instead of arbitrating it |
| Trading/telemetry pipeline needing sub-millisecond jitter | chrt -f + AllowedCPUs= + IRQAffinity= | FIFO class removes CFS preemption; the cpuset keeps RT from starving the kernel |
Ad-hoc rsync or tar on a production host | nice -n 19 ionice -c3 | Cheap, no configuration, good enough for one-off work |
| Container platform with noisy neighbours | --cpu-shares (weight) + --cpus (quota) | Matches Kubernetes semantics; avoids the classic “limit equals request” trap |
How proportional shares actually compute
cgroup v2 replaced cpu.shares with cpu.weight, and the mapping is not linear at the low end. Weight 100 is the default; the kernel converts weight to a share of the CFS runqueue using a lookup table so that small weights still receive a usable minimum slice.
The practical arithmetic for two competing groups is a simple ratio of weights:
| |
That last line is the reason to prefer weights over quotas for services. A quota of 80% leaves 20% unusable when the service is idle; a weight of 800 lets a neighbour consume everything during quiet periods and still guarantees the ratio under contention.
| |
systemd-cgtop is the most underrated tool in this area. In one screen it shows CPU, memory and I/O per unit and slice, which immediately reveals whether your weights are doing anything at all.
systemd: the four directives that matter
| |
| |
Two semantics worth memorizing. CPUQuota=200% means two cores, not “twice as much as before” — percentages are relative to a single core. And CPUWeight only matters when the parent slice is contended: if the API and the batch job live in different slices with different weights, the slice-level weight dominates the per-unit weight, because cgroup v2 distributes CPU hierarchically, level by level.
That hierarchy is the single most common reason “weights did nothing”. A unit’s effective share is the product of its share within its parent slice and the parent slice’s share of the root. Check the tree before tuning the leaves:
| |
Quotas: when they help and how they hurt
A quota is enforced by the CFS bandwidth controller: each group gets quota microseconds of runtime per period microseconds. When the group exhausts its quota it is throttled until the next period, no matter how many cores are idle.
| |
If nr_throttled climbs while latency spikes, the quota is too tight for the workload’s burst pattern. Two fixes work: raise the quota, or lengthen the period so short bursts can be absorbed.
| |
For most services, the right configuration is no quota at all and a weight instead, with quotas reserved for multi-tenant isolation or chargeback. Quotas turn a scheduling preference into a hard failure mode, and throttling shows up as latency, not as an error — which makes it hard to diagnose after the fact.
nice, autogroup, and why it disappoints
nice adjusts a task’s weight inside the CFS class. Values range from −20 (most favourable) to +19 (least), and only root can lower the value (raise priority).
| |
The catch is autogroup. The kernel’s sched_autogroup feature groups tasks by session (TTY), and CFS then schedules groups rather than individual tasks. On a machine with autogroup enabled, nice on one process inside a session can have far less effect than expected, because the whole session competes as a unit.
| |
On servers, the practical conclusion is that nice is fine for interactive one-offs, but it is the wrong layer for durable service policy. systemd’s Nice= directive is more reliable than a shell nice because it applies at exec time to the exact service process, but it still acts on a single task, not a workload — which is why CPUWeight is the better answer for a multi-process service such as a forking web server.
Real-time scheduling without taking down the host
SCHED_FIFO and SCHED_RR bypass CFS entirely: a runnable FIFO task runs until it blocks or a higher-priority RT task preempts it. A busy loop at chrt -f 99 on a shared host will make the box unreachable.
| |
Two safety mechanisms exist, and both must be respected:
- RT throttling.
sched_rt_runtime_us(default 950000 of a 1000000 µs period) reserves 5% of each period for non-RT tasks. Do not disable it. If your RT workload needs more than 95% of a core, it needs its own core. - CPU isolation. Pin RT workloads with
cpusetand move interrupts off those cores:
| |
AllowedCPUs= writes cpuset.cpus, which is stricter than affinity: children inherit it and the scheduler will not migrate the group outside the set. CPUAffinity= sets the initial affinity mask but permits later changes by the process. Use AllowedCPUs when isolation is a correctness requirement.
Containers and Kubernetes
Docker’s flags map directly onto the cgroup files:
| |
| |
In Kubernetes the mapping is subtler and it is where most latency incidents originate:
| Kubernetes field | cgroup v2 effect | Behaviour |
|---|---|---|
requests.cpu: 500m | cpu.weight proportional to 500 | Guaranteed share under contention |
limits.cpu: 2000m | cpu.max = 2 cores | Hard throttle at 2 cores |
| Both set, ratio 1:4 | weight from request, quota from limit | Burst up to 4×, guaranteed 1× |
| Only limits set | quota, default weight | Burstable pods with equal weight — noisy neighbour risk |
Setting only limits.cpu gives every pod the default weight, so two pods with 2-core limits and identical weight split available CPU evenly — even if one is a batch worker and the other serves user traffic. Always set requests, and make the request reflect real need rather than a copy of the limit.
Verification: proving the settings took effect
| |
The ps format above uses cls and rtprio, which is the fastest way to find a stray FIFO thread. cls values are TS (other), FF (FIFO), RR (round-robin), B (batch), IDL (idle) and DLN (deadline).
Pitfalls that make tuning silently useless
- Tuning a unit whose parent slice is the real bottleneck. Effective share is hierarchical. Fix the slice weight first.
- Setting
CPUQuotaand calling it priority. A quota caps; it does not prioritize. Under contention, a capped group still loses to an uncapped one with a higher weight. CPUQuota=100%on a multi-threaded service. That is one core. Threads will queue behind the quota and latency will spike at every period boundary.- Using
niceon a systemd service from a shell. The shell’sniceaffects the shell. UseNice=in the unit, orsystemctl set-property api.service Nice=-5. - Setting
chrt -fwithout a cpuset. One infinite loop and the host needs a hard reset. Always isolate RT workloads. - Disabling RT throttling.
sched_rt_runtime_us=-1removes the kernel’s safety valve and turns a bug into an outage. - Forgetting IRQ affinity. Pinning a latency-sensitive process to core 3 while the NIC queue for that process also lands on core 3 reintroduces the jitter you removed.
- Expecting
CPUAffinity=to prevent migration of children. It sets the affinity mask;AllowedCPUs=(cpuset) is the enforcing version. - Ignoring
cpu.statafter a change. Without the throttle counters you cannot distinguish “quota too tight” from “workload grew”.
A configuration that works for the common case
| |
| |
This gives the API physical isolation from the batch job, a 100:1 weight advantage if they ever share cores (for example after a CPU hotplug or a cpuset fallback), and no hard caps that could throttle either workload into a latency cliff. It is boring, and boring is what a production scheduler should be.
Related reading on this site: the kernel sysctl and tuned guide covers the global knobs that sit above cgroup policy, the OOM prevention guide explains how memory pressure interacts with CPU starvation and PSI, and the block I/O latency tracing walkthrough is the right next step when latency spikes turn out to be I/O waits rather than scheduling.
FAQ
Is CPUWeight the same as CPU shares in cgroup v1?
Functionally yes, but the numbers differ. cgroup v1 used cpu.shares with a default of 1024 and a range up to 262144. cgroup v2 uses cpu.weight with a default of 100 and a range of 1–10000, mapped through a kernel table so small values still receive usable time. systemd’s CPUWeight= writes the v2 file.
Why did raising CPUWeight change nothing?
Three usual causes: the group is not actually contended (weights only matter under oversubscription), the parent slice has a low weight so the unit’s share within it is capped hierarchically, or the workload is being throttled by cpu.max so weight cannot help. Check cpu.stat and the slice tree before changing numbers again.
Should I use CPUQuota or CPUWeight for a background job?
Use CPUWeight if you want the job to consume all idle capacity and yield under load. Use CPUQuota only when you need a hard, auditable ceiling for capacity planning or tenant isolation. Quotas waste idle capacity and introduce throttling latency; weights never waste capacity but cannot guarantee an upper bound.
Does nice still matter on a cgroup v2 system?
Inside a single cgroup it still orders tasks relative to each other, and Nice= in a unit is applied reliably at exec time. Across groups, cpu.weight dominates because CFS schedules groups hierarchically. For durable service policy, prefer weights; keep nice for ad-hoc shell commands.
How do I give one process real-time priority without risking the host?
Pin it to isolated cores first (isolcpus= at boot, AllowedCPUs= in the unit, IRQs moved away), then apply chrt -f with a modest priority. Leave sched_rt_runtime_us at its default so the kernel reserves headroom for non-RT work, and monitor cpu.stat plus the RT throttling counters in /proc/sys/kernel/sched_rt_*.
What is the Kubernetes equivalent of nice?
There is none, and that is deliberate. Kubernetes expresses priority through requests.cpu (which becomes weight) and QoS classes. Guaranteed pods (requests equal limits) get the most predictable treatment; BestEffort pods (no requests or limits) are the first to be squeezed. Use PriorityClass for preemption, not for share ratios.
💰 想测试你的市场判断力?我用 Polymarket 做预测市场交易——这是全球最大的预测市场平台,从大选结果到技术监管时间线,什么都可以押注。和赌博不同,这是真正的信息市场:你懂的信息越多,胜率越高。我靠预测技术相关事件的走向已经赚了不少。用我的邀请链接注册:Polymarket.com