Self-Hosted Linux Performance Profiling: perf vs bcc-tools vs sysstat (2026)

When self-hosted servers experience performance issues — slow database queries, high CPU usage, memory leaks, or I/O bottlenecks — the first step is always measurement. Linux offers a rich ecosystem of performance profiling and monitoring tools, each with different strengths, depths of visibility, and operational complexity.

This guide compares three essential Linux performance toolkits — perf (the kernel’s built-in profiler), bcc-tools (eBPF-based tracing utilities), and sysstat (system statistics collectors) — with practical examples for diagnosing production issues on self-hosted infrastructure.

The Performance Debugging Hierarchy

Effective performance troubleshooting follows a layered approach:

1. System-level metrics — CPU, memory, disk, and network utilization over time (sysstat)
2. Kernel-level tracing — what the kernel is actually doing: syscalls, page faults, context switches (bcc-tools)
3. Application-level profiling — where time is spent inside your code: function hotspots, call graphs (perf)

Each tool occupies a different layer. sysstat tells you what is happening, bcc-tools tells you why at the kernel level, and perf tells you where at the application level.

sysstat: System Statistics Collection

sysstat (3,319 stars on GitHub, last updated May 2026) is the oldest and most widely deployed Linux performance monitoring toolkit. It collects and reports system-level metrics through a suite of complementary commands, making it the go-to tool for baseline performance assessment and historical trend analysis.

Key Features

• sar (System Activity Reporter) — collects and reports CPU, memory, I/O, and network statistics
• iostat — detailed disk I/O statistics per device, including wait times and throughput
• mpstat — per-CPU utilization reports for multi-core systems
• pidstat — per-process resource usage (CPU, memory, I/O, context switches)
• Historical data — stores metrics for post-incident analysis (configurable retention)

Installation and Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Install sysstat
apt install sysstat -y
# or
yum install sysstat -y

# Enable data collection (edit /etc/default/sysstat on Debian)
ENABLED="true"

# Restart to begin collection
systemctl enable --now sysstat

# Data is stored in /var/log/sysstat/saDD (DD = day of month)

Essential sysstat Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Current CPU utilization (refresh every 2 seconds)
sar -u 2 5

# Per-CPU utilization
mpstat -P ALL 2 5

# Disk I/O statistics
iostat -x 2 5
# Output columns: %util, await, svctm, r/s, w/s, rkB/s, wkB/s

# Network statistics
sar -n DEV 2 5

# Memory and swap usage
sar -r 2 5

# Historical data for a specific day
sar -f /var/log/sysstat/sa23 -u

# Per-process stats (CPU, memory, I/O)
pidstat -urdh 2 5

Identifying I/O Bottlenecks with iostat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Extended I/O stats with human-readable units
iostat -xh 2 10

# Key columns to watch:
# %util > 80% → device is saturated
# await > 10ms → high latency (possible I/O bottleneck)
# r/s + w/s → IOPS (compare against device specs)
# rkB/s + wkB/s → throughput

# For NVMe devices, expect:
# - await < 1ms under normal load
# - %util varies with queue depth
# - IOPS can exceed 100,000

Docker Compose Monitoring Setup

sysstat runs on the host and monitors all container activity:

1
2
3
4
5
6

# Monitor container I/O specifically
# Find container PIDs
docker inspect --format '{{.State.Pid}}' <container_name>

# Then use pidstat for that PID
pidstat -d 2 5 -p <container_pid>

Best Use Cases for sysstat

• Baseline monitoring — establishing normal performance patterns
• Post-incident analysis — reviewing historical data after an outage
• Capacity planning — trending resource usage over weeks and months
• First-response triage — “is it CPU, memory, disk, or network?”
• Production servers — minimal overhead (typically < 1% CPU for data collection)

bcc-tools: eBPF-Based Dynamic Tracing

bcc-tools (22,414 stars on GitHub, last updated May 2026) is a collection of tracing and performance analysis tools built on eBPF (extended Berkeley Packet Filter). It allows you to observe kernel behavior in production with near-zero overhead, making it the most powerful option for diagnosing live performance issues without restarting services.

Key Features

• Zero-code kernel tracing — attach probes to any kernel function, syscall, or tracepoint
• Production-safe — eBPF programs are verified by the kernel before execution; they cannot crash the system
• Per-process tracing — trace specific PIDs or process groups without affecting other workloads
• Rich pre-built tools — 100+ command-line utilities for common performance scenarios
• Python-based custom tools — write your own tracers using the BCC Python library

Installation and Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Install bcc-tools
apt install bpfcc-tools linux-headers-$(uname -r) -y
# or
yum install bcc-tools kernel-headers -y

# Verify eBPF is available
bpftool feature probe | grep -c "eBPF"
# Should return a high number (100+)

# Tools are installed in /usr/share/bcc/tools/
ls /usr/share/bcc/tools/
# argdist, biosnoop, cachestat, cpudist, execsnoop, fileslower,
# funclatency, kprobe, offcputime, opensnoop, pidpersec, runqlat,
# tcpconnlat, tcpretrans, vfscount, ... (100+ tools)

Essential bcc-tools for Self-Hosted Servers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# What processes are opening files? (diagnose file descriptor leaks)
/usr/share/bcc/tools/opensnoop

# How long are disk I/O operations taking? (identify slow storage)
/usr/share/bcc/tools/biosnoop

# What is the kernel page cache hit rate? (diagnose memory I/O)
/usr/share/bcc/tools/cachestat

# How long do processes spend waiting for CPU? (run queue latency)
/usr/share/bcc/tools/runqlat

# Track TCP connection latency (diagnose network issues)
/usr/share/bcc/tools/tcpconnlat

# Which processes are executing slow syscalls?
/usr/share/bcc/tools/execsnoop

# Profile kernel function latency
/usr/share/bcc/tools/funclatency -p $(pgrep -f postgres) vfs_read

Diagnosing a Slow Database Query

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Step 1: Check if it's I/O-bound
/usr/share/bcc/tools/biosnoop -p $(pgrep -f postgres)

# Step 2: Check page cache behavior
/usr/share/bcc/tools/cachestat 1

# Step 3: Check run queue latency (CPU contention)
/usr/share/bcc/tools/runqlat 1

# Step 4: Profile specific kernel functions
/usr/share/bcc/tools/funclatency vfs_read

Docker Compose with eBPF Monitoring

bcc-tools runs on the host and can monitor specific containers by PID:

1
2
3
4
5
6
7
8

# Find the container's PID
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' postgres)

# Monitor that container's I/O
/usr/share/bcc/tools/biosnoop -p $CONTAINER_PID

# Monitor that container's file opens
/usr/share/bcc/tools/opensnoop -p $CONTAINER_PID

Best Use Cases for bcc-tools

• Live production debugging — trace running systems without restarts
• Intermittent performance issues — catch sporadic slowdowns that don’t appear in aggregate metrics
• Kernel-level root cause analysis — understand why syscalls are slow, which locks are contended
• Network troubleshooting — diagnose TCP retransmits, connection latency, DNS resolution delays
• Storage optimization — identify slow disk operations, cache misses, and I/O patterns

perf: Kernel Performance Counter Profiling

perf (part of the Linux kernel source, 234,051+ stars) is the official Linux performance analysis tool. It uses hardware performance counters and kernel tracepoints to provide function-level profiling with call graphs, making it the most detailed option for understanding where CPU time is spent.

Key Features

• Hardware performance counters — CPU-level metrics (cache misses, branch mispredictions, instructions per cycle)
• Call graph generation — full stack traces showing which functions consume the most CPU time
• Flame graph support — visualize profiling data as interactive flame graphs
• Event-based sampling — profile specific events (cache misses, page faults, context switches)
• Per-thread profiling — isolate profiling to specific threads or processes

Installation and Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Install perf
apt install linux-tools-common linux-tools-generic -y
# or
yum install perf -y

# Verify perf is working
perf version

# Check available events
perf list | head -30

# Check hardware counters available
perf stat -e cycles,instructions,cache-misses,branch-misses sleep 1

Essential perf Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# CPU profiling (sample every 99 Hz)
perf record -F 99 -p $(pgrep -f postgres) -g -- sleep 30

# Generate a report
perf report

# System-wide profiling (all processes)
perf record -a -g -- sleep 10

# Profile specific events
perf stat -e cpu-clock,task-clock,context-switches,cpu-migrations     -p $(pgrep -f nginx) sleep 10

# Profile a single command
perf stat -d nginx -g "daemon off;"

# Trace syscalls
perf trace -p $(pgrep -f redis-server)

# Profile kernel scheduler behavior
perf sched record -- sleep 10
perf sched latency

Generating Flame Graphs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Install flame graph tools
git clone https://github.com/brendangregg/FlameGraph.git

# Record with call graphs
perf record -F 99 -a -g -- sleep 30

# Generate flame graph
perf script | FlameGraph/stackcollapse-perf.pl |     FlameGraph/flamegraph.pl > perf.svg

# Open in browser
firefox perf.svg

Docker Container Profiling

1
2
3
4
5
6
7
8

# Profile a specific container
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' mysql)

# Record with call graphs (follow child processes)
perf record -F 99 -p $CONTAINER_PID -g -- sleep 30

# For containers with --pid=host, perf works directly inside
docker run --pid=host --cap-add SYS_ADMIN     -v /tmp:/tmp     ubuntu perf record -a -g -- sleep 10

Best Use Cases for perf

• CPU bottleneck analysis — which function or code path is consuming the most CPU time?
• Cache performance optimization — are cache misses degrading throughput?
• Application profiling — understanding hotspots in custom code or open-source services
• Kernel debugging — analyzing scheduler behavior, lock contention, and interrupt handling
• Benchmarking — measuring performance before and after configuration changes

Comparison Table

Choosing the Right Tool for the Job

Why Self-Host With Performance Profiling Tools?

When you run self-hosted infrastructure, you are the performance engineer. Cloud providers abstract away the underlying hardware, making deep performance analysis difficult or impossible. On your own server, you have full visibility into every layer — from hardware performance counters to application-level function profiles.

Performance profiling tools are essential for:

• Capacity planning — knowing exactly when to scale up or out
• Incident response — quickly identifying the root cause of slowdowns
• Optimization validation — measuring the impact of configuration changes before and after
• Cost efficiency — maximizing hardware utilization without over-provisioning

A self-hosted PostgreSQL server running on properly tuned hardware can outperform a cloud-managed instance at a fraction of the cost — but only if you have the tools to measure and optimize. sysstat provides the baseline, bcc-tools reveals kernel-level bottlenecks, and perf pinpoints application-level hotspots.

For I/O-side optimization that complements profiling, see our I/O scheduler comparison for storage tuning. For memory optimization, our HugePages management guide covers RAM-side performance. And for container-level resource monitoring, check our cgroup monitoring tools guide for process isolation metrics.

FAQ

Which profiling tool should I install first on a new self-hosted server?

Start with sysstat. It provides continuous baseline monitoring with near-zero overhead, and the historical data it collects is invaluable for post-incident analysis. Enable it during initial server setup — the data it accumulates over the first week establishes performance baselines that make future anomalies easy to spot. Add bcc-tools and perf as needed for deeper analysis.

Do bcc-tools require kernel recompilation?

No. Modern Linux kernels (4.9+) include eBPF support by default. bcc-tools load eBPF programs into the running kernel at runtime using the bpf() syscall. The kernel’s eBPF verifier ensures programs are safe before executing them. You only need matching kernel headers installed (linux-headers-$(uname -r) on Debian, kernel-headers on RHEL).

Can I use perf inside Docker containers?

Yes, but you need elevated privileges. Run the container with --cap-add SYS_ADMIN and --pid=host, or mount the host’s /tmp directory for perf data storage. For production, it’s safer to run perf on the host and target specific container PIDs using docker inspect --format '{{.State.Pid}}' <container>.

How much overhead does sysstat add to production systems?

Typically less than 1% CPU. sysstat’s data collection interval defaults to 10 minutes, which is negligible. You can increase collection frequency (e.g., every 1 minute in /etc/cron.d/sysstat) with minimal impact. The sar command reads pre-collected data — it doesn’t add overhead at query time.

Why would I use bcc-tools instead of perf for kernel tracing?

bcc-tools are better for specific kernel event tracing (which files are opened, how long syscalls take, TCP connection states) because they provide targeted, pre-built tools for each scenario. perf is better for CPU profiling and call graphs (which functions consume the most CPU time). They’re complementary: use bcc-tools to identify the problematic subsystem, then use perf to find the exact code path.

How do I profile a production database without impacting performance?

Use perf record -F 49 (lower frequency, 49 Hz instead of the default 99 Hz) for reduced sampling overhead. Limit profiling duration to 30-60 seconds. For bcc-tools, the overhead is already minimal (eBPF JIT compilation means near-native speed). For sysstat, there’s virtually no overhead since it reads existing kernel counters. Always profile during low-traffic periods when possible, and avoid perf record -a (system-wide) on production — target specific PIDs instead.