When your infrastructure generates terabytes of logs daily, storing and searching every log line becomes prohibitively expensive. Log sampling - the practice of intelligently selecting a representative subset of log data - lets you maintain observability while controlling storage costs.

This guide compares three self-hosted tools for log sampling and reduction: Vector (log sampling pipelines), Fluent Bit (lightweight log processor with sampling filters), and Grafana Loki (log aggregation with built-in sampling at ingestion).

Overview

FeatureVectorFluent BitGrafana Loki
GitHub Stars20,000+17,000+25,000+
LanguageRustCGo
Memory UsageLowVery LowModerate
Sampling MethodsThrottle, sample, reduceSample, throttle, grepStructured metadata sampling
Log TransformationFull VRL supportLua/regexVia pipeline stages
Log AggregationNo (shipper only)No (shipper only)Full backend
Rate LimitingBuilt-inBuilt-inVia ingestion limits
Log DeduplicationYesLimitedVia indexed fields
Output Targets80+ destinations60+ destinationsGrafana only
ConfigurationTOMLYAML/INIYAML (loki-config)
Kubernetes SupportDaemonSet/SidecarDaemonSetRead/Write path
Prometheus MetricsNativeNativeNative

Vector: High-Performance Log Sampling Pipelines

Vector is a high-performance observability data pipeline written in Rust. Its sampling transforms allow you to reduce log volume while preserving important events.

Sampling Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# vector.toml - Log sampling pipeline
[sources.k8s_logs]
type = "kubernetes_logs"

# Sample 10% of all log lines (random sampling)
[transforms.random_sample]
type = "sample"
inputs = ["k8s_logs"]
rate = 10
key = "message"

# Throttle: max 100 logs/second per source
[transforms.throttle_sample]
type = "throttle"
inputs = ["k8s_logs"]
threshold = 100
window_secs = 1
key_field = "kubernetes.pod_name"

# Reduce: deduplicate consecutive identical log lines
[transforms.dedup]
type = "reduce"
inputs = ["k8s_logs"]
group_by = ["kubernetes.pod_name", "kubernetes.container_name"]
merge_strategies.message = "array"

# Drop DEBUG logs in production, keep WARN+
[transforms.level_filter]
type = "filter"
inputs = ["k8s_logs"]
condition = ".level != "DEBUG""

[sinks.loki]
type = "loki"
inputs = ["throttle_sample", "dedup"]
endpoint = "http://loki:3100"
encoding.codec = "json"

[sinks.loki.labels]
severity = "{{ level }}"
namespace = "{{ kubernetes.pod_namespace }}"

VRL-Based Conditional Sampling

Vector Remap Language (VRL) enables sophisticated sampling logic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

[transforms.smart_sample]
type = "remap"
inputs = ["k8s_logs"]
source = "always_keep = includes(["ERROR", "CRITICAL"], .level)
if always_keep { .sampling_decision = "keep" } else { .sampling_decision = "sample" }"

[transforms.after_sample]
type = "filter"
inputs = ["smart_sample"]
condition = ".sampling_decision == "keep""

Docker Compose for Vector

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

version: "3.8"
services:
  vector:
    image: timberio/vector:latest
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    environment:
      - VECTOR_LOG=info
    restart: unless-stopped

Fluent Bit: Lightweight Log Sampling

Fluent Bit is a lightweight log processor from the CNCF. Its filter plugins provide sampling, throttling, and log reduction with minimal resource overhead.

Sampling Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# fluent-bit.conf - Log sampling with filters
[SERVICE]
    Flush         1
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/containers/*.log
    Parser        docker
    Tag           kubernetes.*
    Refresh_Interval  5

# Sample filter: keep 10% of logs
[FILTER]
    Name          sample
    Match         kubernetes.*
    Rate          10

# Throttle filter: max 50 logs/sec per tag
[FILTER]
    Name          throttle
    Match         kubernetes.*
    Rate          50
    Window        60

# Grep filter: drop DEBUG logs
[FILTER]
    Name          grep
    Match         kubernetes.*
    Exclude       log ^.*level.*DEBUG.*$

[OUTPUT]
    Name          loki
    Match         *
    Url           http://loki:3100
    Labels        job=fluentbit, severity=$level
    Line_Format   json

Lua-Based Advanced Sampling

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

-- sample.lua - Custom sampling logic
function sample_log(tag, timestamp, record)
    -- Always keep error logs
    if record["log"] and string.find(record["log"], "ERROR") then
        return 1, timestamp, record
    end
    
    -- Hash-based sampling for INFO logs
    if record["log"] and string.find(record["log"], "INFO") then
        local hash = 0
        for i = 1, string.len(record["log"]) do
            hash = hash * 31 + string.byte(record["log"], i)
        end
        if hash % 100 < 5 then
            return 1, timestamp, record
        end
        return 0, 0, nil
    end
    
    -- Drop DEBUG entirely in production
    return 1, timestamp, record
end

Docker Compose for Fluent Bit

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

version: "3.8"
services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - ./sample.lua:/fluent-bit/etc/sample.lua:ro
      - /var/log:/var/log:ro
    restart: unless-stopped

Grafana Loki: Log Aggregation with Sampling

Grafana Loki is a horizontally scalable log aggregation system. It supports sampling through ingestion rate limits, structured metadata, and collector-side pipelines.

Loki Sampling Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# loki-config.yaml - Ingestion rate limiting
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  max_global_streams_per_user: 10000
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 100
  max_query_series: 10000
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  per_stream_rate_limit: "5MB"
  per_stream_rate_limit_burst: "10MB"

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true
  retention_delete_delay: 2h

storage_config:
  filesystem:
    directory: /loki/chunks

Promtail with Sampling Pipeline

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# promtail-config.yaml - Pre-ingestion sampling
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: kubernetes
    pipeline_stages:
      - drop:
          expression: ".*level.?=.?DEBUG.*"
      - regex:
          expression: ".*level=(?P<level>\\w+).*service=(?P<service>\\w+).*"
      - labels:
          level:
          service:
    kubernetes_sd_configs:
      - role: pod

Docker Compose for Loki Stack

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

version: "3.8"
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yaml:ro
      - /var/log:/var/log:ro
    command: -config.file=/etc/promtail/config.yaml
    restart: unless-stopped

volumes:
  loki-data:

Sampling Strategies Comparison

StrategyBest ToolUse Case
Random sampling (N percent)VectorGeneral log volume reduction
Rate-based throttlingFluent BitPrevent log storms from noisy services
Level-based filteringAll threeDrop DEBUG in production
Hash-based deterministicFluent Bit (Lua)Consistent sampling for debugging
DeduplicationVector (reduce transform)Eliminate repeated identical logs
Error-first samplingVector (VRL)Always keep errors, sample everything else
Structured metadataLokiSample based on labels/streams
Ingestion rate limitingLokiBackend-side sampling enforcement

Why Self-Host Log Sampling Infrastructure?

When logs contain sensitive data - credentials, PII, internal service names - sending them through a third-party log management service creates compliance risks. Self-hosted log sampling keeps your data within your infrastructure while still achieving the cost savings of reduced log volume.

For the broader logging ecosystem, see our syslog aggregation guide and log parsing comparison. For centralized journal collection, our systemd journal remote guide covers aggregation patterns that complement sampling.

Self-hosted log sampling delivers:

  • GDPR/SOC2 compliance - Sensitive log data never leaves your network
  • Cost predictability - Control storage costs by capping ingestion rates
  • Performance isolation - Log storms do not impact your observability backend
  • Custom sampling logic - Implement domain-specific rules (e.g., always keep payment-related logs)
  • Retention optimization - Sampled logs require less storage, enabling longer retention windows

FAQ

What is log sampling and when should I use it?

Log sampling is the practice of processing only a subset of generated log entries - either randomly, by rate limits, or through intelligent filtering. Use it when log volume exceeds your storage budget, when noisy services generate excessive low-value logs, or when you need to prevent log storms from overwhelming your observability backend.

Is it safe to sample logs? Will I miss important events?

A well-designed sampling strategy always preserves ERROR and CRITICAL logs while sampling lower-severity entries. Deterministic sampling (hash-based) ensures that the same request is consistently sampled, making debugging possible. Random sampling should only be applied to high-volume, low-value log levels like DEBUG and INFO.

What is the difference between log sampling and log filtering?

Filtering removes logs based on specific criteria (e.g., drop all DEBUG logs). Sampling reduces log volume statistically (e.g., keep 10 percent of INFO logs). Sampling is useful when you need a representative view of traffic patterns; filtering is used when you know certain log types are never needed.

Can I combine multiple sampling strategies?

Yes. A production setup typically layers: (1) level-based filtering to drop DEBUG, (2) rate-based throttling to cap per-service log rates, (3) random sampling for remaining INFO logs, and (4) deduplication to collapse identical consecutive messages.

How does Loki handle sampling differently from Vector and Fluent Bit?

Loki applies sampling at the ingestion layer through rate limits (per-stream and global). Vector and Fluent Bit sample before shipping, at the collector level. The best practice is to use both: collector-side sampling reduces network transfer, while Loki-side rate limits provide a safety net against collector misconfiguration.

What sampling rate should I use for production?

Start with: 100 percent for ERROR and CRITICAL, 10-25 percent for WARN, 1-5 percent for INFO, and 0 percent for DEBUG (drop entirely). Adjust based on your storage budget and query patterns. Monitor sampling effectiveness by tracking the ratio of dropped-to-kept logs.

JSON-LD