Kernel Logging Mastery: Forwarding dmesg via Systemd and Vector

When a production server hangs or a driver crashes, the first place I look is the kernel ring buffer. If you aren't capturing those logs centrally, you’re flying blind during a post-mortem. I’ve spent too many hours SSHing into unresponsive nodes, hoping the local logs haven't been overwritten or lost in a hard reset.

The most reliable way to handle kernel logging is to treat the kernel ring buffer as just another source in your observability pipeline. By leveraging the existing integration between dmesg and systemd-journald, we can pull these events into Vector.dev Log Management: Real-Time Routing on Dockerized VPS without writing custom polling scripts.

Why the Kernel Ring Buffer is Tricky

The kernel ring buffer is a fixed-size memory area. If your system is noisy—say, a failing NIC throwing thousands of interrupts—it wraps around quickly. If you rely solely on dmesg commands, you might miss the "smoking gun" that happened ten minutes before the crash.

We need to persist these messages immediately. Systemd-journald does this out of the box by reading from /dev/kmsg and injecting those logs into the binary journal. Once they're in the journal, they’re immutable and ready to be shipped.

Configuring the Pipeline

First, ensure your journald is actually capturing kernel messages. Check your /etc/systemd/journald.conf. You don’t need much, but verify that ForwardToSyslog or Storage is set correctly. I usually prefer Storage=persistent to survive reboots, which is vital for diagnosing kernel panics that occur during boot—a topic I’ve touched on before when Optimizing Linux Boot Times: A Practical Guide for VPS.

Once the logs are in the journal, we let Vector do the heavy lifting. Vector is incredibly efficient at tailing the journal and filtering out the noise.

Implementing Vector for Kernel Observability

To forward these logs, define a journald source in your vector.yaml. I typically run this as a sidecar or a host-level agent:

YAML
sources:
  kernel_logs:
    type: journald
    include_units:
      - kernel

transforms:
  filter_kernel:
    type: filter
    inputs:
      - kernel_logs
    condition: .priority <= 3 # Only alert on Errors, Critical, or Emergencies

sinks:
  observability_stack:
    type: elasticsearch # Or loki, datadog, etc.
    inputs:
      - filter_kernel
    endpoint: http://your-log-aggregator:9200

By filtering at the source, you save on egress costs and storage. You don't need every info or debug message from the kernel in your main dashboard; you need the stuff that indicates hardware failure or memory corruption.

The Trade-offs of Forwarding dmesg

We first tried using a simple cron job to dump dmesg to a file and tailing that file with Filebeat. It was a disaster. The file would grow until it hit disk limits, or worse, the cron job would lock the file during a write, causing the kernel log to stall.

Moving to systemd-journald + Vector removed that contention. However, keep in mind that the kernel ring buffer is still a finite resource. If your kernel is flooding the buffer, it will drop messages regardless of how well you've configured your logging stack. If you suspect hardware issues, you might also need Linux kernel security: How to harden your Docker host with LKRG to gain better visibility into runtime integrity.

Dealing with "Missing" Logs

Sometimes you'll notice a gap in your logs. This usually happens when the kernel is so overwhelmed that the journald process can't keep up with the read stream from /dev/kmsg.

If you're running a high-load environment, increase the RateLimitIntervalSec and RateLimitBurst in journald.conf. Just be careful: if you set these too high, a runaway process logging to the kernel can quickly consume all your disk space. I usually set RateLimitBurst to around 2000 messages—enough to catch a sudden failure, but not enough to fill a 50GB partition in minutes.

A Final Caveat

Remember that logs are only as good as your ability to alert on them. Don't just ship these to a "logs" bucket. Set up an alert in your observability platform to ping you whenever a kernel source log hits a priority level of 3 or lower.

I’m still experimenting with using eBPF to capture these events directly from the kernel tracepoints instead of relying on the ring buffer. It’s more complex to manage, but it avoids the "buffer wrap" problem entirely. For now, the journald approach is the "good enough" solution that keeps my production systems manageable without adding unnecessary overhead.