Vector.dev Log Management: Real-Time Routing on Dockerized VPS

I spent three hours last week staring at a 4GB log file on a production VPS, trying to grep my way through a connection timeout issue. It was a classic "ops tax"—I had the data, but I didn't have a way to make it useful. If you're tired of manually digging through container logs, it's time to set up a proper pipeline for Linux observability.

Integrating Vector.dev into your Docker stack is a game-changer for log management. It’s lightweight, written in Rust, and it handles the heavy lifting of parsing, filtering, and shipping logs before they ever hit your disk.

Why Vector.dev over the alternatives?

We’ve all tried the standard route: installing a heavy agent like Filebeat or Fluentd. They work, but they often consume more RAM than the microservices they're supposed to be monitoring. When you're running on a $5/month VPS, every megabyte of memory counts.

I initially tried piping logs directly to a remote Graylog instance via a custom shell script. It worked for about two days until a network blip caused the script to hang, locking up my main application process. That’s the danger of doing it yourself. Vector.dev handles backpressure, retries, and data transformation out of the box. It acts as a buffer, ensuring your app stays performant even when your logging backend is struggling.

Getting started with Vector.dev on Docker

To get this running, I deploy Vector as a sidecar container or a standalone service on the host. For a typical Dockerized setup, running it as a container with access to the Docker socket is the most flexible approach.

Here is how I structure my docker-compose.yml for Vector:

YAML
version: '3.8'
services:
  vector:
    image: timberio/vector:0.39.0-debian
    container_name: vector
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./vector.yaml:/etc/vector/vector.yaml:ro
    environment:
      - VECTOR_LOG=info

The magic happens in the vector.yaml configuration file. You define sources (where logs come from), transforms (how you clean them up), and sinks (where they go).

Implementing Vector.dev log management

The real power of Vector.dev is the ability to drop "noise" at the edge. If your Rails or Node.js app is logging health checks every three seconds, you don't need those in your long-term storage.

YAML
sources:
  docker_logs:
    type: docker_logs

transforms:
  filter_health_checks:
    type: filter
    inputs: [docker_logs]
    condition: '!"/health" in .message'

sinks:
  log_to_file:
    type: file
    inputs: [filter_health_checks]
    path: /var/log/app.log
    encoding:
      codec: json

With this setup, I’ve seen my log storage requirements drop by roughly 40% on high-traffic nodes. You aren’t just storing logs; you’re curating them. If you’re already using other tools to track health, Uptime Kuma Self-Hosted Monitoring: A Simple Guide for VPS Health is a great companion to ensure your services stay up while Vector handles the diagnostic data.

Infrastructure engineering and performance

When you're building out your Linux observability stack, remember that log routing is a trade-off. If you perform complex regex transformations on every log line, you will see an increase in CPU usage.

I typically keep my transformations simple:

1. Drop: Remove health checks and debug logs.
2. Parse: Extract JSON fields from structured logs.
3. Annotate: Add host metadata or environment tags.

If you find yourself needing to track more granular file-system changes, Linux Docker Inotify Auditing: Real-Time File Monitoring with Systemd provides a deeper look into the kernel events that log files often miss.

What I'm still figuring out

Vector is incredibly powerful, but the learning curve for the Vector Remap Language (VRL) is steep. I still occasionally write a complex filter that breaks my entire pipeline because of a single syntax error. If I were doing this over, I would focus more on testing my configuration files locally using vector validate before pushing them to production.

Also, be careful with log rotation. Even with Vector handling the streaming, if you’re sinking to a local file, you need to ensure logrotate is configured on your Linux host. Otherwise, you’ll wake up one day to a full disk.

It’s easy to get caught up in the "perfect" observability stack, but start small. Get your logs into a single file or a remote sink, filter out the obvious garbage, and go from there. Your future self—the one debugging a production outage at 2 AM—will thank you for the clean, searchable data.