Linux Performance Tuning: Managing Swap and OOM for Docker VPS

I remember sitting at 3:00 AM, watching a production VPS crawl to a halt. My htop screen was a sea of red, with the system stuck in an I/O wait loop because the kernel decided to swap out active Docker container memory to disk. It’s the classic "death spiral": the system tries to free memory, triggers swap, slows down, and then triggers the OOM (Out of Memory) killer on the wrong process.

If you’re running Docker-heavy workloads, you’ve likely felt this pain. When your host hits memory pressure, the default Linux kernel settings aren't always your friend. Let’s look at how to get control back.

Understanding the Swap Problem

Most default VPS distributions are configured with a vm.swappiness value of 60. This tells the kernel to start swapping memory to disk when you've still got 40% of your RAM free. For a web server or a database container, this is usually catastrophic.

When you have a mix of containers, the kernel doesn't inherently know which one is critical. If your application starts thrashing, it will dump page caches or process memory to disk, causing your latency to spike from a few milliseconds to several seconds.

We first tried solving this by simply disabling swap entirely. That was a mistake. We ended up with hard kernel panics the moment we hit an OOM event because the system had no "cushion." Instead, we needed a more surgical approach to Linux performance and memory management.

Tuning Swappiness for Docker Hosts

The goal is to keep the kernel from being too eager to swap. I usually set vm.swappiness to 10 or even 1. This forces the kernel to favor keeping application memory in RAM, only swapping as a last resort.

Check your current setting:

Bash
cat /proc/sys/vm/swappiness

To change it immediately, run:

Bash
sudo sysctl vm.swappiness=10

To make it permanent, add vm.swappiness=10 to /etc/sysctl.conf. If you're managing multiple hosts, I recommend using Linux Performance: Cgroups v2 and Systemd Slices for VPS to isolate container resources, which acts as a secondary layer of defense against memory-hungry processes.

Configuring the OOM Killer

Even with perfect swap settings, you might hit a hard memory limit. This is where the OOM killer steps in. By default, it’s a bit of a wildcard. You don't want it killing your database container just because your logging agent had a memory leak.

You can influence the OOM killer by adjusting the oom_score_adj for specific processes. For Docker, it's cleaner to manage this via the Docker daemon or docker-compose files.

In your docker-compose.yml, you can set a reservation:

YAML
services:
  web:
    image: nginx:latest
    deploy:
      resources:
        limits:
          memory: 512M

If you need to ensure a specific container is the last one to be killed, you can tweak its score. However, be careful—setting this to -1000 effectively makes the process "un-killable" by the OOM killer, which can lead to a hard kernel freeze if the system truly runs out of RAM.

When Docker Memory Management Fails

Sometimes, the issue isn't swap; it's the kernel's tracking of memory. If you've already optimized your swap and the OOM killer is still misbehaving, you might be dealing with kernel-level bottlenecks.

I've found that when dealing with high-traffic proxies, Linux Kernel Tuning: Fixing Socket Exhaustion in Docker Proxies is just as important as memory tuning. If your connections are stuck in TIME_WAIT, they consume memory structures that the kernel struggles to reclaim, mimicking a memory leak.

My Workflow for VPS Stability

I don't rely on a single knob to fix everything. My current "production-ready" checklist for a new VPS looks like this:

1. Set Swappiness: Keep it at 10 to reduce disk I/O pressure.
2. Limit Containers: Every container gets a memory_limit in Compose. Never let a container run wild.
3. Monitor Entropy: If you see weird spikes in CPU usage alongside memory pressure, check if your containers are starving for random numbers, as discussed in Linux performance: Managing Entropy Issues in Docker Containers.
4. Use ZRAM (Optional): On smaller VPS instances with limited RAM, I’ve had success using ZRAM. It creates a compressed swap device in RAM, which is significantly faster than disk-based swap.

Final Thoughts

None of these settings are "set it and forget it." Your application's memory profile changes as you add features or scale traffic. I’m still experimenting with eBPF-based tools to get better visibility into exactly which process is triggering the memory pressure before it reaches the swap stage.

If you find yourself constantly fighting the OOM killer, don't just tune the kernel—look at your application's memory usage patterns. Sometimes the best optimization isn't a sysctl setting; it's fixing a memory leak in your code.

Frequently Asked Questions

Q: Should I disable swap entirely on a VPS? A: Generally, no. A small swap partition (even 1-2GB) prevents the kernel from panicking when it hits a temporary memory spike. It's better to have a slow system for a few seconds than a crashed one.

Q: Does oom_score_adj work inside a container? A: Yes, but it's relative to the host's view of the processes. It’s usually better to manage this at the host level or via Docker's native resource constraints.

Q: How do I know if I'm thrashing? A: Watch your si (swap in) and so (swap out) columns in vmstat. If those numbers are consistently non-zero, your system is struggling to keep up with memory demands.