Logging Drivers: A Guide to Docker Log Management and Rotation

Previously in this course, we explored Working with Container Logs: A Practical Guide to Debugging, where we covered the basics of streaming and inspecting standard output. While that works for local development, production environments require more robust handling. In this lesson, we will shift from default local storage to advanced logging drivers, enabling us to forward data to external systems and implement essential log rotation.

Understanding Logging Drivers

By default, Docker uses the json-file logging driver. It collects the stdout and stderr streams of your container, serializes them into JSON, and stores them in a file on your host machine. While simple, this approach has a critical flaw: if your application is noisy, it will eventually consume all available disk space, potentially crashing the host.

A logging driver acts as a middleman. Instead of writing files directly, Docker hands the logs to a specific "driver" configured to handle them—whether that’s sending them to a remote Syslog server, a cloud-based aggregator, or simply rotating the files to keep disk usage in check.

Configuring the Syslog Driver

The syslog driver is a standard choice for Linux-based infrastructure. It forwards container logs to a local or remote syslog daemon. This is particularly useful for centralized management, where you can aggregate logs from many containers into one place, as discussed in Strategic Logging: Mastering Observability and Debugging.

To use the syslog driver for a specific container, you pass the --log-driver flag during startup:

Bash
docker run -d \
  --log-driver syslog \
  --log-opt syslog-address=udp://127.0.0.1:514 \
  --name my-web-app \
  nginx

In this example, Docker stops writing to local JSON files and instead pushes every log line to the UDP address 127.0.0.1:514.

Managing Log Rotation

Even if you aren't using an external aggregator, you must manage your log files. Without rotation, a single container can grow its log file until the partition is full. You can enforce rotation even while using the default json-file driver.

Let's update our configuration to limit log size and retention:

Bash
docker run -d \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  --name my-rotated-app \
  my-app-image

What this does:

• max-size=10m: Once a log file reaches 10MB, Docker triggers a rotation.
• max-file=3: Docker keeps only the three most recent files, automatically deleting the oldest ones to save space.

Comparison of Logging Approaches

Hands-on Exercise: Implement Rotation

1. Start a container that generates logs (we'll use alpine with a loop): docker run -d --name logger-test --log-opt max-size=1k --log-opt max-file=2 alpine sh -c "while true; do echo 'log entry'; sleep 1; done"
2. Check the size of the log file on your host. If you are on Linux, it is typically in /var/lib/docker/containers/<id>/<id>-json.log.
3. Wait for the files to rotate. You will notice that once the files hit the 1KB limit, Docker creates a second file, then deletes the first one to maintain a total of two files.

Common Pitfalls

• Ignoring the Driver Global Default: You can set the logging driver globally in /etc/docker/daemon.json. If you change this, existing containers won't automatically switch; you must recreate them.
• Network Latency: Using remote drivers (like gelf or fluentd) can cause your application to block if the network connection is slow or the log server is unreachable. Always consider using a local buffer or a sidecar agent.
• Missing Logs: If you change the driver to something like none, you lose all visibility. Only do this if you have an alternative way to capture application-level logs inside the container.

FAQ

Can I change the log driver of a running container? No. The logging driver is configured when the container is created. You must stop and remove the container, then run it again with the new configuration.

How do I view logs if I'm not using json-file? If you use syslog or fluentd, you view the logs on the destination server or the monitoring platform where they are forwarded, not via docker logs.

Is there a way to send logs to Cloud providers? Yes. Docker supports drivers like awslogs (for CloudWatch) and gcplogs. These require specific permissions/credentials setup on the host machine.

Recap

We’ve moved from simple local log files to structured management. By using logging drivers and rotation, you prevent disk space issues and ensure your logs are piped to where your operations team can actually find them. Remember that robust logging is a cornerstone of Monitoring Deployed APIs: Logs, Health Checks, and Status.

Up next: We will dive into Container Health Monitoring to ensure that if your app crashes, Docker knows exactly when to restart it.