Running background workers with systemd for production reliability

I spent an entire weekend chasing down a ghost in a production environment where background tasks kept silently dying. We were using a crude shell script and a cron job to "ensure" the workers were running, but it was brittle, leaked memory, and failed to restart after a kernel update. If you're tired of babysitting your application processes, it's time to stop hacking together custom solutions and start running background workers with systemd.

Systemd is built into almost every modern Linux distribution, and it’s arguably the most reliable way to manage persistent application processes. It handles process monitoring, automatic restarts, and logging out of the box.

Why systemd beats the alternatives

Before I committed to systemd, I tried using supervisord. While it’s a decent tool, it’s yet another dependency to manage, update, and secure. Systemd, on the other hand, is already part of your OS. When you use it, you get native integration with journald for logs and cgroups for resource limiting.

If you're dealing with reliable background jobs: mastering Laravel queues, retries, and idempotency, you need the worker process itself to be as stable as the database connection. A process manager that restarts your worker in 50ms after a crash is the difference between a minor delay and a support ticket.

Creating your first service file

To get started, you'll need to create a unit file in /etc/systemd/system/. Let’s call it worker.service. A basic configuration looks like this:

INI
[Unit]
Description=Background Worker for App
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/var/www/app
ExecStart=/usr/bin/php /var/www/app/artisan queue:work --sleep=3 --tries=3
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

A few things to note here:

• Restart=always: This ensures that if your worker crashes due to a segmentation fault or an OOM (Out of Memory) event, systemd will bring it back up.
• RestartSec=5: I prefer a 5-second delay to avoid thrashing your database if the worker is failing because of a connection issue.
• User/Group: Never run these as root. Use a dedicated service account with the minimum permissions required to execute the code.

Managing logs and environment variables

One of the biggest headaches with background workers is environment configuration. Instead of hardcoding secrets in your service file, use the EnvironmentFile directive.

INI
[Service]
EnvironmentFile=/etc/default/app-worker
ExecStart=/usr/bin/php /var/www/app/artisan queue:work

Your /etc/default/app-worker file should be restricted to chmod 600 so only the owner can read your sensitive keys. For logging, stop worrying about custom file handlers. Since you’re running background workers with systemd, you can simply stream your output to stdout and stderr. Systemd captures this automatically, and you can view it with journalctl -u worker.service -f.

Handling multiple workers

If your workload grows, you might need more than one instance. You could copy the service file, but that’s tedious. Instead, use template units. Rename your file to worker@.service and change the ExecStart line to use %i as an identifier.

You can then start multiple instances with: systemctl start worker@1 systemctl start worker@2

This is a massive upgrade over managing separate files. It’s clean, version-controlled, and easy to audit. While scaling Laravel queues on Kubernetes: A KEDA implementation guide is the right move for massive, elastic infrastructure, systemd is perfectly capable of handling dozens of workers on a single VPS or dedicated bare-metal server.

Common pitfalls to watch for

I’ve learned the hard way that systemd isn't magic. Here are two things that caught me off guard:

1. Environment pathing: Systemd doesn't load your .bashrc or .profile. If your worker relies on an installed tool like ffmpeg or node, use the full absolute path in your ExecStart command (e.g., /usr/local/bin/ffmpeg instead of just ffmpeg).
2. Zombie processes: If your application spawns sub-processes that don't clean up, they can hang around. Systemd helps here, but you should ensure your application code handles SIGTERM signals correctly so that when you run systemctl stop, the worker shuts down gracefully rather than being killed mid-job.

Conclusion

Running background workers with systemd has saved me hours of on-call time. It’s predictable, it’s standard, and it keeps your services humming without needing a complex orchestrator.

If you’re still using a while true loop inside a screen session or a sketchy cron script, do yourself a favor and migrate to systemd this week. You’ll spend less time debugging process states and more time building features. I’m still experimenting with using PrivateTmp=true to further isolate these processes, but for now, the basic service definition is rock solid.