GitHub Actions Self-Hosted Runners: Scaling Ephemeral Docker Containers

Last month, our CI/CD pipeline started hitting a bottleneck. We were running a single, permanent VM for our GitHub Actions self-hosted runners, and it was constantly choking under the weight of concurrent PR builds. The queues were getting long, and the "dirty" state left behind by previous builds was causing intermittent test failures that took me about three hours to debug one Friday afternoon.

I realized then that static runners are an anti-pattern. If you're still managing persistent VMs for your CI/CD, you're inheriting state drift. Instead, I moved our infrastructure to ephemeral Docker runners that spin up, execute one job, and vanish.

Why Ephemeral Runners Matter

When you use persistent runners, every job is a gamble. One build might leave behind a stray node_modules folder or a corrupted temp file, and the next build inherits that mess. By using ephemeral Docker containers, you enforce a clean state for every single execution.

This approach is fundamentally different from zero-downtime deploy with GitHub Actions where you manage persistent service availability. Here, the goal is total disposal. Once the GitHub Action job finishes, the container is killed, ensuring no secrets or artifacts leak into subsequent runs.

Architecting the Docker-Based Runner

To get this working, you don't need a complex Kubernetes cluster. You can start with a standard Linux host running Docker. The core idea is to use the actions-runner-controller (ARC) or a simpler sidecar pattern if you're on a single host.

For our setup, I opted for a lightweight wrapper around the official GitHub runner image. Here is the basic Dockerfile I use:

Dockerfile
FROM ghcr.io/actions/actions-runner:latest

# Install your specific build dependencies
RUN sudo apt-get update && sudo apt-get install -y \
    build-essential \
    python3-pip \
    && sudo rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["./bin/Runner.Listener", "run", "--once"]

The --once flag is the secret sauce. It tells the runner to exit immediately after it finishes a single job. By combining this with a simple docker-compose setup or a systemd service, you can ensure that your CI/CD pipeline stays responsive without manual intervention.

Scaling Self-Hosted Runners Automatically

If you have a burst of activity, a single container won't cut it. You need a way to spin up multiple instances on demand. While ephemeral Linux environments: bootstrapping with Cloud-Init and Terraform are great for full-stack preview environments, for simple build jobs, I prefer a simple Docker-in-Docker (DinD) approach on a beefy host.

To achieve auto-scaling without the overhead of a full K8s implementation, we use a loop in a shell script that checks the pending job count via the GitHub API and spawns containers accordingly:

Bash
#!/bin/bash
# A crude but effective auto-scaler
PENDING_JOBS=$(curl -s -H "Authorization: token $GH_TOKEN" \
    https://api.github.com/repos/org/repo/actions/runs?status=queued | jq '.total_count')

if [ "$PENDING_JOBS" -gt 0 ]; then
    docker run -d --name runner-$(date +%s) my-runner-image
fi

This isn't as elegant as a horizontal pod autoscaler, but it works reliably for small-to-medium teams. It keeps costs low because you aren't paying for idle CPU cycles.

Managing Security and Cleanup

One risk with self-hosted runners is that they have access to your internal network. I’ve found that using WireGuard mesh networking: secure your VPS cluster communication is the best way to isolate these runners. Even if a runner is compromised during a build, it’s contained within a specific network segment that only has access to the bare minimum resources it needs.

Also, don't forget to prune your Docker system. Even with ephemeral containers, you'll accumulate dead images and volumes. I use a systemd timers: the better way to handle Linux automation job to run docker system prune -f every night at 3 AM.

Common Pitfalls

• Image Bloat: Don't put your entire production environment in the runner image. Keep it lean.
• Job Timeouts: Ephemeral runners can take a few seconds to start. If your jobs are tiny, the overhead of pulling the image might exceed the build time.
• Orphaned Containers: If the runner crashes, the container might stick around. Always add a --rm flag to your docker run command or use a cleanup script.

I’m still tinkering with the auto-scaling logic. Occasionally, the GitHub API rate limits us if we poll too aggressively, so I’m looking into using Webhooks to trigger new runner creation instead of polling. It’s a work in progress, but the move to ephemeral runners has already saved us about 10 hours of "fix-the-broken-build" time per week.

FAQ

Can I run these on a Raspberry Pi? Yes, as long as you build the runner image for ARM64. The GitHub runner binary supports it out of the box.

What happens if the host runs out of memory? If you don't set Docker resource limits (--memory="2g"), a rogue build can crash the host. Always limit your containers.

Is it faster than GitHub-hosted runners? Usually, yes, because you can provision the hardware to have faster NVMe storage and more RAM than the standard GitHub runner tiers.