Introduction to Jobs: Running One-Off Tasks in Kubernetes

Previously in this course, we explored service accounts to manage identity for your running applications. While most of our work so far has focused on persistent services like Nginx or web APIs, real-world infrastructure often requires executing finite, batch-oriented tasks.

This lesson introduces Jobs, the Kubernetes primitive designed for tasks that perform a specific action and then terminate.

What is a Kubernetes Job?

In Kubernetes, a Job is a controller that creates one or more Pods and ensures that a specified number of them successfully terminate. Unlike a Deployment—which expects its Pods to run indefinitely and restarts them if they crash—a Job is designed for "run-to-completion" workloads.

Think of a Job as a wrapper for a task. Common use cases include:

• Database schema migrations.
• Generating a report or processing a batch file.
• Running a one-time script for system cleanup.

If the Pod finishes its work successfully (exits with code 0), the Job considers the work done. If the Pod fails, the Job controller will retry it based on your configuration until it succeeds or reaches a retry limit.

Running Your First One-Off Task

To run a Job, we define a Job manifest. It looks very similar to the Pod manifests you created in Creating Your First Pod Manifest, but with the kind: Job specification.

Create a file named task.yaml:

YAML
apiVersion: batch/v1
kind: Job
metadata:
  name: pi-calculator
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4

Key Components:

1. restartPolicy: Never: For Jobs, you must set this to Never or OnFailure. You cannot use Always because the goal is for the container to exit successfully.
2. backoffLimit: 4: This tells Kubernetes how many times to retry if the Pod fails before giving up.
3. The Command: The command field overrides the default container entrypoint to run our specific Perl script.

Apply this to your cluster:

Bash
kubectl apply -f task.yaml

Monitoring Job Completion

Unlike long-running services that stay in a Running state, a Job's lifecycle moves through phases: Pending, Running, and eventually Succeeded or Failed.

Check the status of your job:

Bash
kubectl get jobs

You will see output similar to this:

TEXT
NAME            COMPLETIONS   DURATION   AGE
pi-calculator   1/1           12s        15s

The COMPLETIONS column shows that our task finished successfully. You can also inspect the Pod created by the Job:

Bash
kubectl get pods -l job-name=pi-calculator

To see the result of the calculation, fetch the logs:

Bash
kubectl logs job/pi-calculator

Common Pitfalls

1. Forgetting restartPolicy: If you leave the default (Always), Kubernetes will reject your Job manifest. Jobs must be configured to stop.
2. Leaving Jobs in the Cluster: Unlike ephemeral Pods that you delete, completed Jobs stay in your cluster until you manually remove them. Over time, these can clutter your API server. Always clean up old jobs with kubectl delete job <name>.
3. Infinite Loops: If your script is buggy and hangs indefinitely, the Job will never reach the Succeeded phase. Kubernetes provides activeDeadlineSeconds in the spec to forcibly terminate Jobs that run longer than expected.

Exercise

Modify the task.yaml file to run a simple shell command that fails, such as command: ["/bin/sh", "-c", "exit 1"]. Apply it, observe the status using kubectl get jobs, and watch how Kubernetes retries the Pod based on the backoffLimit. Then, delete the job to clean up the failed attempts.

Summary

Jobs are the primary tool for batch processing in Kubernetes. By moving from manual scripts to defined Jobs, you gain the benefit of cluster-managed retries, status reporting, and logs for every execution. This is a significant step toward robust automation, whether you're managing simple scripts or complex Linux task automation.

FAQ

Q: Do I need to delete a Job after it succeeds? A: Yes. Kubernetes keeps completed Jobs around so you can inspect their logs and status. Once you've verified the output, you should delete the resource.

Q: Can a Job run multiple Pods? A: Yes. By setting parallelism and completions in the Job spec, you can spin up multiple Pods to process a large queue of work in parallel.

Q: What happens if I update a Job manifest? A: You generally cannot update a Job in place once it has started. You typically delete the existing job and create a new one.

Up next: We will take this one step further by automating these tasks on a schedule using CronJobs.