Multi-Job Workflows: Orchestrating Pipelines with 'needs'

Previously in this course, we covered artifact management to pass files between steps. While helpful, putting every single task into one giant job makes your pipeline hard to read and debug. In this lesson, we add modularity to our orchestration by splitting our pipeline into multiple jobs that depend on each other.

The Problem with Monolithic Jobs

When you first started with understanding runners and jobs, you likely kept everything—linting, testing, and building—in one job. This works for small projects, but as your CI/CD pipeline grows, this "all-in-one" approach becomes a bottleneck.

If your build fails, you want to know immediately if it was because of a code style issue (linting) or a logic error (testing). By splitting these into separate jobs, you create distinct stages that provide clearer feedback in the GitHub UI.

Orchestration with the 'needs' Keyword

In GitHub Actions, jobs run in parallel by default. To create a sequence, you use the needs keyword. It tells the workflow engine: "Do not start this job until the listed jobs have completed successfully."

Think of this as a Directed Acyclic Graph (DAG) of tasks. Similar to how you might manage LLM agent orchestration using state machines, we are defining a state-based flow for our code delivery.

Worked Example: A Two-Stage Pipeline

Let’s split our pipeline into a lint job and a test job. The test job will only run if the linting passes.

YAML
name: Multi-Job Pipeline

on: [push]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Linter
        run: echo "Checking code style..." && exit 0 # Replace with your actual linter

  test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Tests
        run: echo "Running unit tests..." && exit 0 # Replace with your actual tests

In this example, if the lint job fails, the test job is automatically skipped. This prevents wasting runner time on tests when the code doesn't meet your quality standards, much like how you would enforce business logic vulnerabilities checks before finalizing a process.

Hands-on Exercise

1. Open your existing workflow file in .github/workflows/.
2. Split your current single job into two: lint and test.
3. Add needs: lint to your test job.
4. Commit and push your changes.
5. Observe the GitHub Actions tab; you should see two boxes connected by an arrow, representing the dependency.

Common Pitfalls

• Missing Dependencies: If you forget the needs keyword, your jobs will run simultaneously. This can cause race conditions if, for example, your test job tries to run before the build job has finished preparing the application files.
• Circular Dependencies: You cannot have Job A need Job B, while Job B needs Job A. GitHub will reject the YAML configuration.
• Failure Propagation: Remember that if a parent job fails, all downstream jobs that "need" it will be skipped. This is usually intended behavior, but it can be confusing if you aren't expecting it.

FAQ

Q: Can a job depend on multiple jobs? A: Yes. You can provide a list to the needs keyword: needs: [job1, job2]. The job will wait for both to complete successfully.

Q: Do I need to share files between jobs? A: Yes. Since each job runs on a fresh runner instance, they do not share a file system. You must use actions/upload-artifact and actions/download-artifact to move files between them.

Q: What if I want a job to run even if the previous job failed? A: You can use conditional expressions like if: always() or if: failure(). We will cover this in detail when we discuss conditional execution.

Recap

Multi-job workflows allow you to structure your CI/CD pipeline into logical, sequential stages. By using the needs keyword, you ensure that tasks only run when their prerequisites are met, providing cleaner logs and faster feedback loops.

Up next: We will look at Parallel Execution, where we learn how to run independent jobs at the same time to speed up your pipeline.