Mastering Matrix Builds: Multi-Version Testing in CI/CD

Previously in this course, we explored parallel execution to run independent jobs at the same time. While parallel jobs help speed up your pipeline, they often lead to "configuration drift" if you manually copy-paste jobs to test against different versions. Today, we add the matrix build strategy to your toolkit, allowing you to scale your test coverage without scaling your YAML file size.

The Problem: Combinatorial Explosion

As your project matures, you need to ensure it runs correctly on various configurations—perhaps different versions of Python (3.9, 3.10, 3.11) or multiple operating systems (Ubuntu, macOS, Windows).

If you define these manually as separate jobs, your workflow file becomes bloated and hard to maintain. If you decide to add a new Python version, you have to remember to update every single job definition. A matrix build solves this by defining the dimensions of your testing environment, letting the CI provider generate the combinations for you automatically.

Implementing a Strategy Matrix

A matrix build uses the strategy and matrix keywords within a job. Instead of hardcoding values, you define variables that the runner will iterate through.

Here is how you transform a standard job into a matrix-powered machine:

YAML
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.9', '3.10', '3.11']
        os: [ubuntu-latest, windows-latest]
    
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run tests
        run: pytest

In this example, GitHub Actions creates a "Cartesian product" of the variables. It will spawn 6 distinct jobs:

1. Python 3.9 on Ubuntu
2. Python 3.9 on Windows
3. Python 3.10 on Ubuntu
4. Python 3.10 on Windows
5. Python 3.11 on Ubuntu
6. Python 3.11 on Windows

This is the power of cross-platform and cross-version testing: you cover your bases with minimal code. For those working with complex environments, this approach ensures your software maintains the same reliability as first steps into unit testing across every supported platform.

Hands-on Exercise: Configure Your Matrix

1. Open your existing test workflow YAML file.
2. Replace the single runs-on value with a strategy block as shown above.
3. Update your setup-python action (or equivalent for your language) to use the ${{ matrix.python-version }} syntax.
4. Push the changes to your repository.
5. Observe the "Actions" tab in your repository. You will see a grid of jobs running simultaneously.

Common Pitfalls

• Forgetting fail-fast: By default, if one matrix job fails, GitHub Actions cancels all other jobs in that matrix. If you want to see the results of all combinations regardless of individual failures, add fail-fast: false under the strategy block.
• Over-complicating the matrix: Adding too many variables (e.g., 5 OS versions × 5 language versions) can quickly hit your account's concurrent job limits. Keep your matrix focused on the configurations that matter most to your users.
• Ignoring Platform Differences: Sometimes, tests fail on Windows that pass on Linux due to path separators. Ensure you are using cross-platform coding practices, like those discussed in working with the path module in Node.js, to avoid "false negative" test failures.

FAQ

Q: Can I exclude specific combinations? A: Yes. You can use the exclude key under matrix to skip specific combinations that don't make sense (e.g., testing on an unsupported OS/version pair).

Q: Does every job in a matrix run the same steps? A: Yes, but the environment variables differ. You can use ${{ matrix.variable_name }} to inject the current iteration's value into your shell commands or action configurations.

Q: How do I identify which job failed in the logs? A: GitHub’s UI displays the matrix values next to the job name in the sidebar, making it immediately clear which configuration triggered a failure.

Recap

Matrix builds are the industry-standard way to manage multi-version testing. By defining variables in a strategy block, you reduce code duplication, improve test coverage, and gain a clear view of how your application behaves across different environments. You’ve moved from manually managing single-environment tests to orchestrating a robust, automated validation suite.

Up next: We will learn how to use the if keyword to apply conditional logic to your jobs, allowing you to skip steps based on branches or event types.