Running Tests in CI: Automating Your Validation Pipeline

Previously in this course, we covered the fundamentals of writing local test scripts in Introduction to Automated Testing: Building Reliable Code. Now that you have a functioning test suite on your machine, it's time to bring that reliability into your GitHub Actions pipeline.

In this lesson, we are moving from manual verification to automated testing, which ensures that every change you push is validated before it ever reaches your production environment.

Why CI Needs Automated Testing

When you run tests locally, you are the only one who sees the results. If you forget to run them, or if you accidentally push code that breaks an existing feature, no one knows until it's too late.

By adding a test execution step to your CI pipeline, you create a "gate." If the tests fail, the pipeline fails. This provides immediate, non-negotiable feedback, ensuring that your codebase remains in a healthy state.

Adding Test Execution to Your YAML

GitHub Actions determines the success or failure of a step based on the exit code of the command you run. In Unix-like systems, an exit code of 0 means success, while any non-zero code indicates a failure.

Let's update your .github/workflows/main.yml file to include a testing step. Assuming you have a Python project with a pytest suite, your workflow should look like this:

YAML
name: CI Pipeline

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: pip install pytest

      - name: Run tests
        run: pytest

Interpreting Pass/Fail Signals

When the pytest command runs, the GitHub runner inspects the exit status.

• If all tests pass: pytest exits with 0. The step is marked green in the GitHub UI, and the job continues (or completes successfully).
• If any test fails: pytest exits with a non-zero code. GitHub Actions immediately stops the execution of that job and marks it as "Failed."

This behavior is critical: you don't want to deploy or merge code that has failed its validation.

Hands-on Exercise: The Broken Build

1. Open your project repository.
2. Ensure you have a test_app.py file with at least one test that asserts a condition (e.g., assert 1 == 1).
3. Update your .github/workflows/main.yml as shown in the example above.
4. Commit and push your changes to GitHub.
5. Navigate to the Actions tab in your repository and watch the workflow run.
6. Challenge: Intentionally break your test (change the assertion to assert 1 == 0), push the change, and observe how the GitHub UI flags the job in red.

Common Pitfalls

• Missing Dependencies: A common mistake is forgetting to install your application's requirements in the CI runner. Remember, the runner is a fresh, empty environment. You must install your libraries (e.g., pip install -r requirements.txt) before running tests.
• Ignoring the Exit Code: If you write a custom shell script to run tests, ensure the script itself exits with the appropriate code. If your script runs tests but exits with 0 regardless of the outcome, the CI will report a success even when the tests fail.
• Environment Mismatches: Sometimes tests pass locally but fail in CI due to different environment variables or missing system-level packages. Always keep your CI environment as close to your local development environment as possible.

FAQ

Q: What happens if I have multiple steps? Does one failing step stop the rest? A: Yes, by default, if a step fails, subsequent steps in that job are skipped. This is intentional; there is no point in running deployment steps if the tests failed.

Q: Can I run multiple test suites? A: Absolutely. You can add multiple run commands in the same step, or create separate steps for different test types (e.g., one for unit tests, one for integration tests).

Q: How do I see why a test failed? A: Click on the failed job in the GitHub Actions UI. You can expand the "Run tests" step to see the full standard output (stdout) from your test runner, which will show the specific line of code that triggered the failure.

Recap

We’ve successfully moved beyond "Hello World" by turning our pipeline into a validation tool. By adding a dedicated run command for your test suite, you’ve ensured that your CI pipeline now provides meaningful signals about the quality of your code on every commit.

Up next: We will explore how to refine these triggers so your pipeline runs only when it's most impactful, by mastering the on: push configuration.