CI/CD for ML: Automating MLOps Pipelines and Model Versioning

Previously in this course, we covered Project Milestone: Inference Optimization for Production, where we tuned our models for speed. Today, we shift from optimization to reliability: how to build the infrastructure that ensures your model remains performant and bug-free as you iterate.

In standard software engineering, CI/CD is a solved problem. In MLOps, however, we deal with "dual-versioning": you aren't just versioning code, you are versioning the model weights and the data that produced them. A failure in your deployment pipeline shouldn't just break the build; it should prevent a degraded model from reaching production.

The MLOps CI/CD Architecture

To achieve true automation, we must treat the model artifact as a first-class citizen in our CI/CD pipeline. The pipeline needs to handle three distinct phases:

1. Continuous Integration (CI): Validating the model code and training logic.
2. Continuous Testing (CT): Running automated performance benchmarks against a hold-out test set.
3. Continuous Deployment (CD): Packaging the verified model and promoting it to the inference registry.

Implementing Automated Unit Tests for Model Logic

Most engineers test their data loaders and API endpoints but skip the "model logic." If your custom transformer layer has a bug in its attention mask, traditional unit tests won't catch it. You need to test the tensor shapes and numerical stability of your components.

PYTHON
import torch
import unittest

class TestTransformerBlocks(unittest.TestCase):
    def test_attention_mask_shape(self):
        # Ensure your custom attention mechanism handles masks correctly
        batch, seq_len, head_dim = 2, 10, 64
        model = CustomAttention(head_dim=head_dim)
        mask = torch.ones(batch, 1, 1, seq_len)
        
        output = model(torch.randn(batch, seq_len, head_dim), mask=mask)
        self.assertEqual(output.shape, (batch, seq_len, head_dim))

    def test_forward_pass_nans(self):
        # Catch numerical instability early
        model = MyProductionModel()
        input_tensor = torch.randn(1, 128)
        output = model(input_tensor)
        self.assertFalse(torch.isnan(output).any(), "Model produced NaNs!")

Integrate these into your pytest suite and run them on every commit. If the forward pass produces NaNs or the shapes don't match, the build fails before any training starts.

Version Control for Model Weights

Code lives in Git, but model weights are too large. Storing weights in Git is a common anti-pattern that leads to repository bloat. Instead, use a model registry (like MLflow, DVC, or a simple S3 bucket with versioning enabled).

Your CI/CD pipeline should generate a unique identifier for every model artifact:

1. Git Commit SHA: Links the code version to the training run.
2. Experiment ID: Links the hyperparameters and data version.
3. Semantic Versioning: Allows you to tag models as v1.0.0-rc1 or v1.0.0-stable.

When you deploy, your deployment script fetches the model via this identifier, ensuring "what you tested is what you deploy."

Hands-on Exercise: The Artifact Promotion Workflow

In your current project, create a test_model.py script that validates your model’s output against a small "golden" dataset (a set of inputs with known expected outputs).

1. Write a test that loads the latest model artifact from your local storage.
2. Pass the golden dataset through the model.
3. Assert that the output matches the expected metrics (e.g., Accuracy > 0.85).
4. Integrate this into a GitHub Action that triggers on git push.

Common Pitfalls

• Ignoring Environment Parity: You might test on a CPU and deploy on a GPU, leading to silent numerical differences. Always test in an environment that mimics production—use Containerization Basics: Packaging ML Pipelines for Deployment to keep your runtime consistent.
• Assuming Code Versioning Equals Model Versioning: Just because the code is the same doesn't mean the weights are. Always log your metadata (Git hash + DVC hash) together.
• Manual Deployment: If you are still manually copying .pth or .onnx files to a server, you are at risk. Your CI/CD should handle the promotion of the artifact to the registry and notify the inference service to pull the new version.

Recap

Effective MLOps requires treating the model as an immutable artifact. By enforcing unit tests for model logic, versioning weights via registries, and automating the promotion process, you ensure that your production environment remains stable. Remember, Prompt management strategies for reliable LLM deployment pipelines often follow similar patterns—if you can automate the test, you can automate the deployment.

Up next: We’ll dive into Continuous Training (CT) Pipelines, where we trigger automatic retraining when our model performance begins to degrade in the wild.