Handling Large Files in CI: Optimization and Storage Strategies

Previously in this course, we covered Automating Releases to handle versioning. This lesson adds a critical layer of performance: keeping your CI pipelines lean by effectively managing large files and assets that don't belong in your standard Git history.

The Problem with Large Files in Git

Git is designed to track changes in source code—typically text. When you add large binary files (like high-resolution images, compiled assets, or datasets) to your repository, you inflate the .git folder size for every developer and every CI runner.

Every time a runner executes git clone or git fetch, it must download the entire history of those large files. This leads to slow job initialization, massive network egress, and eventual pipeline timeouts. If your repo is bloated, your CI isn't just slow; it's fragile.

Strategy 1: Git LFS (Large File Storage)

Git LFS replaces large files in your repository with tiny "pointer files." The actual data is stored in a separate server, and the LFS client downloads the heavy content only when you explicitly need it.

Implementation Steps:

1. Install LFS: Ensure it is installed on your local machine (git lfs install).
2. Track Files: Define which files to track: git lfs track "*.psd".
3. Commit the .gitattributes file: This file tells Git how to handle these extensions.

In GitHub Actions, you don't need to manually install LFS. The actions/checkout action handles it automatically if the repository is configured:

YAML
steps:
  - name: Checkout code
    uses: actions/checkout@v4
    with:
      lfs: true # This flag ensures the runner downloads the LFS objects

Strategy 2: External Storage for Non-Versioned Assets

If your "large files" are build artifacts, temporary datasets, or heavy dependencies that change frequently, do not put them in Git at all.

Instead, treat your repository as a source-only container and fetch assets from a dedicated storage service (like AWS S3, Google Cloud Storage, or an Artifactory instance) during the pipeline.

Worked Example: Fetching from External Storage

Suppose your application needs a 500MB pre-trained model file. Instead of committing it, write a script to fetch it only when the build runs:

YAML
steps:
  - name: Download Model from S3
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_KEY }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET }}
    run: |
      aws s3 cp s3://my-company-assets/model-v1.zip ./assets/
      unzip ./assets/model-v1.zip -d ./assets/

This keeps your repository small (KB or MB) while your CI runner handles the heavy lifting only when necessary.

Comparison: When to use which?

Hands-on Exercise: Optimize Your Repository

1. Identify one file in your project that is larger than 5MB.
2. If it is a static asset that needs versioning, use git lfs to track it.
3. If it is a generated file or a large dataset, move it to an external storage bucket or a public URL, and update your main.yml to use curl or wget to fetch it during the build job.
4. Verify that your git clone time remains under 10 seconds.

Common Pitfalls

• Forgetting lfs: true: If you use LFS but forget the lfs: true flag in actions/checkout, your build will receive the pointer file instead of the actual data, causing "file not found" errors in your build scripts.
• Over-tracking: Don't track small files with LFS. It adds unnecessary complexity. Only use it for files that truly hinder cloning speeds.
• Public exposure: If you move files to external storage, ensure the bucket permissions are correctly scoped to your CI credentials; don't make them publicly readable if they contain proprietary data.

FAQ

Q: Does Git LFS count against my repository storage limit? A: Yes, GitHub provides a specific amount of storage and bandwidth for LFS, which is separate from your standard repository limit.

Q: Can I use both LFS and external storage? A: Absolutely. Use LFS for versioned assets tracked in the repo and external storage for dynamic, non-versioned, or massive files.

Q: Is caching better than LFS? A: They serve different purposes. Caching is for transient data that can be re-generated (like node_modules). LFS is for static binary files that are part of your source code.

Recap

We've moved beyond simple code management by tackling the bloat that kills pipeline performance. By choosing between Git LFS for versioned assets and external storage for everything else, you ensure that your CI/CD environment stays fast and efficient.

Up next: Cross-Repository Workflows — learning how to trigger pipelines across different projects.