Handling Large Files: Mastering Git LFS for Performance

Previously in this course, we covered analyzing repository health to ensure your project remains maintainable. Today, we address a common pain point: what happens when your project needs to include assets like high-resolution images, videos, or compiled binaries.

Git is designed to store text efficiently by calculating the differences (deltas) between versions. When you add a large binary file, Git cannot calculate these deltas effectively. Instead, it stores a full copy of that file in the repository's internal database for every single version. This leads to massive, slow repositories that take forever to clone.

The Git LFS Solution

Git Large File Storage (LFS) replaces large files with text pointers inside your Git repository, while storing the actual file content on a separate remote server. When you clone or pull a branch, Git LFS downloads the specific versions of the files you need, keeping your local repository lightweight and performant.

Installing and Configuring Git LFS

Before you can track large files, you need to ensure the extension is installed on your machine.

1. Installation: If you are on macOS, run brew install git-lfs. On Windows or Linux, download the installer from the official Git LFS website.
2. Initialization: Once installed, you must initialize it once per user account:

   Bash
   git lfs install
   

   This command sets up the necessary global Git hooks to intercept your file operations.

Tracking Large Files

Tracking a file with LFS is simple. Instead of just adding the file to Git, you tell LFS to watch for that specific file pattern.

Suppose your project needs a high-resolution logo in a assets/ folder. Instead of running git add, follow these steps:

1. Track the file type:

   Bash
   git lfs track "assets/*.png"
   

   This creates (or updates) a file named .gitattributes in your repository. This file is crucial—it tells Git which files should be handled by LFS.

2. Add the attribute file:

   Bash
   git add .gitattributes
   

3. Add and commit your large file:

   Bash
   git add assets/logo.png
   git commit -m "Add high-res logo using LFS"
   

When you run git show, you'll notice that the file doesn't show the binary content, but a small text pointer:

TEXT
version https://git-lfs.github.com/spec/v1
oid sha256:4d7a...
size 1234567

Hands-on Exercise

To practice, follow these steps in your project directory:

1. Create a dummy "large" file: dd if=/dev/zero of=data.bin bs=1M count=10 (this creates a 10MB file).
2. Initialize LFS tracking for this file: git lfs track "data.bin".
3. Add both .gitattributes and data.bin to the staging area and commit them.
4. Verify the LFS status by running git lfs ls-files.

Common Pitfalls

• Forgetting .gitattributes: If you don't commit the .gitattributes file, others on your team will download the full binary instead of the pointer, or worse, they won't be able to access the file at all.
• Converting existing files: If you have already committed a large file to your standard Git history, git lfs track won't automatically move it. You would need to use tools like git filter-repo to rewrite your history, which is an advanced operation. Always track large files before committing them.
• Exceeding Storage Limits: While LFS keeps your repo small, it consumes storage on GitHub. Keep an eye on your account's LFS data quota.

FAQ

Q: Do I need to run git lfs install for every repository? A: No, git lfs install is a global command. You only need to run it once per machine.

Q: Can I use LFS for source code? A: No. LFS is for static binary assets. Storing source code in LFS prevents Git from showing diffs or merging changes, which defeats the purpose of version control.

Q: How do I know if a file is being tracked by LFS? A: Run git lfs ls-files to see a list of all files currently managed by the LFS extension.

Recap

We've learned that Git isn't built to handle large binaries and that using Git LFS provides a way to maintain performance by replacing massive blobs with lightweight pointers. By properly configuring .gitattributes, you ensure your team members aren't forced to download gigabytes of data just to view a project history. Proper file management is an essential part of handling large files in CI workflows, which we'll touch on later.

Up next: We will learn how to finalize your project documentation with a proper README file.