sdlc.dev

Git path · Scale and trust · Topic 12 of 13

Large repositories: shallow, partial and sparse clones, and LFS

Not started

By default git clone downloads every object in the history of every branch and writes every file to disk. For a large monorepo or a repository with years of binary assets that can mean gigabytes and many minutes. Git has three built-in ways to take less: a shallow clone cuts history, a partial clone skips file contents until they are needed, and sparse checkout writes only some directories. Git LFS takes a different approach and keeps large binaries out of the object database altogether.

Why it matters: A monorepo or a repository full of assets can take many minutes to clone in full. Most jobs need a fraction of it.

  1. 1Shallow clones cut history

    --depth 1 fetches only the latest commit. Fast for CI builds, but log, blame and merge-base don't see older history.

  2. 2Partial clones fetch file contents on demand

    --filter=blob:none gets all commits and trees but only the blobs you check out; the rest arrive when needed.

  3. 3Sparse checkout limits the working tree

    Only the directories you list are written to disk, which keeps status and builds fast in a monorepo.

  4. 4LFS keeps big binaries out of the object database

    Git stores a small pointer file; the real content sits on an LFS server and is downloaded at checkout.

1

How it works

The explanation, with examples you can run.

Try all three on a small repository

A local repository with five services stands in for a big one. Partial clone needs the server to allow filters, which hosting services do; locally you switch it on with uploadpack.allowFilter.

mkdir big-demo && cd big-demogit init -q -b main srcfor n in 1 2 3 4 5; do mkdir src/svc$n; echo "code $n" > src/svc$n/main.txt; git -C src add .; git -C src commit -q -m "Commit $n"; donegit -C src config uploadpack.allowFilter truegit clone -q --depth 1 "file://$PWD/src" shallowgit -C shallow log --onelinegit -C shallow rev-parse --is-shallow-repository
b2ac409 Commit 5true

The shallow clone has only the tip commit. That is ideal for CI that builds once and throws the clone away, and it is what many CI systems do by default. The cost: git log, git blame and finding a merge base stop at the cut. git fetch --deepen=50 or git fetch --unshallow brings history back.

git clone -q --filter=blob:none "file://$PWD/src" partialgit -C partial log --oneline | wc -lgit -C partial config remote.origin.partialclonefilter
       5blob:none

The partial clone has all five commits and every tree, but only the blobs needed for the files checked out. When you check out an old commit or run a diff that needs an old version, Git fetches the missing blobs from the remote (the "promisor") on demand. For day-to-day development in a big repository this is usually the best default. --filter=tree:0 also skips trees and suits throwaway builds; it is slower if you later need history.

git clone -q --filter=blob:none --sparse "file://$PWD/src" sparsels sparsegit -C sparse sparse-checkout set svc2ls sparse
svc2

With --sparse, the clone starts with only top-level files (there are none here), and sparse-checkout set adds the directories you work on. Cone mode, the default, matches whole directories, which keeps it fast. Combined with a partial clone, Git downloads only the blobs for those directories. wc -l pads its count with spaces on macOS; on Linux it prints just 5.

Git LFS for large binaries

Git stores every version of every file, and binary files such as videos, models and design files don't delta-compress well. Once a big file is committed, every future clone downloads it, even if a later commit deletes it. Git Large File Storage, a separate extension, replaces tracked files in the repository with small text pointers and stores the real content on an LFS server (GitHub, GitLab and Bitbucket provide one).

git init -q -b main lfs-demo && cd lfs-demohead -c 100000 /dev/urandom > design.psdgit lfs installgit lfs track "*.psd"git add .gitattributes design.psdgit commit -q -m "Add design file via LFS"cat .gitattributesgit cat-file -p HEAD:design.psd
Updated Git hooks.Git LFS initialized.Tracking "*.psd"*.psd filter=lfs diff=lfs merge=lfs -textversion https://git-lfs.github.com/spec/v1oid sha256:3bc10117a15eedb0de1bbf832527122ba5724febe85bcf29cf32600f646887acsize 100000

git lfs track writes a rule to .gitattributes. The blob Git committed for design.psd is the three-line pointer: the SHA-256 of the real content (random here, so yours will differ) and its size. The 100 kB of real content sits in .git/lfs/objects and is uploaded to the LFS server on push; clones download it only for the commits they check out. LFS is a separate program, not part of Git: this was run with git-lfs 3.4.1 in an Ubuntu container, since it is not installed on the Mac used for the other examples. Moving files that are already committed into LFS means rewriting history with git lfs migrate.

Other levers for very large repositories: git maintenance start schedules background repacking and prefetching, and scalar (shipped with Git) clones with partial clone, sparse checkout and maintenance already set up.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A CI job only builds the latest commit. Which clone is the usual choice?
  2. 2.What does a blob:none partial clone leave out?
  3. 3.Why is committing a 2 GB video directly into Git a problem even after you delete it in a later commit?

0 of 3 right so far. Get all three to finish this topic.

Go deeper