When a repository grows past tens of gigabytes, Git starts to crawl. Developers notice it first during simple commands. A simple git status that once took milliseconds begins to hang for five, ten, or thirty seconds. Running git fetch turns into a coffee break. This slowdown is not a minor annoyance. It directly hits developer velocity and breaks continuous integration pipelines.
Git was designed in 2005 to manage the Linux kernel. The kernel is large, but it is not an enterprise monorepo containing millions of files and gigabytes of binary assets. Git is a content-addressable storage system that tracks files as blobs, directories as trees, and commits pointing to those trees. When you run a command like git status, Git must compare the state of every file in your working directory with the index file (.git/index) and the current commit tree. If your repository contains two million files, Git performs two million stat system calls to check for modifications. This file system traversal is the primary bottleneck.
The index is a flat binary file listing every path in the repository along with its SHA-1 hash, file size, and modification time. Every time you run status or switch branches, Git reads and writes this file. When the index file grows to hundreds of megabytes, disk I/O and serialization overhead become massive. Even if you only care about a single folder containing three files, Git still scans the entire index.
Historically, engineers tried to solve this with shallow clones:
git clone -depth 1 <repository-url>This fetches only the latest commit. While it speeds up initial downloads, it creates problems down the road. You lose history, which makes running git log or git blame impossible. Merging branches becomes error-prone because Git lacks the common ancestor history to resolve conflicts cleanly.
A better alternative is partial cloning, introduced in newer Git versions. Instead of truncating history, partial clones omit file contents (blobs) or directory structures (trees) until they are actually needed. You run:
git clone -filter=blob:none <repository-url>This command downloads the entire commit history and all tree objects, but skips downloading the actual file contents. The initial clone is small and fast. When you check out a branch or open a file, Git automatically downloads the missing blobs from the remote server in the background.
If your repository contains 50GB of historical files and code, a standard clone downloads all 50GB. A blob-less partial clone might only download 500MB of metadata. You save network bandwidth and disk space immediately. The trade-off is a slight delay when you first open a file that you have not accessed before, as Git must perform an on-demand HTTP request to fetch the blob. For developers working on a small subset of a massive codebase, this trade-off is highly favorable.
Tree-less clones go a step further. You run:
git clone -filter=tree:0 <repository-url>This omits both trees and blobs, downloading only the commit graph. Git downloads trees and blobs on demand as you checkout specific commits. While this makes the initial clone incredibly fast, it makes operations like git log -p or branch switching slow because Git has to fetch tree objects constantly. For most monorepo setups, blob-less clones represent the sweet spot between speed and offline usability.
Even with a partial clone, your working directory still contains every single directory structure if you do a full checkout. This means your file system still has to track millions of files, keeping the status bottleneck alive. To solve this, you use sparse checkouts. Sparse checkout allows you to tell Git to only populate a specific subset of directories in your working directory.
You initialize it by running:
git sparse-checkout init -coneThe -cone option is critical. It restricts the patterns you can use to full directories rather than arbitrary glob patterns. This restriction allows Git to use highly optimized algorithms to write the index and scan the working directory. Without cone mode, Git has to evaluate complex regular expressions against every file path, which kills performance.
Once initialized, you define the directories you want to work on:
git sparse-checkout set apps/payment shared/typesGit immediately updates your working directory, removing all other files and folders. They still exist in the remote repository and the Git history, but they do not occupy space on your hard drive, and Git does not scan them during status checks. This drops the file count in your working directory from millions to thousands, bringing status times back down to milliseconds.
Even with sparse checkouts, Git still has to query the operating system to see if files have changed. On macOS and Windows, this can still be slow if the repository is large. You can bypass Git's native file system scanning by enabling the built-in file system monitor (FSMonitor). You run:
git config core.fsmonitor trueInstead of scanning the disk itself, Git queries a background daemon that integrates with the operating system's native file change notification APIs, such as FSEvents on macOS or ReadDirectoryChangesW on Windows. The daemon keeps a running list of changed files. When you run git status, Git asks the daemon what changed since the last check. The daemon returns a tiny list of files, and Git only updates those in the index. This turns a linear scan of the working directory into a constant-time lookup.
Another optimization is upgrading the index file format. By default, Git often uses index format version 2 or 3. Version 4 introduces path compression, which reduces the size of the index file on disk by up to 70%. A smaller index file means faster read and write times. You can enable this by running:
git config -global index.version 4This is particularly effective when combined with sparse checkouts. The index file size drops, and operations like git add and git commit speed up because Git has less data to serialize and write to disk.
As a repository accumulates millions of commits, traversing the commit history to generate logs or calculate merge bases becomes slow. Git solves this by building a commit-graph file. The commit-graph file is a structured binary representation of the commit history that avoids parsing raw commit object files. You can generate this manually:
git commit-graph write -reachableRunning these optimization commands manually is tedious. Modern Git contains a built-in maintenance tool that automates these tasks in the background. You can enable it by running:
git maintenance startThis schedules background tasks that periodically run garbage collection, write commit-graphs, and optimize the index. It ensures the repository performance does not degrade over time.
For codebases that are too massive for even partial clones and sparse checkouts, virtual file systems are the ultimate solution. Microsoft originally developed VFS for Git (formerly GVFS) to scale Windows development, which lived in a single giant repository. VFS for Git virtualization works at the OS file system driver level. It projects the entire repository onto the disk as if all files are present, but the files are actually empty placeholders.
When an application attempts to read a file, the virtual file system driver intercepts the read call, downloads the file content from the remote server, populates the file on disk, and allows the read call to complete. To the operating system and developer tools, the repository looks like a standard local directory.
VFS for Git required custom OS kernel drivers, which made it difficult to maintain and port to other operating systems. Microsoft and the Git community shifted focus to Scalar. Scalar is a command-line tool that configures Git to use built-in features like partial clones, sparse checkouts, and FSMonitor to achieve similar performance without needing custom kernel-level file system drivers.
To use Scalar, you register a repository using scalar register or clone a new one:
scalar clone <repository-url>Scalar configures all the advanced Git settings automatically. It sets up background maintenance, enables the file system monitor, configures partial clone filters, and sets up cone-mode sparse checkouts. It provides a standardized way to manage giant repositories without requiring developers to manually configure a dozen different Git variables.
Server-side infrastructure must also be configured to support these techniques. By default, some Git servers disable partial clone filtering to protect CPU resources. When a client requests a partial clone, the server has to dynamically filter the object packfile during the transfer. This is CPU-intensive.
To support partial clones on a self-hosted Git server, you must enable filtering in the Git configuration:
[uploadpack]
allowFilter = true
allowAnySHA1InWant = trueThe allowAnySHA1InWant setting is necessary because when a client requests a missing dependency on demand, it asks for the object by its SHA-1 hash directly. Without this setting, the server only allows clients to request objects that are referenced by tips of branches or tags.
Scaling developer machines is only half the battle. Continuous integration servers must also handle these massive codebases. Running a full clone on every CI job will quickly saturate network bandwidth and disk space on your build runners.
For CI runners, you should use partial clones with a blob filter. Since build runners usually only need to compile the code and run tests on the current commit, you can run:
git clone -filter=blob:none -depth 1 <repository-url>If your build only targets a specific service, combine this with a sparse checkout. For example, a build runner testing the billing service only needs to check out the billing code and its shared dependencies. This reduces build setup times from minutes to seconds.
Avoid discarding the Git directory between runs if possible. Instead, persist the .git directory across builds using runner caches. If Git can reuse the existing packfiles, it only needs to fetch the delta commits, saving massive amounts of network traffic.
Tools like Nx, Turborepo, or Bazel work hand-in-hand with sparse checkouts. These build systems understand the dependency graph of your codebase. You can write scripts that query the build system to find which directories a developer needs based on the target they want to build. The script can then dynamically update the Git sparse checkout paths.
For example, if a developer wants to run the frontend application, the script queries the build tool for all dependencies of that application. The build tool returns a list of paths. The script then runs:
git sparse-checkout set <paths>The developer gets exactly what they need to run their task, and nothing more. This keeps the workspace clean and fast.
Here is a practical workflow to clone and configure a massive monorepo. First, clone the repository using a blob-less partial clone:
git clone -filter=blob:none -sparse <repository-url>The -sparse flag initializes the sparse checkout in cone mode automatically, populating only the files in the root directory.
Next, navigate into the directory and enable the file system monitor:
git config core.fsmonitor trueThen, set the directories you need to work on:
git sparse-checkout set apps/api libs/databaseFinally, enable background maintenance to keep the repository optimized:
git maintenance startThis setup gives you a working environment that scales, regardless of how many gigabytes of history or millions of files exist in the remote repository.
