Troubleshooting rate limit exceeded errors on GitLab.com due to Git LFS traffic
Description
Users on GitLab.com may intermittently receive rate limit exceeded errors while performing Git operations (clone, fetch, pull, push) against repositories that use Git LFS. This can affect multiple developers simultaneously, especially when most of the traffic come from Git LFS requests, with individual users and a shared office IP generating thousands of requests per minute and exceeding GitLab.com rate limits.
Impacted offerings:
- GitLab.com (SaaS)
Impacted features:
- Git over HTTPS/SSH with Git LFS
- CI/CD pipelines using Git LFS
Typical network setup:
- Many developers behind one or a few NAT/public IPs
Symptoms
- Developers see “rate limit exceeded” or HTTP 429 errors when:
- Cloning or fetching LFS-heavy repositories
- Running CI/CD pipelines that pull LFS objects
- Multiple users are impacted at the same time, often from the same office/network.
- Some developers can still work, while others experience ongoing intermittent failures throughout the workday.
- Issue correlates with:
- Heavy Git LFS usage
- Recent network changes (e.g., new HQ, new egress IP ranges).
Cause
On GitLab.com, rate limits are instance-wide and shared across customers. Relevant limits in this scenario are:
- Authenticated API requests: ~2,000 requests per minute per user
- IP-based limit: ~2,000 requests per minute per IP
- Git LFS: subject to the authenticated web request rate limit (e.g., ~1,000 requests per minute per user)
This Rate Limit cannot be changed.
How to confirm (for Support engineers)
- Open Rate-Limit Dashboard on Kibana
- Clear any existing filter value if present
- Update the relevant field value to filter for the affected user (i.e
json.meta.root_namespace.keyword,json.username.keyword) and relevant timeframe
In the result, you should see request counts per minutes filtered by each users and User-Agent (UA) that's making the request. The number of requests for each users or IP should stay within the Rate-Limit set on GitLab.com.
Resolution / Workarounds
1. Reduce unnecessary Git LFS downloads on local machines
a. Skip automatic LFS downloads on clone
Configure GIT_LFS_SKIP_SMUDGE so that clones do not automatically download all LFS objects:
Option 1: Global configuration
git lfs install --skip-smudge
git config --global filter.lfs.smudge "git-lfs smudge --skip -- %f"
git config --global filter.lfs.process "git-lfs filter-process --skip"
Option 2: Environment variable for a single clone
export GIT_LFS_SKIP_SMUDGE=1
git clone git@gitlab.com:group/project.git
This ensures a clone only fetches Git metadata initially, avoiding a burst of LFS requests.
b. Pull only the LFS files you actually need
When a developer needs specific LFS assets:
git lfs pull --include="path/to/needed/files/*"
This avoids downloading full historical or unused binary assets.
2. Reduce Git LFS concurrency
High concurrency amplifies bursts. Reducing parallel LFS transfers should lower the number of requests.
Global setting:
git config --global lfs.concurrenttransfers 1
Per-repository:
git config lfs.concurrenttransfers 1
For heavy environments, we'd recommend experimenting with 2 and 3, but starting from 1 is safest to avoid repeated rate limiting.
3. Optimize Git LFS usage in CI/CD
a. Disable automatic LFS downloads for CI jobs by default
In the .gitlab-ci.yml, set GIT_LFS_SKIP_SMUDGE globally to minimize unnecessary LFS traffic in jobs that don’t require the assets:
default:
variables:
GIT_LFS_SKIP_SMUDGE: "1"
b. Explicitly pull only required LFS content in jobs that need it
For jobs that do need LFS objects:
lfs_assets_job:
stage: prepare
script:
- git lfs fetch
- git lfs checkout path/to/needed/files
This pattern:
- Prevents all jobs from pulling all LFS data.
- Localizes LFS traffic to specific jobs and paths.
4. Consider IP distribution (when relevant)
If many developers are behind a very small set of NAT IPs (for example, /29 ranges):
- Multiple users’ traffic is aggregated into the same per-IP rate limit.
- Moving some traffic through additional egress IPs or VPN exit nodes can help, if aligned with the customer’s security/networking policy.