Error 403 when starting pipelines (with parallel jobs) on a runner fleet

Description

Users may receive application-wide 403 errors when starting GitLab pipelines that launch numerous parallel jobs across a large runner fleet with multiple IP addresses. This prevents the pipeline from executing successfully, can block CI/CD workflows and lead to degraded user experience.

Impacted offerings:

GitLab Dedicated
GitLab Self-Managed

Impacted versions:

GitLab 16.1 and later

Symptoms

Users receive 403 HTTP status responses when initiating pipelines with many parallel jobs
Errors occur specifically when jobs are distributed across runners with different IP addresses
Workhorse and Puma logs (production_json.log) show status: 403 entries without detailed error messages
In this scenario, the most notable increase in 403 responses will be from the Repositories::GitHttpController
The issue affects the entire application experience for the user, not just the specific pipeline

Solution

Option 1: Disable IP Address Limiting (Recommended for dedicated runner environments)

Navigate to Admin > Settings > Reporting > Spam and Anti-bot Protection
Uncheck "Limit sign in from multiple IP addresses"
Save the configuration

Option 2: Increase IP Address Limit (Recommended for mixed environments)

Navigate to Admin > Settings > Reporting > Spam and Anti-bot Protection
Keep "Limit sign in from multiple IP addresses" enabled
Increase the "IP addresses per user" setting to accommodate your runner fleet size
Save the configuration

Note: Consider your security requirements when choosing between these options. Some users might prefer option 2 to maintain some protection while allowing legitimate pipeline operations.

Cause

This issue is caused by GitLab's "Limit sign in from multiple IP addresses" security feature. When a pipeline distributes jobs across runners with different IP addresses, GitLab interprets the requests from these various IPs as potential suspicious activity and blocks the user account to prevent unauthorized access for the set duration (60 seconds per default).

Additional Information

The 403 errors may appear intermittently depending on job scheduling and runner availability.