Load spike on PostgreSQL after upgrading GitLab
Description
After upgrading GitLab to a new version, the load (CPU / Memory / IO utilization) on an external PostgreSQL server has increased. Even if the background migrations are complete or have been paused, the related SQL queries may still be running and consuming resources.
Impacted offerings
- GitLab Self-Managed
Solution
Set a statement_timeout of 60000 milliseconds or lower, per the database requirements. A PostgreSQL restart is required after the statement_timeout has been changed to:
- Apply the new configuration
- Clean-up existing long-running queries
Cause
Some background migration steps will run indefinitely, as they're expected to time-out on large GitLab instances. They will then be retried in a batched fashion per the job retry mechanism, as opposed to migrate entire tables at once.