Container registry tags disappear after upgrading to 18.0.1 when using s3_v2 storage driver

 

Description

Customers upgrading to 18.0.1 that are already using s3_v2 storage driver may observe container registry image tags disappear post update. They might see some of these errors in their logs.

storage driver health check: s3_v2: operation error S3: HeadObject, serialization failed: serialization failed: input member Key must not be empty

 

Environment

  • Impacted offerings:

    • GitLab Self-Managed

  • Impacted versions:

    • GitLab 18.0.1

Workaround

  1. Update the existing registry configuration and change the storage driver from s3_v2 to s3. See example below.
# current
registry['storage'] = {
  's3_v2' => {
    'accesskey' => '<s3-access-key>',
    'secretkey' => '<s3-secret-key-for-access-key>',
    'bucket' => '<your-s3-bucket>',
    'region' => '<your-s3-region>',
    'regionendpoint' => '<your-s3-regionendpoint>'
  }
}
# updated
registry['storage'] = {
  's3' => {
    'accesskey' => '<s3-access-key>',
    'secretkey' => '<s3-secret-key-for-access-key>',
    'bucket' => '<your-s3-bucket>',
    'region' => '<your-s3-region>',
    'regionendpoint' => '<your-s3-regionendpoint>'
  }
}
  1. Reconfigure the instance by running gitlab-ctl reconfigure.

Cause

The issue appears to be a regression in the s3_v2 driver implementation in GitLab 18.0.1. Multiple users have reported the same issue in the feedback thread: #525855

Related Links

On this page...