Failed to pull OCI Image from GitLab Container Registry

 

Image pull fails with: manifest unknown: OCI index found, but accept header does not support OCI indexes

Overview

Pulling an OCI Image from the GitLab container registry fails with:

manifest unknown: OCI index found, but accept header does not support OCI indexes

Description

The issue only impacts downloading OCI images from the GitLab container registry. Downloading the configuration digest does not hit this issue. The request fails and reports a 404 error.

The following script will help isolate the issue by using curl commands to check accessing the configuration digest and the manifest digest of your OCI image (Note: Check the image in question as it might only have a single digest, therefore you will need to adjust the script accordingly):

#!/bin/bash

set -e

PAT='glpat-xx Token here'
REPOSITORY_PATH="(image published to path)"
CONFIGURATION_DIGEST="sha256:1234"
MANIFEST_DIGEST="sha256:1234"
TOKEN_NAME="<token name>"

JWT_TOKEN=$(curl --user "${TOKEN_NAME}:${PAT}" "https://GITLAB_DOMAIN.com/jwt/auth?service=container_registry&scope=repository:${REPOSITORY_PATH}:pull&offline_token=true&client_id=docker" | jq -r .token)

curl -Lv -O -H "Authorization: Bearer ${JWT_TOKEN}" https://REGISTRY_DOMAIN.com/v2/${REPOSITORY_PATH}/blobs/${CONFIGURATION_DIGEST}
curl -Lv -O -H "Authorization: Bearer ${JWT_TOKEN}" https://REGISTRY_DOMAIN.com/v2/${REPOSITORY_PATH}/manifests/${MANIFEST_DIGEST}

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

  • All

Resolution

You need to add the OCI manifest V1 type Accept header for the GitLab container registry to accept the request and report a 200 response. Without the Accept header, the request returns a 404 error.

curl -v -O -H "Authorization: Bearer ${JWT_TOKEN}" -H "Accept: application/vnd.oci.image.manifest.v1+json" https://registry.sr-env-3ed07b92-omnibus.env-3ed07b92.gcp.gitlabsandbox.net/v2/${REPOSITORY_PATH}/manifests/${MANIFEST_DIGEST}

In this example, the issue was between the CDN (CloudFront) and docker client. CloudFront has expecting the correct header to be sent first, but docker sends a number of headers and the order varies each time. Adding the function to CloudFront worked around the Accept header order issue.

These troubleshooting steps will help isolate the issue accept header does not support OCI indexes from a GitLab perspective.

Cause

In this MR, GitLab container registry added support for Docker OCI Images by adding the application/vnd.oci.image.manifest.v1+json Accept header:

OCI_MANIFEST_V1_TYPE = 'application/vnd.oci.image.manifest.v1+json'

Symptom

The GitLab container registry log reports using the curl example:

ERROR: Job failed: failed to pull image "gitlab-registry.xxxxx.com/Namespace/project/platform/ci/images/<OCI Image>:v0.3" with specified policies [if-not-present]: Error response from daemon: manifest for gitlab-registry.xxxxx.com/Namespace/project/platform/ci/images/<OCI Image>:v0.3 not found: manifest unknown: OCI index found, but accept header does not support OCI indexes (manager.go:251:0s)

Docker pull request reports:

~$ docker pull registry.gitlab.xxx.com/test/oci-build/deployer:latest
Error response from daemon: manifest for registry.gitlab.xxx.com/test/oci-build/deployer:latest not found: manifest unknown: OCI manifest found, but accept header does not support OCI manifests

Root Cause

In this example, CloudFront was expecting OCI manifest V1 type accept header to be the only accept header to be passed per https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2. But Docker passes multiple accept headers and unordered. The workaround was to add the following function to CloudFront to select the correct header, independent of the order of the headers from the docker client:

async function handler(event) {
  const request = event.request;
  Object.keys(request.headers).forEach(function(key) {
      if (request.headers[key].multiValue) {
          request.headers[key].value = request.headers[key].multiValue.map(function(mv){
              return mv.value;
          }).join(",");
      }
  })
  return request;
}

 

On this page...