GCS cache not uploading/downloading with GitLab runner in GKE using Workload Identity

Description

  • GitLab runner configured to use GCS cache in Google Kubernetes Engine (GKE) fails to upload or download cache.
  • Job logs show the error: "No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted."
  • This issue occurs even when the runner cache is configured and Workload Identity binding is in place.

Environment

  • Impacted offerings:

    • GitLab.com
    • GitLab Dedicated
    • GitLab Self-Managed

  • Impacted versions:

    • GitLab Runner v17.0 and newer

Solution

  1. Verify that Workload Identity is enabled on the cluster

    # Check if Workload Identity is enabled on the cluster.
# This should return the workloadPool, e.g. 'PROJECT_ID.svc.id.goog' 
gcloud container clusters describe $CLUSTER_NAME \
 --project=PROJECT_ID \
 --region=GCP_ZONE \
 --format="value(workloadIdentityConfig.workloadPool)"

  2. Ensure the correct service account role bindings are in place:

    gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com \
 --role roles/iam.workloadIdentityUser \
 --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

# Add token creator role
gcloud projects add-iam-policy-binding PROJECT_ID \
 --role="roles/iam.serviceAccountTokenCreator" \
 --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

  3. Add the necessary annotation to the Kubernetes ServiceAccount used by the GitLab Runner. Update your Helm values file or kubectl command to include:

    serviceAccount:
  create: true
  serviceAccountName: <your-service-account-name>
  annotations:
    iam.gke.io/gcp-service-account: IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com

    or to manually add the annotation after service account creation:

    kubectl annotate serviceaccount <your-service-account-name> \
 --namespace <runner namespace> \
 iam.gke.io/gcp-service-account=IAM_SA_NAME@PROJECT_ID.iam.gserviceaccount.com

  4. Verify the annotation is correctly applied to the ServiceAccount:

    kubectl get sa/<service-account-name> -n <namespace> -o jsonpath='{ .metadata.annotations }'

    You can also check whether Workload Identity is configured properly by querying the metadata endpoint from within a pod, from a CI/CD job for example. The service account associated with the cluster should be returned:

    script:
  - "echo 'Testing GCS access...'"
  - "curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"

  5. If using Helm to deploy the GitLab Runner, update your deployment with the new configuration.

  6. Restart the GitLab Runner pods to apply the changes.

Cause

The issue occurs when the Kubernetes ServiceAccount used by the GitLab Runner is not correctly mapped to the GCP service account with the necessary permissions for GCS access and/or the required annotation is missing. The mapping and annotation are crucial for the Workload Identity feature to function properly in GKE.

Additional Information

  • The error "No URL provided, cache will not be uploaded/downloaded" typically indicates that the runner cannot generate a signed URL for GCS access.
  • Workload Identity is the recommended way to access Google Cloud services from GKE, replacing the need for static credentials.
  • Proper configuration ensures that the GitLab Runner can authenticate and access the GCS bucket for caching purposes.

Related Links

On this page...