GCS cache not uploading/downloading with GitLab runner in GKE using Workload Identity
Description
- GitLab runner configured to use GCS cache in Google Kubernetes Engine (GKE) fails to upload or download cache.
- Job logs show the error: "No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted."
- This issue occurs even when the runner cache is configured and Workload Identity binding is in place.
Environment
-
Impacted offerings:
- GitLab.com
- GitLab Dedicated
- GitLab Self-Managed
-
Impacted versions:
- GitLab Runner v17.0 and newer
Solution
-
Ensure the correct service account binding is in place:
gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com \\ --role roles/iam.workloadIdentityUser \\ --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
-
Add the necessary annotation to the Kubernetes ServiceAccount used by the GitLab Runner. Update your Helm values file or kubectl command to include:
serviceAccount: create: true serviceAccountName: <your-service-account-name> annotations: iam.gke.io/gcp-service-account: IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com
-
Verify the annotation is correctly applied to the ServiceAccount:
kubectl get sa/<service-account-name> -n <namespace> -o jsonpath='{ .metadata.annotations }'
You can also check whether Workload Identity is configured properly by querying the metadata endpoint inside the toolbox pod. The service account associated with the cluster should be returned:
$ curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email example@your-example-project.iam.gserviceaccount.com
-
Ensure the GCP service account has the necessary permissions, including the
roles/iam.serviceAccountTokenCreator
role. -
If using Helm to deploy the GitLab Runner, update your deployment with the new configuration.
-
Restart the GitLab Runner pods to apply the changes.
Cause
The issue occurs when the Kubernetes ServiceAccount used by the GitLab Runner is not correctly mapped to the GCP service account with the necessary permissions for GCS access and/or the required annotation is missing. The mapping and annotation are crucial for the Workload Identity feature to function properly in GKE.
Additional Information
- The error "No URL provided, cache will not be uploaded/downloaded" typically indicates that the runner cannot generate a signed URL for GCS access.
- Workload Identity is the recommended way to access Google Cloud services from GKE, replacing the need for static credentials.
- Proper configuration ensures that the GitLab Runner can authenticate and access the GCS bucket for caching purposes.