Gitaly Permission Denied Errors Due to UID/GID Mismatch Between Nodes

Description

When setting up GitLab nodes on different operating systems or AMI versions, Gitaly may fail to start with "permission denied" errors. This occurs when the git user and other GitLab service users have different UID/GID values when migrating data between nodes, causing file ownership mismatches that prevent services from accessing their required files.

The error typically manifests as:

{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: permission denied","level":"error","msg":"find process","pid":71204,"time":"2025-04-10T10:57:29.488Z","wrapper":71204}

Impacted offerings:

  • GitLab Self-Managed

Impacted versions: 

All versions when using GitLab Geo across different host environments

Specific scenarios:

  • GitLab Geo deployments with primary and secondary nodes on different OS versions (e.g., Ubuntu 20.04 vs Ubuntu 22.04)
  • Cross-region Geo deployments using different AMI images
  • Disaster recovery scenarios involving volume reattachment to new hosts
  • Infrastructure provisioned with GitLab Environment Toolkit (GET)

Solution

Configure explicit numeric user and group identifiers in your GitLab configuration to ensure consistency across all nodes:

  1. Identify the UID/GID values from your primary node:
    cat /etc/passwd | grep -E "(git|gitlab-www|gitlab-prometheus|gitlab-consul)"
  2. Add the numeric identifiers to your GitLab configuration (/etc/gitlab/gitlab.rb) on all secondary nodes:
    # Specify numeric user and group identifiers
user['uid'] = 996
user['gid'] = 997
web_server['uid'] = 997
web_server['gid'] = 998
gitlab_prometheus['uid'] = 995
gitlab_prometheus['gid'] = 996
gitlab_consul['uid'] = 994
gitlab_consul['gid'] = 995
  3. Reconfigure GitLab:
    sudo gitlab-ctl reconfigure
  4. Verify the file permissions are correct:
    ls -la /var/opt/gitlab/gitaly/
  5. Restart GitLab services:
    sudo gitlab-ctl restart

Cause

This issue occurs because:

  • Different operating systems or AMI versions may assign different UID/GID values to users with the same name
  • GitLab's Chef cookbooks create users without specifying explicit numeric identifiers, allowing the system to assign available UIDs/GIDs
  • File permissions are stored using numeric UID/GID values, not usernames
  • When volumes are inherited or synced between hosts with different user mappings, files become inaccessible to the intended services

For example, if the primary node has git:x:996:997 but the secondary has gitlab-prometheus:x:996:997 and git:x:997:998, files owned by UID 996 will be accessible by gitlab-prometheus instead of git on the secondary node.

Additional Information

Prevention: When planning GitLab deployments across different environments, always configure explicit UID/GID values from the beginning to avoid this issue. This affects Geo in particular.

Verification: After applying the fix, ensure critical GitLab files have the correct ownership:

# Check Gitaly files
ls -la /var/opt/gitlab/gitaly/
# Should show git:git ownership for gitaly.pid and gitaly.socket

# Check other service directories
ls -la /var/opt/gitlab/prometheus/
ls -la /var/opt/gitlab/nginx/

Note: This issue is not caused by Gitaly sync operations but rather by the underlying host configuration differences.

Related Links

On this page...