Encountering cannot stat Git binary after upgrading GitLab

Issue

After GitLab upgrade:

  • Users encounter a 500 error trying to access GitLab pages which reference repository data. An error message similar to the following is found in production_json.log:

    • getting Git version: cannot stat Git binary: stat /var/opt/gitlab/gitaly/run/gitaly-32441/git-exec-3486633472.d/git: no such file or directory.

  • The GitLab Admin area Gitaly servers page (/admin/gitaly_servers) may show server versions that do not match the post-upgrade GitLab version.

Environment

  • Impacted offerings:

    • GitLab Self-Managed

  • Impacted versions:

    • All versions

Cause

Gitaly failed to restart during the upgrade process, resulting in stale symbolic links.

The newer version of GitLab contains a new Git binary. Gitaly recreates symlinks to the new binary on start up. Since Gitaly failed to restart, the symlinks are not recreated and no longer point to the current Git binary.

Resolution

Restart all instances of the Gitaly service in your GitLab environment:

  • gitlab-ctl restart gitaly

Additional information

  • You can confirm if Gitaly is out of date from the UI in Admin Area > Gitaly servers. The Server version should match the upgraded version of GitLab.
  • Gitaly may also show a longer run time than other services after running:
    • sudo gitlab-ctl status

Related links

On this page...