500 error on GitLab homepage or project page, and "permission denied: wrong hmac signature" error in Gitaly/Praefect logs

Description

Navigating to the GitLab homepage, or to some project pages, throws a 500 error; the corresponding error in Gitaly logs (and Praefect logs, if applicable) is "permission denied: wrong hmac signature".

This is most likely to happen when initially configuring Gitaly authentication, configuring Gitaly cluster, or when rotating the Gitaly authentication token.

Impacted offerings:

GitLab Self-Managed (2,000-user reference architecture or larger)

Impacted versions:

All

Resolution

Confirm that the Gitaly authentication token / PRAEFECT_INTERNAL_TOKEN is consistent across all relevant nodes in the GitLab configuration file (default location in Omnibus deployments is /etc/gitlab/gitlab.rb)

For instances with at least one standalone Gitaly node (non-Gitaly cluster): confirm that the Gitaly authentication token is consistent between all Gitaly nodes and Gitaly clients:

# On Gitaly nodes

gitaly['configuration'] = {
   # ...
   auth: {
     # ...
     token: 'abc123secret',
   },
}

# On Gitaly clients, either:

gitlab_rails['gitaly_token'] = '<AUTH_TOKEN>'

# or:

gitlab_rails['repositories_storages'] = {
  'default'  => { 'gitaly_address' => 'tcp://gitaly1.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_1>' },
  'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_1>' },
  'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075', 'gitaly_token' => '<AUTH_TOKEN_2>' },
}

For instances using Gitaly Cluster: confirm that the PRAEFECT_INTERNAL_TOKEN is consistent across all Praefect and Gitaly nodes:

# On Praefect nodes 

praefect['configuration'] = {
   # ...
   virtual_storage: [
      {
         # ...
         name: 'default',
         node: [
            {
               storage: 'gitaly-1',
               address: 'tcp://GITALY_HOST_1:8075',
               token: 'PRAEFECT_INTERNAL_TOKEN'
            },
            {
               storage: 'gitaly-2',
               address: 'tcp://GITALY_HOST_2:8075',
               token: 'PRAEFECT_INTERNAL_TOKEN'
            },
            {
               storage: 'gitaly-3',
               address: 'tcp://GITALY_HOST_3:8075',
               token: 'PRAEFECT_INTERNAL_TOKEN'
            },
         ],
      },
   ],
}

# On Gitaly nodes

gitaly['configuration'] = {
   # ...
   auth: {
      # ...
      token: 'PRAEFECT_INTERNAL_TOKEN',
   },
}

After identifying and correcting the misconfigured token, run sudo gitlab-ctl reconfigure on that GitLab node.

Cause

Incorrect Gitaly authentication token / Praefect internal token on one or more nodes. Token was configured with a different value than other nodes, preventing proper communication between services.

Symptoms

A 500 error on the home page of your GitLab instance, or when you navigate to specific projects. Use the correlation ID beneath the 500 error to track down related logs, and look for the error messages below.
On Gitaly nodes, the log file /var/log/gitlab/gitaly/current will show this error message:

{
  "component": "gitaly.UnaryServerInterceptor",
  "correlation_id": "XXXXXXXXXXXXXXX",
  "error": "permission denied: wrong hmac signature",
  "grpc.code": "PermissionDenied",
  "grpc.meta.auth_version": "v2",
  "grpc.meta.client_name": "gitlab-sidekiq",
  "grpc.meta.method_operation": "accessor",
  "grpc.meta.method_scope": "repository",
  "grpc.meta.method_type": "unary",
  "grpc.method": "RepositoryExists",
  "grpc.request.deadline": "XXXX-XX-XXTXX:XX:XX.XXX",
  "grpc.request.fullMethod": "/gitaly.RepositoryService/RepositoryExists",
  "grpc.request.glProjectPath": "gropu/subgroup/project",
  "grpc.request.glRepository": "project-xxxx",
  "grpc.request.payload_bytes": 174,
  "grpc.request.repoPath": "@hashed/7c/bc/7cbc...bb1.git",
  "grpc.request.repoStorage": "default",
  "grpc.response.payload_bytes": 0,
  "grpc.service": "gitaly.RepositoryService",
  "grpc.start_time": "XXX-XX-XXTXX:XX:XX.XX",
  "grpc.time_ms": 0.085,
  "level": "warning",
  "msg": "finished unary call with code PermissionDenied",
  "pid": 12345,
  "remote_ip": "x.xx.xx.xx",
  "span.kind": "server",
  "system": "grpc",
  "time": "XXXX-XX-XXTXX:XX:XX.XXXZ",
  "user_id": "1",
  "username": "user"
}

On Praefect nodes, the log file /var/log/gitlab/praefect/current will show the same error message ("error":"permission denied: wrong hmac signature")