GitLab Pull Mirror SSH Authentication fails after migrating to Rocky Linux 9 or RHEL 9

Description

After migrating GitLab from CentOS to Rocky Linux 9 or RHEL 9, pull mirrors may fail with SSH authentication errors. This is primarily due to stricter system-wide cryptographic policies on the new operating system, but critically, it can also be compounded by the remote Git service's SSH server not supporting modern cryptographic algorithms.

You will see this error in the GitLab UI when GItLab attempts to update a repository from your pull mirror:

13:fetch remote: "Warning: Permanently added '[mirror.host]:443' (RSA) to the list of known hosts.\r\nsign_and_send_pubkey: no mutual signature supported\r\*****@mirror.host Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n": exit status 128.

Environment

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

All GitLab versions when migrating from older OS (CentOS 7/8) to Rocky Linux 9 or RHEL 9

Solution

This solution involves adjusting the GitLab server's cryptographic policy to enable support for algorithms that might be the only common ground between your Rocky Linux 9/RHEL 9 client and the remote SSH server.

  1. Check if the cryptographic policy is set to DEFAULT. If it is, then you will need to proceed with the rest of these steps.
    ❯ update-crypto-policies --show
DEFAULT
  2. Enable the SHA1 cryptographic policy:
    ❯ update-crypto-policies --set DEFAULT:SHA1
  3. Reboot the system to apply the new cryptographic policy
  4. Test the pull mirror and wait for the 5-minute timeout period, then manually force a mirror update.
  5. Alternative approach (if the issue persists after rebooting):
    1. Delete and recreate pull mirror configurations to regenerate SSH keys compatible with current policies
    2. Add the new public keys to the external Git service (Bitbucket, external GitLab etc).

Cause

Rocky Linux 9 and RHEL 9 implement stricter system-wide cryptographic policies that, by default, disable older or weaker cryptographic algorithms, including SHA1 signatures for RSA keys.

When GitLab is migrated from an older OS eg. CentOS 7/8, SSH authentication for pull mirrors can fail due to an algorithm mismatch between the GitLab client and the remote Git service's SSH server. The no mutual signature supported error signifies that the client and server cannot agree on a common, mutually acceptable cryptographic method for signature verification during authentication.

This conflict arises because:

  • The Rocky Linux 9/RHEL 9 client, under its default strict policy, will not offer or accept older algorithms like SHA1.
  • The remote Git service's SSH server, especially if it's an older system, might only offer or predominantly use these older algorithms eg. ssh-rsa with SHA1

Consequently, the client and server cannot find a compatible algorithm to proceed with authentication. Enabling the DEFAULT:SHA1 policy on the GitLab server broadens the range of algorithms it can use, making it possible to find a common signature method with SSH servers that rely on SHA1.

Related Links

On this page...