500 error when accessing project settings or container registry: "Neither PUB key nor PRIV key"

Description

  • Users encounter HTTP 500 errors when viewing project settings or container registry pages
  • Container registry CLI operations fail to authenticate
  • Error logs show: "exception.message":"Neither PUB key nor PRIV key" originating from OpenSSL
  • The stack trace indicates an RSA key error: 
    "exception.class":"ActionView::Template::Error" or "exception.class: "OpenSSL::PKey::RSAError"

"exception.message":"Neither PUB key nor PRIV key"
"exception.backtrace":["openssl (3.2.0) lib/openssl/pkey.rb:356:in `initialize'"
"openssl (3.2.0) lib/openssl/pkey.rb:356:in `new'"
"openssl (3.2.0) lib/openssl/pkey.rb:356:in `new'"
"lib/json_web_token/rsa_token.rb:34:in `key'"
"lib/json_web_token/rsa_token.rb:15:in `encoded'"
"app/services/auth/container_registry_authentication_service.rb:97:in `access_token'"
"lib/container_registry/base_client.rb:51:in `token_from'"
"lib/container_registry/base_client.rb:45:in `with_dummy_client'"]

Environment

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

  • All supported versions

Solution

If you have a multi-node deployment, select one of your Rails nodes to complete these steps on.

  1. Verify if the registry key is valid by running the following command:

    /opt/gitlab/embedded/bin/openssl rsa -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key -check

    Successful output will look like:

    RSA key ok
writing RSA key

    Error outputs that indicate a problem include:

    Could not find private key from /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key

    or

    Could not read private key from /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key
  2. Inspect the registry key file manually to confirm its format and validity: 
    cat /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key
  3. Ensure the key has the correct format with proper header and footer: 
    -----BEGIN RSA PRIVATE KEY-----
[key content]
-----END RSA PRIVATE KEY-----
  4. If the key has an incorrect format (such as -----ENDRSA PRIVATE KEY----- instead of the correct -----END RSA PRIVATE KEY-----), edit the file to fix the format.
  5. After correcting the key file, confirm if the changes are valid: 
    /opt/gitlab/embedded/bin/openssl rsa -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key -check
  6. When a reconfigure is executed, the /etc/gitlab/gitlab-secrets.json file is used to generate the registry key so this file will need to be corrected as well. Examine the registry.internal_key value and ensure it matches the changes you made to the registry key file.
  7. Reconfigure GitLab to apply the changes: 
    gitlab-ctl reconfigure
  8. Confirm if the registry key is still valid after reconfiguring: 
    /opt/gitlab/embedded/bin/openssl rsa -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key -check
  9. Restart the registry service: 
    gitlab-ctl restart registry
  10. For multi-node deployments:
    1. Ensure that /etc/gitlab/gitlab-secrets.json file is the same across all nodes.
    2. On all Rails nodes running the registry service:
      1. Rebuild the registry key - gitlab-ctl reconfigure
      2. Restart the registry - gitlab-ctl restart registry

Cause

This error typically occurs when openssl cannot properly parse the RSA private key /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key used for container registry authentication. The most common cause is an incorrectly formatted key file, particularly with invalid header/footer formatting or corrupted key data.

The issue frequently happens when GitLab administrators are using automated tools to generate or manage their gitlab-secrets.json file. These automation scripts may inadvertently introduce formatting errors in the RSA keys, especially in the header/footer sections.

Related Links

Troubleshooting the GitLab container registry

On this page...