Certificate or other TLS errors after upgrade when using proxy

Description

  • Certificate errors occur after upgrade to 17.10 or higher.
  • Duo code completions fail with errors like: 
    FetchError: request to https://cloud.gitlab.com/ai/v2/completions failed, reason: self signed certificate in certificate chain
  • Any Workhorse related connections can be affected.
  • Proxy may indicate a handshake timeout.
  • This can also affect container registry connections behind a proxy
  • Timeouts and failures in TLS handshake

Environment

This affects environments with a proxy configured.

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

  • 17.9 and later

Solution

Recommended:

  1. Please upgrade proxy to a version supporting post-quantum-secure cryptography
  2. If using Amazon Network Firewall include “ssl_state:client_hello” in the drop tls rule entry has been reported to resolve this problem. (please contact AWS support for more information)

Alternative:

  1. Add the following block into your gitlab.rb of servers running Workhorse

    gitlab_workhorse['env'] = {
  'GODEBUG' => "tlskyber=0"
}

  2. Add the following block to your gitlab.rb of servers running the container registry

    registry['env'] = {
    'GODEBUG' => "tlskyber=0"
}

  3. Reconfigure gitlab-ctl reconfigure

    global:
  extraEnv:
    GODEBUG: "tlskyber=0"

  4. Redeploy helm with new values. (helm upgrade)

Cause

In Go 1.23, post-quantum key exchange mechanism is enabled by default. The size of this can cause some proxies to timeout during handshake.

Additional Information

  • This may present differently depending on the proxy in use.
  • X25519MLKEM768 key exchange is unlikely to be in use so it is safe to disable at this time.

Related Links

On this page...