TLS handshake timeout or SSL connection timeout when using FIPS enabled Runners
Description
When using FIPS enabled Runners, connections to the GitLab server or other HTTPS endpoints may hang or fail with error messages such as TLS handshake timeout or SSL connection timeout, due to insufficient entropy.
Impacted offerings:
- GitLab.com
- GitLab Dedicated
- GitLab Self-Managed
Impacted versions:
All
Resolution
Increase the entropy available to the system by installing rng-tools (for hardware generated entropy) or any tools that can provide software generated entropy in the host machine (not in the container).
For Debian based distributions, run apt install rng-tools. For RedHat flavours, run yum install rng-tools to install the package.
Cause
A common cause of TLS handshake failures in FIPS-enabled helper containers is OpenSSL getting stuck on the getrandom() system call. This typically happens when there is insufficient entropy available in the container, which prevents OpenSSL from completing the cryptographic operations required during the handshake.
By default, OpenSSL relies on the underlying operating system for generating randomness (check Seeding source):
bash-5.1# openssl version -a OpenSSL 3.2.2 4 Jun 2024 (Library: OpenSSL 3.2.2 4 Jun 2024) built on: Wed Jan 29 00:00:00 2025 UTC platform: linux-x86_64 options: bn(64,64) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DREDHAT_FIPS_VERSION="\"3.2.2-622cc79c634cbbef\"" -D_GNU_SOURCE -DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config" OPENSSLDIR: "/etc/pki/tls" ENGINESDIR: "/usr/lib64/engines-3" MODULESDIR: "/usr/lib64/ossl-modules" Seeding source: os-specific CPUINFO: OPENSSL_ia32cap=0xfefa32035f8bffff:0x1c2ffb
Symptom
Connections to the GitLab server or other HTTPS endpoints may hang or fail with error messages such as TLS handshake timeout or SSL connection timeout.
curl requests from the helper container may also hang or fail for the same URL.
Root Cause
FIPS 140-2 mandates a minimum of 256 bits of entropy. If the available entropy is equal to less than this, then the getrandom() call can get stuck.
To check the available entropy, run the command:
# Check entropy of host cat /proc/sys/kernel/random/entropy_avail # Check entropy of the container # Remember to change the image version docker run -it --rm registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-x86_64-v17.10.0 cat /proc/sys/kernel/random/entropy_avail
If the commands return a number less than or equal to 256 , apply the suggested resolution.
If there is sufficient entropy available and the error still occurs the cause may be different. To determine what system call is occurring when the request hangs or fails, run strace on a curl command targeting the same URL.
Instructions for building the helper image with strace and curl is available here: Install extra tools in fips enabled Runner helper.