Description

We have recently upgraded our GitLab instance to version 17.7, which included an upgrade of the GitLab Linux package to OpenSSL 3. After this upgrade, we are experiencing failures when uploading artifacts, LFS objects, attachments, etc to our NetApp StorageGRID object storage:

unexpected eof while reading (OpenSSL::SSL::SSLError)

 

exception.class": "Excon::Error::Socket",
  "exception.message": "2:6: FATAL: XML declaration allowed only at the start of the document (Nokogiri::XML::SyntaxError)",
  "exception.backtrace": [
    "nokogiri-1.16.8-x86_64 (linux) lib/nokogiri/xml/sax/push_parser.rb:48:in `native_write'",
    "nokogiri-1.16.8-x86_64 (linux) lib/nokogiri/xml/sax/push_parser.rb:48:in `write'",
    "fog-xml (0.1.3) lib/fog/xml/sax_parser_connection.rb:31:in `block in request'",
    "excon (0.99.0) lib/excon/response.rb:180:in `parse'",
    "excon (0.99.0) lib/excon/middlewares/response_parser.rb:7:in `response_call'",
    "excon (0.99.0) lib/excon/connection.rb:459:in `response'",

Environment

Impacted offerings:

  • GitLab Self-Managed 

Impacted versions:

  • 17.7.0 and later with NetApp used as an object storage

Solution

The solution involves 2 steps:

  • Upgrade your GitLab instance to version 17.7.7, 17.8.4, 17.9.1 or later.
  • Configure an environment variable in your /etc/gitlab/gitlab.rb file and reconfigure GitLab to apply changes:
    gitlab_rails['env'] = { 'SSL_IGNORE_UNEXPECTED_EOF' => '1' }

Cause

This error occurs because OpenSSL 3 requires the server to send a close_notify signal before shutting down the connection. See this merge request for details.

Related Links

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/181759

On this page...