How to set up a centrally managed pipeline secret detection configuration applied via Scan Execution Policy

Description

This guide shows how to set up a ruleset configuration file in a private remote repository. With this file you can modify, replace, or extend the default ruleset for the analyser. This guide focuses on remotely extending the default ruleset with extra custom rules via git passthrough, then applying the secret scanning job via a Scan Execution Policy to enforce the job with extended configuration.

Storing the secret detection scanner configuration in a centralized project enables security teams to extend the ruleset without updating each individual project's configuration. This approach allows more effective management of changes at scale, making it easy to adjust the scanner ruleset universally. After implementation, ruleset configuration changes can be made in one place and applied to all targeted projects.

Environment

  • Impacted offerings:
    • GitLab.com
    • GitLab Dedicated
    • GitLab Self-Managed
  • Impacted versions:
    • GitLab 17.2 and later

Prerequisites

  • Access to create and manage security policies
  • Owner access to create projects and implement security scanning tools

Solution

A high level overview

To implement this solution, you need a central configuration project, target projects, and a policy to apply the secret detection jobs. The secret scanner requires several key steps to utilize an extended ruleset in this approach:

  1. The scanner is enforced to run with a scan execution policy specifying SECRET_DETECTION_RULESET_GIT_REFERENCE to point to a remote custom ruleset.
  2. During job execution, the scanner clones the project specified in SECRET_DETECTION_RULESET_GIT_REFERENCE to access the ruleset configuration.
  3. The job then reads the ruleset configuration from the clone, and attempts a second clone to access the extended gitleaks configuration file using a git passthrough.
  4. With the extended ruleset configuration file now accessible, the scanner analyzes the target project using the extended ruleset.

You'll need two files:

  • A scanner ruleset file called secret-detection-ruleset.toml that instructs the project where to look for the extended config
  • An extended config file called gitleaks.toml that holds the extended Gitleaks rules

And two group level variables:

  • $CONFIG_URL that tells the scanner where to look for the extended Gitleaks config file, with appropriate credentials
  • $SECRETS_CONFIG_TOKEN that holds the gl-pat group token credentials to access the config project repository

Part One - Create the ruleset file

  1. Create a private blank project to be the central store for the configuration files. For this guide, it will be called secret-detection-config, and created in a top level group called my-org.
  2. In this project, create a .gitlab directory
  3. Add a file to the new directory called secret-detection-ruleset.toml
  4. Populate the file with appropriate ruleset configuration to tell the scanner where to look for config:
[secrets]
  description = "My private remote ruleset"
  interpolate = true

  [[secrets.passthrough]]
    type   = "git"
    ref    = "main"
    value  = "$CONFIG_URL"    # we will create a group ci variable later that this will reference
    subdir = "config"   # subdir is optional, so you may also remove this and add the configuration file at the root of the remote repository referenced in $CONFIG_URL

Note: The interpolate above should be used with caution, as described here.

Part Two - Create the extended config file

  1. Create a config directory in secret-detection-config project
  2. In my-org/secret-detection-config/config create a file called gitleaks.toml
  3. Populate this file with your extended configuration. In this case, add 2 extra custom rules for the scanner:
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[[rules]]
  id = "example_api_key"
  description = "Example Password Leak"
  regex = '''Password123'''

[[rules]]
  id = "example_api_secret"
  description = "Example Service API Secret"
  regex = '''example_api_secret'''

Note: New custom rules must follow the custom rule format.

Part Three - Create a Group Access Token and the Required CI Variables

The target projects will need to access the project with the stored configuration files in their CI pipelines. For this we'll use a Group Access Token with appropriate privileges. The group access token must have the read_repository scope and at least the Reporter role.

  1. Navigate to group my-org
  2. Select Settings > Access Tokens > Add new token
  3. Create Group Access Token Secret-Detection-Config-Token with read_repository scope and reporter role
  4. Note down the token value, and the bot user associated with the token (See bot users for groups to learn how to find the username associated with a group access token.)
  5. Save the gl-pat token as a masked group variable called SECRETS_CONFIG_TOKEN in the same group Settings > CICD > Variables > Add variable
  6. Create a second masked group variable called CONFIG_URL - this will be used when the scanner reads the secret-detection-ruleset.toml and uses the value to clone for the gitleaks.toml. The value should be <token-bot-user>:<password>@<url_to_config_project> for example:
group_303_bot_abd353dcb476dc31ff87fb883e8051fe:glpat-12345678abcdefg@gitlab.example.net/my-org/secret-detection-config

Note: Make sure the URL is pointing to the right GitLab host, depending on if you are using Self-Managed, or GitLab.com

Part Four - Create the Scan Execution Policy

  1. Locate or create your target project (in this guide it will be my-org/target-project) and navigate to its parent group (my-org), where we will create a group level Scan Execution Policy: Secure > Policies > New Policy
  2. Create the Scan Execution Policy
  3. Fill out the required fields and scope the policy to the projects in the group
  4. Select Add new CI variables, adding in the SECRET_DETECTION_RULESET_GIT_REFERENCE variable to specify where the ruleset file is located. The value needs to be in the format <token-bot-user>:<password>@<url>:
SECRET_DETECTION_RULESET_GIT_REFERENCE: 'group_303_bot_abd353dcb476dc31ff87fb883e8051fe:$SECRETS_CONFIG_TOKEN@gitlab.example.net/my-org/secret-detection-config'

The resulting policy should look something like:

name: Customised Secrets Detection
description: Applies secret-detection jobs with extended rulesets to all target projects.
enabled: true
policy_scope:
  projects:
    excluding: []
rules:
  - type: pipeline
    branches:
      - '*'
actions:
  - scan: secret_detection
    variables:
      SECRET_DETECTION_RULESET_GIT_REFERENCE: >-
        group_303_bot_abd353dcb476dc31ff87fb883e8051fe:$SECRETS_CONFIG_TOKEN@gitlab.example.net/my-org/secret-detection-config
type: scan_execution_policy
skip_ci:
  allowed: false

Verification

With the above configuration in place, you can verify the scanner is working with extended ruleset using the following steps:

  1. Navigate to a project where the policy has been applied
  2. Ensure there are pipelines configured and running for the project when a commit is made
  3. Create a new file in the project with the following content, and observe the scanner detect the custom rule along with the standard rules:
#whoopsie.txt

Password123

glpat-123456789abcdefghijk

Observe the scanner operate with the extended ruleset, securely obtained from a centrally managed project. It will detect Password123 from the custom rule we have defined earlier, and the gl-pat token per the scanner's default ruleset.

Related links

On this page...