GitLab.com hosted macOS runners intermittently stuck preparing the instance

Description

Jobs using macOS runners occasionally hang during the Preparing the "instance" executor.
saas-macos-large-m2pro runners will log multiple retries to create the instance environment:

Enforcing VM Isolation
Creating nesting VM tunnel
Creating nesting VM macos-15-xcode-16
Dialing nesting daemon
ERROR: Preparation failed: creating instance environment: creating vm tunnel: creating nesting vm: rpc error: code = Unknown desc = cloning vm: reading vm config: open /Volumes/VMData/images/macos-15-xcode-16/config.json: no such file or directory
Will be retried in 3s ...

saas-macos-medium-m1 runners will hang until the job timeout is reached

Enforcing VM Isolation
Creating nesting VM tunnel
Creating nesting VM macos-14-xcode-15
Dialing nesting daemon
ERROR: Job failed: execution took longer than 1h0m0s seconds

Environment

GitLab hosted macOS runners

Impacted offerings:

GitLab.com

Solution

Open a ticket with GitLab Support, who will be able to work with the internal teams to fix the impacted hosts.

Please provide a few example job logs when creating the ticket. Full job logs will have the name of the impacted hosts.

Workaround

Since this issue is intermittent, retrying the job should eventually allow the job to complete.

If an expected job runtime is known, you can add a timeout and retry to the affected job for a quicker / automated failure and retry cycle.

  timeout: 10m
  retry:
      max: 3
      when:
        - job_execution_timeout

Cause

The cause and a long term solution are still under investigation. The macOS hosts seem to degrade over time. As the host begins to age, more picked up jobs may be stuck.

Additional Information

Hosted macOS runners are currently a Beta feature.