Skip to content

Git Tag Synchronization across "Sibling" Nightly Jobs

Introduction

There are a few dozen nightly branches built from the Git development and production branches (currently main and 24.0). Scheduling all jobs to start at the standard time of 21:00 is not feasible due to the following reasons:

  • Jobs overlap is not allowed on any node
  • Staggering job start times overnight improves resource efficiency
  • Only four slow ARM nodes are available, causing some ARM jobs being scheduled much later in the morning

Motivation for Tag Synchronization

When multiple jobs targeting the same branch run at slightly different times, it is beneficial to ensure they all use the exact same codebase. Synchronizing git tags across these jobs guarantees consistency in the nightly builds.

Solution

  • Use nightly git tags to synchronize code versions across job families for the main and 24.0 branches. These tags are created by the tag_build.sh script (developed by Frank Winklmeier) during most nightly jobs
  • A short Jenkins job generates default sync git tags daily at 21:00 for the 24.0 and main branches:
    • nightly/24.0/2025--T21:00
    • nightly/main/2025--T21:00
  • Nightly jobs for the 24.0 and main git branches scheduled within 24 hours of sync tag creation run git checkout using these synchronized tags
    • Nightly release name (timestamp) reflects the time of the synchronizing git tag
  • Exclude legacy nightly branches that do not belong to large branch families

Current Status

  • Feature currently is deployed for 24.0 nightly branches only

Restarting Nightly Jobs

To manually restart a nightly job:

  • Navigate to the Jenkins web UI
  • Click Build with Parameters
  • Set the parameter USE_DEFAULT_2100_NIGHTLY_TAG to NO, CREATE NEW NIGHTLY TAG (from choice menu)

By default, this parameter is set to YES, instructing the job to use the Git tag created at 21:00. Setting it to NO, CREATE NEW NIGHTLY TAG will direct the job to generate a new git tag using the current timestamp as the release name.

Note

Restarting a job without enabling this option will fail if a release with the same name (i.e., timestamp) already exists.

Error Handling

Problem Handling
git clone fails job stops
default sync tag is not found in git Job continues with current timestamp; a corresponding Git tag is created
git checkout fails job stops
release already exists in the db job stops

Further Information

For implementaion details and discussion history refer to the JIRA ticket ATLINFR-5724