Containers in the Nightly System¶

Introduction¶

The ATLAS Nightly System offers abundant opportunities for updating existing software and developing new tools within a multi-platform environment. Many build jobs run on native EL9 operating systems, while container technology is also supported and utilized in specific nightly branches.

Legacy builds are conducted within CentOS7 containers
The dedicated nightly branch running within an EL9 container to verify that container-related functionalities are operating correctly
Several nightly branches build release images
On-demand container build functionality is available for developers who wish to use selected nightly releases in containers

Containerized Nightly Jobs¶

As of September 2024, the Nightly System supports 12 legacy nightly branches running builds against 21.0, 21.6, 22-mc20, 23.0 git branches. These branches are aggregated under the group LEGACY in the nightlies global view
Legacy builds are executed as non-interactive payloads within the containerized environment provided by the ATLAS Local Root Base (ALRB) user interface. The default ATLAS CentOS7 Apptainer container is used
The main--containerized_Athena_x86_64-el9-gcc13-opt job builds Athena against the main git branch. Its primary purpose is to validate the default EL9 container and related functionalities. This build is scheduled to run at least once a week

Nightly Jobs with Image Builds¶

Eight weekly testing image builds are run to ensure container-related functionalities are operating correctly (as of September 2024). Currently, images are created for four projects: EL9 images - for builds against the main Git branch and CentOS7 images - for builds against 21.0, 21.2, 23.0 Git branches. These branches are aggregated under the group CONTAINERS in the nightlies global view.
Images are built in Docker format using podman
The on-demand EL9 image building Jenkins job and CentOS7 image building Jenkins job are available for building images for releases of the main, production, and legacy (21.0, 21.2, 23.0) branches. It can be triggered using a process similar to that for conventional on-demand release builds.
- Job configurations require defining a few parameters at the start, with helpful hints provided
- The primary purpose of this job is to create an image. Consequently, the traditional direct installation on CVMFS is disabled by default. However, it can be enabled by modifying a specific job parameter. In this case, the requestor will receive both the image stored in the Harbor registry and the traditional release on CVMFS. The CVMFS installation area is :
```
/cvmfs/atlas-nightlies.cern.ch/repo/sw/ondemand_<project name>_<platform name>
```
Jobs utilize the Nightly Dockerfile collection. These files are similar to the files used for standalone container builds
Images undergo a few basic sanity tests, with the test scripts available from GitLab
The image build and test logfile can be accessed on the BigPanDA nightly release results view in the ImageCreator package build log. For example, on the main--testContainers_AnalysisBase_x86_64-el9-gcc13-opt summary page click in the corresponding cell in the "Compilation errors" column, then search for the ImageCreator package

Tip

On-demand containers are built on a full OS image base and include external software stacks. They do not include data files or databases, which can be accessed via bind-mounted AFS, EOS, or CVMFS systems. Developers are encouraged to request any necessary additions to on-demand containers by opening ATLAS Infrastructure JIRA tickets under the 'Containers' component.

Storage of Nightly Images¶

Nightly images are stored in the CERN Harbor Registry within dedicated repositories for the four supported projects. The retention period for these images is 7 days

Note

Containers from the Harbor registry are not recommended for PanDA jobs on the Grid due to the significant time it takes for frequent pulls for every worker node, leading to inefficient resources utilization. For PanDA jobs, it is strongly advised to use direct installations from CVMFS. Ensure that the CVMFS option is enabled when starting the on-demand container Jenkins build prompting installation on /cvmfs/atlas-nightlies.cern.ch/.

Use of Nightly Images¶

ATLAS Software documentation portal provides instructions on setting up and running Athena containers. The example command for starting a nightly container with setupATLAS:

setupATLAS -c docker://registry.cern.ch/atlas-nightlies-analysis/athanalysis:<image name>

The image name format is

<stable release number>_<timestamp>_<platform>

Where

Stable release name refers to the stable release candidate for which the nightly release was built
Timestamp is the nightly release name (which follows a timestamp format)
Platform name follows the standard ATLAS platform name format, such as x86_64-el9-gcc13-opt

It is possble to use getarch keyword, which will be replaced with the correct architecture name.

Examples:

25.2.27_2024-09-11T0130_x86_64-el9-gcc13-opt
25.2.27_2024-09-11T0130_getarch-el9-gcc13-opt

Note

Currently images are provided only for x86_64. There are plans to provide images for ARM in the future.