Using EL9 Farm Nodes
This page is a preview of the above content, for the full web page follow the link above.
Red Hat Enterprise Linux 7 and its derivatives ("EL7"), which includes the Farm's CentOS Linux 7 distribution, reach end-of-life in June 2024. Migration of Farm nodes to AlmaLinux 9 ("EL9") is underway. Unlike our regular monthly minor y- and z-stream updates, this is not only a "major" version update, but a skip, from 7 to 9, including changes to system software beyond the scope of this article. Here, we'd like to outline some JLab-specific differences with the previous configuration of our CentOS Linux 7 nodes.
Access
EL9 nodes are available to all Farm users. The temporary interactive node can be accessed at ifarm9.jlab.org and hardware for two new interactive nodes has been ordered.
- Getting Started, usage of 2factor Login Nodes
- Computing Coordinators, access approvals
The batch system is not separate: EL9 nodes are part of the same batch system as the EL7 nodes, and jobs can be submitted for EL9 compute from the EL7 ifarms (1801, 1802, and 1901) and vice versa, but doing so may require more attention to how Slurm propagates environment from submit hosts.
Filesystem Changes
To better isolate dependencies on CUE filesystems and reduce outages due to maintenance of those systems, we have made a few impactful changes.
/apps removed
The /apps area has been replaced. To provide better portability of compute jobs, we have moved to CVMFS. See the related article on modulefile usage.
/site removed
The /site area housed several SciComp utilities. Those tools (e.g. CacheManager, Jasmine commands, swif2) have been moved to local installs on the EL9 interactive node. They are not available on compute nodes.
/tmp noexec
This is consistent with other recent SciComp service node deployments. Please use local scratch (/scratch/$USER interactively and /scratch/slurm/$SLURM_JOB_ID in jobs), which is generally faster and better isolated from system processes. To help with this for things like container builds on ifarm, we set $TMPDIR accordingly (to /scratch/$USER) at the system-level, for all users.
/farm new cluster-based top-level
- /farm/home (future SciComp homedirs)
- /farm/logs (equivalent to /farm_out)
- /farm/lustre
- /farm/scigroup
- ...
Compiling/Building Software
Continue to use the interactive node (ifarm9.jlab.org) for building your software.
- Provided modules/modulefiles
- Python 3.9 is the Alma9 default
If you find you need other modules/versions to be supplied, please submit a ServiceNow incident so it can be addressed.
Running Jobs
Usage of EL7 nodes is currently the default. Using Slurm, we provide a way to opt-in to EL9 production nodes. At a later date, you will have to specifically request EL7 and the default will be moved to EL9.