Release Notes
1. Basic Info
The Qlustar 13 release is based on Ubuntu 22.04. It includes all security fixes and other package updates published before Dec 28th 2022. Available security updates relevant to Qlustar 13, that have appeared after this date, will be announced on the Qlustar website and in the Qlustar security newsletter. Supported edge-platforms are Ubuntu 22.04 (Jammy) and AlmaLinux 8.
2. New features
2.1. QluMan 13
- Hardware Remote Management
-
Support has been added for configuration and operation of cluster node’s Baseboard Management Controller (BMC) via IPMI/Redfish. This initial implementation allows
-
Power control of nodes via IPMI or Redfish
-
Validation and Correction of BIOS settings
- Cumulative Problem Report
-
To ease detection of node configuration problems, a cumulative problem report can now be generated with the QluMan GUI. It lists all nodes with a description of their problem(s) and helps to systematically fix them.
2.2. The Qlustar HPC Core Stack
The HPC core stack is a new component definition within Qlustar and is related to the introduction of Spack as the package manager for HPC applications. Rather than aiming to provide as much HPC functionality as possible by OS packages (debs/rpms), the HPC core stack now makes a clear cut between Qlustar related system software which remains provided as OS packages and HPC application software to be provided by Spack. In practice, the cut is made at the MPI level: All MPIs are to be provided by Spack, whereas the dependencies of MPIs are provided by Qlustar packages integrated into Qlustar image modules.
This has also consequences for Qlustar release cycles. There will be updates of the HPC core stack packages approximately every 9 months tied to the current release cycle of slurm. This ensures that Qlustar always provides a slurm version supported by upstream. At this stage, the following packages are part of the Qlustar HPC core stack:
-
rdma-core
-
slurm
-
pmix
-
ucx
-
Spack
2.3. Spack Integration
Qlustar 13 introduces Spack as the new package manager for HPC related software beyond the HPC core stack. Spack has a huge list of supported software packages and provides hardware optimized versions of them by design. Another big advantage of Spack: Multiple versions of the same software can easily co-exist on the same cluster.
The Qlustar version of Spack is provided as an OS package (deb/rpm) and defines the packages of the Qlustar HPC core stack as so-called external packages thus assuring a flawless integration. The same versions of these packages are provided for all Qlustar edge-platforms hence guaranteeing that Spack HPC applications work the same on all of them.
2.4. Qlustar Cross-Version Compatibility
Starting with this release, we provide cross-version compatibility between the current and the previous Qlustar release. In practice this means that on a cluster with a Qlustar 13 head-node, you can still run nodes with Qlustar 12 images/chroots and vice versa.
This new feature was added to allow smooth release upgrades: Admins can already run a subset of nodes with the new Qlustar release and do the final migration on the head-node(s) only, once users have successfully migrated their applications to it. Once the upgrade is done, it is still possible to run some nodes with the old release in case some problematic applications require it.
3. Major component updates/versions
- Kernel 5.15
-
Qlustar 13 is based on the 5.15 LTS kernel series (Ubuntu only) with initial version 5.15.82.
- Slurm
-
Qlustar 13 introduces the Slurm 22.05 series with the current version being 22.05.6.
- ZFS
-
Qlustar 13 includes the ZFS 2.1.x series with the current version being 2.1.6.
- Nvidia CUDA
-
Qlustar 13 provides optimal support for Nvidia GPU hardware by supplying pre-compiled and up-to-date kernel drivers as well as core libraries of CUDA 11.7 needed for integration with other system components. Full CUDA releases should now be installed via Spack.
- Lustre
-
Qlustar 13 comes with the most recent Lustre LTS version 2.15.1 providing support for clients and servers with ready-to-use image modules.
- BeeGFS
-
Qlustar 13 has integrated the most recent BeeGFS release 7.3.2 for clients and servers with ready-to-use image modules.
4. Other notable package version updates
Packages part of the Qlustar HPC core stack:
-
rdma-core: 43.0 -
hwloc: 2.8.0 -
pmix: 4.1.2 -
ucx: 1.13.0
5. General changes
-
Drop OpenHPC integration (OpenHPC is no longer needed now that Spack is used).
-
NIS support was dropped. Admins should have migrated to LDAP by now.
-
No Qlustar OpenMPI packages (debs) are provided anymore (OpenMPI is now supposed to be installed via Spack) and neither packages based on them (e.g. Linpack/hpl).
-
Openafs support was dropped.
-
CentOS 7 is not supported as a Qlustar 13 edge platform anymore. Running Qlustar 12/CentOS 7 images on Qlustar 13 is supported though.
6. Update instructions
-
Preparations
Upgrading to Qlustar 13 is only supported from the most recent 12.0 release. Please make sure that you have updated to this release as described in the latest security advisory on the Qlustar website.
Make sure that you have no unwritten changes in the QluMan database. If you do, write them to disk as described in the QluMan Guide before proceeding with the update.
-
Optionally clone chroots
Clone existing Ubuntu 20.04 and AlmaLinux/8 chroots based on 12.0 and then afterwards upgrade the clones to 13. That allows for easy rollback.
-
Backup important head-node data
The following commands create a backup of the head-node’s
/etcdirectory, the QluMan database and the LDAP directory.0 root@cl-head ~ # cp -a /etc /var/backups/etc-qlustar-12 0 root@cl-head ~ # /usr/share/qlustar/exec/backup-qlustar-db
-
Update to Qlustar 13 package sources list
The Qlustar apt sources list needs to be changed as follows both on the head-node(s) and in all existing Ubuntu based chroot(s) that should be updated.
0 root@cl-head ~ # apt update 0 root@cl-head ~ # apt install qlustar-sources-list-13.0
To prepare an AlmaLinux 8 based chroot for the upgrade, change into it and execute the following:
(alma8) 0 root@cl-head ~ # yum update (alma8) 0 root@cl-head ~ # yum install qlustar-13.0-repos (alma8) 0 root@cl-head ~ # yum remove qlustar-12.0-repos
-
Update head-node packages
On the head-node execute
0 root@cl-head ~ # apt update 0 root@cl-head ~ # apt dist-upgrade
When asked about what services to restart during the update, delete all services from the suggested list. This will prevent the update from failing in case some services might not restart immediately. Since a reboot is necessary after this update, the restart during the update is superfluous.
When asked about whether you want to update the configuration file for some package, you should answer 'N' (keep the old version) unless you have a specific reason to change it.
You will most likely have received an error message about "Configuring slapd". This is expected and fixed by the next step.
-
Execute post-update script
Running the following script will apply some necessary fixes to correct glitches of the automatic apt update process and offers to initialize the filesystem setup needed for the Spack package manager.
The Qlustar Spack setup expects the Linux group softadm and the Linux user softadm (with primary group softadm) to exist. If they don’t, the post-update script will wait for them to be created (usually via QluMan).
0 root@cl-head ~ # qlustar-post-update.sh
Once the update is complete, you can go through the Spack initialization procedure described in the First Steps Guide to get started with Spack package management.
-
Update chroots
Change into each Ubuntu based chroot you want to update (e.g.)
0 root@cl-head ~ # chroot-focal
and also execute (the Spack package needs to be explicitly installed here if required):
(focal) 0 root@cl-head ~ # apt update (focal) 0 root@cl-head ~ # apt dist-upgrade apt install spack
Change into each AlmaLinux 8 based chroot you want to update (e.g.)
0 root@cl-head ~ # chroot-alma8
and execute (confirm the import of the new Qlustar GPG key):
(alma8) 0 root@cl-head ~ # yum update
-
Reboot head-node(s)
Initially only reboot the head-node(s).
-
Regenerating Qlustar images
Regenerate your Qlustar images with the 13.0 image modules. To accomplish this, you have to select flavour jammy for Ubuntu based images (the flavours of CentOS/AlmaLinux based images doesn’t change) and Version 13 in the QluMan Qlustar Images dialog. If you have new cloned chroots, select those as well.
If your images include image modules that have a version in their name (e.g. lustre-2.12-server), make sure that you change to the corresponding module with the most recent version (e.g. lustre-2.15-server).
7. Changelogs
A detailed log of changes in the image modules can be found in the directories
/usr/share/doc/qlustar-module-<module-name>-*-amd64-13.0. As an example, in the directory
/usr/share/doc/qlustar-module-core-jammy-amd64-13.0 you will find a summary changelog in
changelog.gz, a complete list of packages with version numbers entering the current core
module in core.packages.version.gz, a complete changelog of the core modules package versions
in core.packages.changelog.gz and finally a complete log of changed files in
core.contents.changelog.gz.
8. Feature Updates
8.1. Feature Update 03/23
QluMan 13.1.0 highly improves on its Host Filter capabilities:
-
New filters for Property/Config sets, Property/Config types and Property/Config values have been added. These filters make it much easier to select correct groups of hosts for re-configurations or command execution especially on large in-homogeneous clusters.
-
Command execution or host selection using filters temporarily defined on-the-fly is now possible directly from the Host Filter dialog. This helps a lot in quickly checking on a group of nodes based on any configuration criteria.
AlmaLinux was updated to version 8.7.
8.2. Feature Update 06/23
Version 13.2.0 further improves QluMan with the addition of:
-
An informative Host overview table. It provides a tabular overview of all the hosts in a cluster split by networks. Tabs at the top allow to select the network one is interested in. In each tab, the IP ranges allocated in the chosen network are shown at the top and a list of hosts with their most important config parameters below. An on-the-fly string/regex filter can be applied to the lists restricting the view to the hosts matched.
-
New GUI options to fine-tune the LDAP password policy. This makes it easy to control user password quality and protection against brute-force attacks with appropriate settings.
-
The possibility to assign a Lustre LNet name to the network adapters of nodes and generate a new config file to bring up LNet during boot of Lustre nodes.
-
The automatic creation of a new configuration file during boot that lists a node’s QluMan Generic Properties and their values. This allows to use freely definable Generic Properties as a general key/value store in customization scripts on Qlustar nodes.
AlmaLinux was updated to version 8.8.