Release Notes

1. Basic Info

The Qlustar 12.0 release is based on Ubuntu 20.04. It includes all security fixes and other package updates published before Jan 5th 2021. Available security updates relevant to Qlustar 12.0, that have appeared after this date, will be announced on the Qlustar website and in the Qlustar security newsletter. Supported edge-platforms are Ubuntu 20.04 (Focal), CentOS 7/7.9 (with integrated OpenHPC 1.3.9) and CentOS 8/8.3 with (with integrated OpenHPC 2) .

2. New features

2.1. LDAP replaces NIS

Qlustar now uses LDAP instead of NIS to manage users and groups. A fully configured LDAP instance based on OpenLDAP/slapd is created at installation time. It includes TLS support and an automatically configured sssd setup. A corresponding new QluMan component has been developed for convenient user and group management (see below). Updating NIS based 11.0 installations to LDAP is fully supported (the migration steps are detailed at the end of this document).

2.2. QluMan 12.0

Migration to QT5

In a huge 'under the hood' effort, the QluMan GUI has been ported to the current QT 5 toolkit. Most parts of the GUI had to be adjusted for this. This migration ensures the long-term maintainability of QluMan and also improves stability.

LDAP Management

QluMan has a new optional component to manage the Qlustar LDAP instance. It consists of a User and Group Management interface, as well as a powerful interface to import users/groups from passwd like flat files.

NameService Management

Easy and clean integration of external LDAP or Active Directory services are now possible with the new Nameservice Providers and NameService Config dialogs.

3. Major component updates

Kernel 5.4

Qlustar 12.0 is based on the 5.4 LTS kernel series (Ubuntu only) with initial version 5.4.86.

Slurm

Qlustar 12.0 introduces the Slurm 20.02 series with the current version being 20.02.6.

ZFS

Qlustar 12.0 introduces the ZFS 0.8.x series with the current version being 0.8.5.

Nvidia CUDA

Qlustar 12.0 provides optimal support for Nvidia GPU hardware by supplying pre-compiled and up-to-date kernel drivers as well as CUDA 11.0.

BeeGFS

Qlustar 12.0 has integrated the most recent BeeGFS release 7.2 for clients and servers with ready-to-use image modules.

4. Other notable package version updates

Most of the following package versions apply to Ubuntu only. In most cases, the corresponding CentOS packages are part of OpenHPC.

  • rdma-core: 30.0

  • Intel/PGI Compiler support: The Qlustar wrapper packages have been updated to support the new version of the Intel parallel studio (2020) and Nvidia HPC SDK 20.7 (former PGI community edition) with the new package qlustar-nvidia-hpc-sdk-20. Corresponding OpenMPI package variants for these compilers are also provided.

  • OpenMPI: 4.0.5 and 3.1.6

  • Lustre: 2.12.6 (all edge platforms)

  • singularity: 3.2.1

  • openblas: 0.3.13

  • hwloc: 2.2.0

  • pmix: 3.1.5

  • ucx: 1.8.1

  • libpsm2: 11.2.185

  • OmniPath (OPA) stack: 10.10.3.0.11

5. General changes/improvements

  • The default disk setup for net-boot nodes is now XFS on LVM. ZFS as the previous default has proven sub-optimal for HPC use-cases because of occasional instabilities when memory pressure sets in.

6. Update instructions

  1. Preparations

    Upgrading to Qlustar 12.0 is only supported from a 11.0.x release.

    Make sure that you have no unwritten changes in the QluMan database. If you do, write them to disk as described in the QluMan Guide before proceeding with the update.

  2. Optionally clone chroots

    Clone existing Ubuntu 18.04 and CentOS7/8 chroots based on 11.0 and then afterwards upgrade the clones to 12.0. That allows for easy rollback.

  3. Update to 12.0 package sources list

    The Qlustar apt sources list needs to be changed as follows both on the head-node(s) and in all existing Ubuntu based chroot(s) that should be updated.

    0 root@cl-head ~ #
apt update
0 root@cl-head ~ #
apt install qlustar-sources-list-12.0

    To prepare a CentOS 7 based chroot for the upgrade, change into it and execute the following:

    (centos7-11.0) 0 root@cl-head ~ #
sed -i -e 's/11.0/12.0/g'  /etc/yum.repos.d/qlustar-11.0-centos7.repo

    To prepare a CentOS 8 based chroot for the upgrade, change into it and execute the following:

    (centos8-11.0) 0 root@cl-head ~ #
sed -i -e 's/11.0/12.0/g'  /etc/yum.repos.d/qlustar-11.0-centos8.repo

  4. Update head-node packages

    On the head-node execute

    0 root@cl-head ~ #
apt update
0 root@cl-head ~ #
apt dist-upgrade

    When asked about whether you want to update the configuration file for some package, you should answer 'N' (keep the old version) unless you have a specific reason to change it.

    You will most likely have received some error messages about updating Nagios related packages. This is expected and fixed by the next step.

  5. Execute post-update script

    Running the following script will apply some necessary fixes to correct glitches of the automatic apt update process.

    0 root@cl-head ~ #
qlustar-post-update.sh

  6. Update chroots

    Change into each Ubuntu based chroot you want to update (e.g.)

    0 root@cl-head ~ #
chroot-bionic

    and also execute:

    (bionic) 0 root@cl-head ~ #
apt update
(bionic) 0 root@cl-head ~ #
apt dist-upgrade

    Change into each CentOS 7/8 based chroot you want to update (e.g.)

    0 root@cl-head ~ #
chroot-centos7

    and execute (confirm the import of the new Qlustar GPG key):

    (centos7-11.0) 0 root@cl-head ~ #
yum update
(centos7-11.0) 0 root@cl-head ~ #
yum install qlustar-12.0-repos
(centos7-11.0) 0 root@cl-head ~ #
yum remove qlustar-11.0-repos
(centos7-11.0) 0 root@cl-head ~ #
yum update

  7. Reboot head-node(s)

    Initially only reboot the head-node(s).

  8. Regenerating Qlustar images

    Regenerate your Qlustar images with the 12.0 image modules. To accomplish this, you have to select Flavour focal for Ubuntu based images (the flavour of CentOS based images doesn’t change) and Version 12.0 in the QluMan Qlustar Images dialog. If you have new cloned chroots, select those as well.

    If your images include image modules that have a version in their name (e.g. beegfs-6-server), make sure that you change to the corresponding module with the most recent version (e.g. beegfs-7-server).

  9. Install lmod and migrate module files

    Qlustar 12 has replaced the modules package with lmod. You need to install the corresponding package:

    0 root@cl-head ~ #
apt install lmod

    Now you can move your module files (if any) from the old modulefiles path /etc/qlustar/common/modules/modulefiles to the new one /etc/qlustar/common/lmod/modulefiles.

  10. Migration to LDAP (optional)

    It is possible to migrate user and group management of the cluster from NIS to LDAP. This step is completely optional. A cluster using NIS for this purpose is still fully supported and will remain so in the foreseeable future. The migration can be done at any time once running Qlustar 12.

    If a migration to LDAP is desired, the following steps have to be taken:

    1. Install QluMan LDAP support

      0 root@cl-head ~ #
apt install qlustar-headnode-hpc

      The LDAP admin password to be entered at this stage is not relevant. The real one will be asked later. Also ignore the error messages about the failed QluMan LDAP connection.

    2. Bootstrap LDAP server

      0 root@cl-head ~ #
setup-qlustar-ol --bootstrap

      Enter the LDAP admin password when asked.

    3. Restart qlumand and install sssd

      0 root@cl-head ~ #
service qluman-server restart
0 root@cl-head ~ #
apt install sssd

    4. Import users and groups from NIS

      Copy the files passwd, shadow and group located on the headnode under /etc/qlustar/yp to a directory which is accessible on the machine where you run qluman-qt from. Then follow the instructions from the QluMan guide.

    5. Assign Nameservice config

      Make yourself familiar with the new QluMan Nameservice concept introducing Nameservice Providers and Configs. Then assign the pre-defined Nameservice config Qlustar to those nodes on which users will login using their password (typically FrontEnd nodes). Cluster-internal nodes do not need this assignment, since they use ssh host-based authentification.

    6. Disable NIS / cleanup sssd

      Once you’re confident, that the LDAP setup works correctly, NIS can be disabled by executing

      0 root@cl-head ~ #
apt remove nis
0 root@cl-head ~ #
service qluman-server restart

      and removing the nis entries in /etc/nsswitch.conf.

      qlumand running on the head-node detects whether the nis package is installed and if it isn’t, instructs booting nodes not to start NIS.

      Also prevent cluttering of the head-node’s boot process by disabling unwanted sssd systemd units:

      0 root@cl-head ~ #
for svc in $(ls /etc/systemd/system/sssd.service.wants/sssd*.socket); do \
  systemctl disable ${svc##*/}; done

7. Changelogs

A detailed log of changes in the image modules can be found in the directories /usr/share/doc/qlustar-module-<module-name>-*-amd64-12.0.0. As an example, in the directory /usr/share/doc/qlustar-module-core-focal-amd64-12.0.0 you will find a summary changelog in core.changelog, a complete list of packages with version numbers entering the current core module in core.packages.version.gz, a complete changelog of the core modules package versions in core.packages.version.gz and finally a complete log of changed files in core.contents.changelog.gz.