The Qlustar 11.0 release is based on Ubuntu 18.04. It includes all security fixes and other package updates published before June 13th 2019. Available security updates relevant to Qlustar 11.0, that have appeared after this date, will be announced on the Qlustar website and in the Qlustar security newsletter. Supported edge-platforms are Ubuntu 18.04 (Bionic), CentOS 7 and CentOS 8 with integration of OpenHPC 1.3.8.
dnsmasq is now employed as a central Qlustar component to consolidate three network services previously provided by three independent daemons and hence significantly reduces complexity on the head-node. In addition it provides DNS proxy services per default which had to be configured manually in earlier Qlustar releases.
dnsmasq now provides cluster-internal DHCP services replacing the previously used ISC DHCP server software.
dnsmasq also acts as a TFTP/PXE boot server making the previously used atftpd obsolete.
- Cluster-internal DNS
dnsmasq provides DNS name resolution both for cluster-internal nodes as well as external machines via its proxy functionality. As a consequence, the NIS host map is not needed anymore and has been removed for new installations.
- Legacy support
When updating from an earlier release, you have the option to keep the previous DHCP/TFTP setup through a legacy option for now (see below).
- Network Mount Management
QluMan has a new Config Class that allows to configure and assign network mounts to nodes. Initially NFS mounts including RDMA variants are supported. They are setup on the nodes as systemd automount units. + QluMan automatically activates NFSoRDMA on clients that support it with an option to switch back to TCP if needed. It also allows to define priorities for the available networks, so that the network to be used for a mount is optimally chosen if a node provides multiple paths to the corresponding NFS server.
- DNSmasq Management
The newly introduced dnsmasq service is fully manageable via QluMan. A new dialog has been introduced to add external DNS servers and search domains, as well as name/IPs for cluster external machines and other global network related settings.
- Preview node-specific configs
The context menu of a node in the Enclosure View now includes an entry that allows a preview of all host-specific files/configs that are assigned and sent to a node when booting.
- Kernel 4.19
Qlustar 11.0 is based on the 4.19 LTS kernel series (Ubuntu only). The new kernel option mitigations=off has been added to QluMan for an easy way to run compute nodes without performance penalties from CPU security bug mitigations (Spectre etc.).
Qlustar 11.0 introduces the Slurm 18.08 series with the current version being 18.08.7.
Qlustar 11.0 upgrades to OpenMPI 4.0.1 now using ucx transport per default. Furthermore, we added support for multiple gcc versions, now with a gcc7 flavor based on the Ubuntu default compiler (gcc 7.4.0) and a gcc8 flavor based on gcc 8.3.
- Nvidia CUDA
Qlustar 11.0 provides optimal support for Nvidia GPU hardware by supplying pre-compiled and up-to-date kernel drivers as well as CUDA 10.1.
Qlustar 11.0 has integrated the most recent BeeGFS release 7.1.3 for clients and servers with ready-to-use image modules.
rdma-core: 21.0 (Ubuntu only, on CentOS, the original RHEL OFED stack is used).
Intel/PGI Compiler support: The Qlustar wrapper packages have been updated to support the new version of the Intel parallel studio (2019) and PGI community edition 2018.4/10 (package qlustar-pgi-dev-tools). Corresponding OpenMPI package variants for this compiler are also provided (both Ubuntu only).
OmniPath (OPA) stack: 10.8.0.0.201/2
The automount daemon together with the corresponding NIS maps have been replaced in favor of the new network mount config class provided by QluMan (see above). Legacy setups based on automount/NIS will continue to function and will be supported.
Upgrading to Qlustar 11.0 is only supported from a 10.1.x release.
Make sure that you have no unwritten changes in the QluMan database. If you do, write them to disk as described in the QluMan Guide before proceeding with the update.
Optionally clone chroots
Clone existing Ubuntu 16.04 and CentOS7 chroots based on 10.1 and then afterwards upgrade the clones to 11.0. That allows for easy rollback.
Update to 11.0 package sources list
The Qlustar apt sources list needs to be changed as follows both on the head-node(s) and in all existing Ubuntu based chroot(s) that should be updated.
0 root@cl-head ~ # apt update 0 root@cl-head ~ # apt install qlustar-sources-list-11.0
To prepare a CentOS7 based chroot for the upgrade, change into it and execute the following:
(centos7-11.0) 0 root@cl-head ~ # sed -i -e 's/10.1/11.0/g' /etc/yum.repos.d/qlustar-10.1-centos7.repo
On the head-node execute
0 root@cl-head ~ # apt update 0 root@cl-head ~ # apt dist-upgrade
When asked about whether you want to update the configuration file for some package, you should answer 'N' (keep the old version) unless you have a specific reason to change it.
Change into each Ubuntu based chroot you want to update (e.g.)
0 root@cl-head ~ # chroot-bionic
and also execute:
(bionic) 0 root@cl-head ~ # apt update (bionic) 0 root@cl-head ~ # apt dist-upgrade
Change into each CentOS based chroot you want to update (e.g.)
0 root@cl-head ~ # chroot-centos7
and execute (twice the same command):
(centos7-11.0) 0 root@cl-head ~ # yum update (centos7-11.0) 0 root@cl-head ~ # yum update
Confirm the import of the new Qlustar GPG key.
Initially only reboot the head-node(s).
Regenerating Qlustar images
Regenerate your Qlustar images with the 11.0 image modules. To accomplish this, you’ll have to select Version 11.0 in the QluMan Qlustar Images dialog. If you have new cloned chroots, select those as well.
If your images include image modules that have a version in their name (e.g. beegfs-6-server), make sure that you change to the corresponding module with the most recent version (e.g. beegfs-7-server).
Migration to GRUB PXE booting To enable support for UEFI PXE booting,
GRUBis now used per default as the boot loader for both EFI and PC BIOS netboot nodes. To make this work, the new package
qlustar-headnodemust be installed at this time.
This is absolutely necessary, otherwise PXE booting won’t work anymore.
If for some reason, you want to keep booting PC BIOS nodes via the legacy PXELINUX mechanism, you will have to install the package
Migration to dnsmasq
Migration to dnsmasq is not absolutely required during this upgrade, but highly recommended. However, for PXE booting of UEFI nodes it is mandatory. If you want to delay the migration for now, you can do so by checking the legacy checkbox in QluMan. In this case you still have to disable systemd-timesyncd though in order for ntp to work:
0 root@cl-head ~ # service systemd-timesyncd stop systemctl disable systemd-timesyncd
You also have to install the package
qlustar-netboot-compatin this case. After this, you can reboot the head-node once more and proceed with step 8.
Support for the old legacy setup with separate DHCP and TFTP servers will be discontinued starting with the 12.0 release. So you’ll have to do the migration sometime before updating to 12.0. To do so, just follow the remainder of this step at any time.
To start the migration you first have to install dnsmasq and disable the old DHCP and TFTP server as well as some systemd services:
0 root@cl-head ~ # apt install dnsmasq 0 root@cl-head ~ # for s in isc-dhcp-server atftpd systemd-resolved systemd-timesyncd; do service $s stop systemctl disable $s done
This write step is essential. If you forget it, qlumand won’t be able to start after a reboot and you’ll be left with a system in an inconsistent state that needs manual repair.
Also remove nis from the hosts entry in
0 root@cl-head ~ # sed -ie 's/^hosts:./hosts:\t\tfiles dns/g' /etc/nsswitch.conf*
and change the dns-nameservers entry in
/etc/network/interfacesto localhost, so that the head-node itself is using dnsmasq for DNS lookups.
0 root@cl-head ~ # sed -ie 's/ dns-nameservers./ dns-nameservers localhost/g' /etc/network/interfaces*
Finally reboot the head-node once more. Once it is up again, check that dnsmasq is running:
0 root@cl-head ~ # service dnsmasq status
If everything is working as expected, you can remove the now obsolete packages:
0 root@cl-head ~ # apt remove atftpd isc-dhcp-server
Write config file changes
To activate all remaining changes in the QluMan database that were introduced by the update, they need to be written to disk now. Check the QluMan Guide about how to write such changes.
Reboot all netboot nodes
After the regeneration of the images is complete, and all the above steps have been done, you can reboot all other nodes in the cluster, including virtual FE nodes. This completes the update procedure.
Adjust root bash shell initialization
Starting from this release, Qlustar supplied bash shell functions and some aliases for root on the head-node have been migrated to the qlustar-base package and now live underneath
/etc/qlustar. Previously they have been generated on the fly during installation, and hence were not easily upgradeable.
To move to the new scheme, the following modifications have to be made manually with a text editor on the cluster head-node(s). In the file
/root/.bashrcreplace the content at the end from
# Source other files source_list="/root/.bash_aliases /root/.bash_functions" shopt -oq posix || source_list="$source_list /etc/bash_completion" for c in $source_list ; do [ -r $c ] && . $c ; done; unset c
# Source misc bash stuff: # First global Qlustar aliases/functions (from qlustar-base package) source_list="/etc/qlustar/bash.aliases /etc/qlustar/bash.functions" # ... then the local equivalents source_list="$source_list /root/.bash_aliases /root/.bash_functions" # Finally activate additional completions shopt -oq posix || source_list="$source_list /etc/bash_completion" for c in $source_list ; do [ -r $c ] && . $c ; done; unset c
In the file
/root/.bash_aliasesdelete or comment out the following two lines:
alias console-fe-vm="connect_screen login-vm" alias console-demo-vms="connect_screen demo-system"
These two aliases are now provided globally and the old ones will not work anymore.
A detailed log of changes in the image modules can be found in the directories
/usr/share/doc/qlustar-module-<module-name>-*-amd64-11.0.0. As an example, in the directory
/usr/share/doc/qlustar-module-core-xenial-amd64-11.0.0 you will find a summary changelog in
core.changelog, a complete list of packages with version numbers entering the current core
core.packages.version.gz, a complete changelog of the core modules package versions
core.packages.version.gz and finally a complete log of changed files in