Updating Qlustar

Qlustar updates

Updating a Qlustar cluster is a multi-step process. The detailed steps depend on whether the update involves a kernel update or not (the release notes will mention it, if a new kernel is part of the update) and what module versions you have selected for your images. We’ll explain the differences below. Follow the steps in the following order.

Updating the head-node(s)

Apply the standard Debian procedure to update the head-node(s) installation. Execute as root:

0 root@cl-head ~ #
apt update
0 root@cl-head ~ #
apt dist-upgrade

This will update all packages. In most cases, if new versions of the Qlustar image module packages are available, this will also automatically rebuild the images defined in QluMan. We elaborate on this below.

If the kernel was updated by this process reboot the head-node(s) after executing the above commands, otherwise you’re done with this step.

Updating the chroot(s)

This is either also a standard Debian or a standard CentOS upgrade as explained in the First Step Guide. If you have setup multiple chroots, you’ll have to update all of them.

If a new version of the spack package manager has been installed in the chroot, it may be necessary to run a spack command as a user with write permissions on /apps/local/spack (e.g. spack list as user softadm). Otherwise, unprivileged users might see permission denied errors when working with spack.

Updating the nodes

If the update contains a new kernel, reboot all nodes. In case you have some storage nodes that export a global file-system (e.g. NFS or Lustre), reboot them first, then all the other nodes.

If the update doesn’t contain a new kernel, you have the choice of either rebooting the nodes like above or using the Qlustar online update mechanism (currently the online update has to be executed manually, future versions of QluMan will allow to do this from the GUI) as follows: Execute the following command on the head-node:

0 root@cl-head ~ #
qlustar-image-update -H <hostlist>

where <hostlist> is a list of nodes in hostlist format, e.g. beo-[01-80]. You also have the option to first check what exactly the update will change. To find out, execute the following:

0 root@cl-head ~ #
qlustar-image-update -c -H <hostlist>

This will show what will be changed by this update.

0 root@cl-head ~ #
qlustar-image-update -s -H <hostlist>

This will show all services (daemons) that need to be restarted as a result of this update. If you want more detailed information about the update process, add the option -v to any of the previous command lines.

When will images be updated/rebuilt?

There are a couple of cases to distinguish, depending on what version is selected for an image in QluMan (see also the corresponding section in the QluMan Guide.

  1. Selected image version is of type x.y.z, e.g. 11.0.0. In this case, the image will be rebuilt if and only if there is an update for the selected version (11.0.0) of the modules. Such updates will always be bug-/security-fixes.

  2. Selected image version is of type x.y, e.g. 11.0 with 11.0.0 being the most recent 11.0.z version before applying the update. In this case the image is rebuilt, if the update entails a new maintenance release (e.g. 11.0.1 which will then become the new real image version) or an update for the currently installed (11.0.0) modules.

  3. Selected image version is of type x, e.g. 11 with 11.0 being the most recent 11.y and 11.0.0 being the most recent 11.0.z version before applying the update. In this case the image is rebuilt, if either the update entails a new feature release (e.g. 11.1, the latest 11.1.z will then become the new real image version) or a new maintenance release (e.g. 11.0.1 which will then become the new real image version) or an update for the currently installed (11.0.0) modules. With this option, manual intervention to obtain a new image version is only necessary when upgrading to a new major release (e.g. 11.y.z).