QluMan Guide

The status of the network connection between a GUI session and the relevant QluMan server components (qlumand, qluman-slurmd, etc.) is displayed by LEDs in the status bar of the main QluMan window. The QluNet internal network client ID is also listed there.

When starting up, the QluMan GUI will check whether there is a newer QluMan version available for download. If a newer version is found, or the major version currently in use is no longer supported, a button Update Available will appear in the lower left corner of the main window. Clicking on it will show more details about the version currently in use and the one available for download.

To add a new cluster click the New button and select New Cluster (paste) or New Cluster (file) from the menu depending on whether you want to paste the one-time token or load it from a file. If you don’t have a one-time token for the cluster see Generating the Auth Token.

Paste the one-time token data into the dialog and click Decrypt or select the file containing the token. When asked for the password, enter the pin

that was used when creating the token (in case you didn’t generate the token yourself, you should have been told the pin by your main cluster administrator). The dialog should then show the cluster/head-node infos that where packed into the one-time token. If you started qluman-qt on your workstation, then you might have to change the Local Hostname to use the external hostname of the head-node. Similarly, if you changed the port for qlumand or if you’re

connecting via port forwarding, you have to adjust that too. The Alias is the name this cluster will be shown as, when using the drop-down menu in the Connect Cluster dialog. Click Ok to add the cluster connection.

After adding the new cluster select Save to save the changes. If this is your first cluster then it will create the certificate safe and ask you to enter and confirm a password. Otherwise it will ask for the existing password unless the Remember password check-box was enabled.

If multiple cluster connections are registered, the corresponding entries can be reordered using drag&drop. This allows to move them to the desired location in the list. Clusters can also be grouped in sub-menus by first creating a new group (sub_menu) and then dragging cluster entries into it. The tree structure of the Manage Clusters dialog will be reflected in the drop-down menu

of the Connect Cluster dialog. This allows a nicely structured layout when dealing with a larger number of clusters as e.g. in the case of service providers. Standard cluster admins will most likely not need this feature.

The Manage Clusters dialog allows changing the password for the certificate safe. This requires entering the old password for the safe as well as the new password and a confirmation of the new password. The Ok button will only be selectable if the new password and confirmation matches.

A network can be added to the Network Config by selecting a network definition from the entries below Add Network in the context menu. This opens a dialog where the type and name of the NIC for this network may be selected.

The NIC type should be selected first using the corresponding drop-down menu. A selection of valid types is available to choose from. The drop-down menu for the NIC name lists all the previously used names of the same type for easy selection. A new name can also be entered directly, in case the NIC has a name not previously encountered.

The default name for Infiniband adapters is the kernel name of the IP-over-IB device, which usually has the form ib<N> starting with ib0. Even simpler, the name for IPMI adapters is ipmi0.

After selecting the NIC type and name, click OK and the new entry will appear in the Network Config dialog. If more networks are needed, simply repeat the procedure above for each of them.

In an IB fabric, at least one node (or switch) has to run a subnet manager process that manages the IB routing tables. Qlustar provides OpenSM for this task. If the head-node is also part of the IB network, it’s usually best to configure it to run OpenSM. This might have been chosen during installation, in which case there is nothing more to be done. If not, you have the option to run OpenSM on ordinary nodes too.

In this case, it is advisable to run OpenSM on two or three nodes (not more) for redundancy reasons. It is therefore best, to configure this directly for the chosen hosts, rather than using a Host Template or generic property set. After selecting the host(s) where OpenSM should run in the Enclosure View, open the context menu and select Set Generic Property  OpenSM Ports  ALL. The next time the host(s) boots, the OpenSM daemon will be started on all its Infiniband ports.

If a host has more than one IB port, OpenSM can also be configured to run only on a specific one rather than on all of them. The port can be specified by its number or by its unique ID. As this is an uncommon configuration and the unique ID is unknown beforehand, there is no preset value for this. To create a new value, first select an existing value, e.g. ALL, for the generic property OpenSM Ports. You can then edit the value in the Generic Properties box of a host. Editing the line and pressing Enter will create the new value. Beware that this will only affect one shown host. To assign the new value to other hosts, select them and then change the OpenSM Ports property through the context menu.

In some circumstances, it might be necessary to run OpenSM with extra options. This can also be configured via Generic Properties. The only preset value is the empty string, so you need to create a new value for the options you require. First add the empty value of the generic property OpenSM Options to one host. Then edit the value to your requirements and press Enter to create it. Finally add/change the OpenSM Options generic property for all relevant hosts.

DNS has a hierarchical design. Each correctly configured computer knows about a DNS server that will handle local requests. Should the request fall outside the scope of the local server, it will ask the next higher server in the hierarchy. Starting with Qlustar 11, local DNS requests are handled by dnsmasq which will answer requests concerning any QluMan configured node automatically. The handling of requests about external hosts are affected by three QluMan settings: The external DNS servers, the DNS search domains and Cluster-external Hosts.

The qlustar/common path specifies the location of the cluster-wide configuration directory on the head-node that is used for its NFS export. This path value should not be changed unless there is a good reason for it. A custom value is usually required only for a head-node setup in high-availability mode.

Sometimes the cluster head-node does not have direct access to the internet and requires a proxy server for a connection to the Qlustar repository servers. To enable support for such scenarios, click the check-mark before Http Proxy and enter the hostname of the proxy server together with the proxy port.

If the proxy requires authentication, click the check-mark before Authenticate and enter a username and password. The Http Proxy and User/Pass label will turn green when entries are edited with acceptable input but have not been saved yet. The labels will turn red when the current input is invalid and turn back to black once the input has been saved. The input can be saved by pressing Enter, or will be saved automatically when the input field looses focus. Leaving the user name field empty will disable authentication just the same as clearing the Authenticate check-mark.

Nameservice providers running LDAP or Active Directory (AD) services can be used to provide user/group information and authentication to the cluster. The provider for the cluster-internal LDAP service is defined by default and named Qlustar. Other provider definitions are only needed if external LDAP or AD services are to be used. If so, they need to be configured as described below.

A new Nameservice Provider can be created by clicking the New button and filling out the form. The fields are color coded to show required and valid inputs. Once all fields are filled in correctly, the Create button turns green and clicking it creates the new Nameservice Provider definition. To create an AD provider definition, check the Use AD server with Kerberos check-mark and enter the Kerberos 5 server and Realm information.

Once a Nameservice Provider has been created with its essential configuration, additional options can be added or removed at the bottom through the context menu. Only a few options are pre-defined, those needed for the Qlustar config. Additional ones can be added through the property editor. For details about allowed options and their meaning please refer to the sssd.conf manpage.

Besides adding or removing options, the essential configuration can also be edited by clicking the respective field. Any changes must be saved by clicking the Save button at the bottom. Clicking the Undo button or selecting a different entry in the tree on the left will undo any changes.

Once the configuration is complete, the Preview button allows double checking the configuration that will be generated when the selected Nameservice Provider is used later to generate the sssd.conf of a node. The shown snippet will be a part of the overall sssd configuration of the host.

The boot process for Qlustar has two stages. First the kernel and a minimal initramfs is loaded using PXE support of the node’s BIOS. The initramfs then downloads a squashfs image using multicast provided by the Qlustar Multicast Daemon (ql-mcastd).

The generated configuration of ql-mcastd can be previewed and written as part of the DNSMasq file class. It includes any configured network that has bootable nodes as well as IP and port parameters defined in the MCastd section of /etc/qlustar/qluman/qlumand.cf:

In the unlikely event of a conflict with other services in the same network, the multicast IP, control port and port range used for transfers can be changed. After editing the file, the qluman-server service must be restarted so the new settings will be included in the ql-mcastd.conf. If the control port has been changed, then, after writing ql-mcastd.conf, all Qlustar images must be rebuild as well using:

The cluster node name can be edited at the top. Normally, the node name will also be used as the real hostname (displayed by the hostname command). If a different hostname is desired, it can be entered by clicking the Override hostname check-mark and entering the name in the text field next to it.

To complete the change of the hostname, the network that the chosen new hostname should be part of, must be selected from the list offered in the drop-down menu at the right of the text-field. This choice determines the IP address of the host, which will be used for the hostname in the NIS config, and which applications will see when they resolve the hostname.

In the center of the window, the host’s individual network config is shown. Initially, only the boot network is visible, as shown in the screenshot. Clicking the > button will display the complete list of networks assigned to the host through its Network Config. For each network, the Host IP, MAC address and optional aliases are shown (more details in section Host Networks Settings).

While editing a field, the label corresponding to it, will change color indicating the validity and state of the information entered so far. A green label means that the current input is a valid choice, but has not been saved yet. A yellow label means that the input is incomplete, but can still be completed to become valid (this only applies to the MAC address field, in case a partial address is entered). A red label indicates that the current input is invalid and can not be used.

The tool-tip, displayed when the mouse moves on top of the label, states the reason, why the entered value is invalid. The most likely reason is, that the input is already used by another host. The node name, hostname override, the host IPs, MAC addresses and aliases must all be unique.

Along with the color coding of the label, the Save button will also become enabled and change color, whenever a field is edited and not yet saved. It will turn red if any of the changes are invalid, even if some of them are. Again, the tool-tip will point to the reason why values are invalid. Changes can be saved by pressing return in the text field or by clicking the Save button. Once all changes have been saved in the database, the corresponding labels will turn black again and the Save button will become disabled.

For nodes that are not part of a multi-host enclosure (like a Blade or Twin chassis) the enclosure type can be changed to one of the single-slot host enclosures (1U, 2U, etc.). A new enclosure of the chosen type will then be created if the node is not already part of one. If a node is included in a multi-host enclosure, this field will be ghosted.

The template field allows to select a so-called Host Template for the node. Usually, large groups of nodes have an identical hardware and software configuration and will use the same template. Deviations from the properties coming from the template can be set for individual hosts by direct assignment of either a property/config set or individual properties/configs directly to the host through its context menu. In case of unique properties, direct assignments override settings from the template (or property set), for non-unique properties this is additive.

New enclosures can be added through the context menu. The new enclosure must be given a name and its type can be selected. Currently, enclosure types cannot be manipulated yet. This will change in a future version.

Suitable for ordinary servers, a host being selected in the enclosure view can be placed into a single slot host enclosure directly by selecting the correct type in the host info part of the window (see Enclosure View). For host enclosures that can hold more than one server/node (twin servers, blades etc.), drag&drop may be used to move hosts into them. Moreover, it’s also possible to create larger (non-host) enclosures (like racks) and move host enclosures into them also by using using drag&drop. Note, that a bare host cannot be placed directly into a non-host enclosure, only if it is already inside a host enclosure.

Another option to place hosts into enclosures is by selecting a number of them and then choosing a host enclosure from the context menu. This way, a new enclosure of the selected type is automatically created and all selected hosts are moved into it. If more hosts than can fit into a single enclosure of the chosen type are selected, additional enclosures of the same type will be created such that all hosts can be placed into one of them. This makes it easy to position large numbers of identical hosts into their enclosures. If the selected hosts were in an enclosure before and that enclosure becomes empty and is not itself part of a larger enclosure then the empty enclosure is automatically removed.

Relocating hosts by selecting a different host enclosure is supported not only on directly selected hosts but also on hosts inside selected enclosures. This allows changing the type of enclosure a group of hosts is in by selecting the old enclosure(s) and choosing a new one from the context menu. Note that this procedure does not change the type of the old enclosure but rather creates a new one, moves all the hosts to it and then deletes the now empty old enclosure(s).

Try it out: Place a number of hosts into a large enclosure (like a blade), then select the enclosure and choose a small enclosure (like 1U) to relocate them. In general, such an operation will create one enclosure of the new type and fill all its slots before creating a second one. Hosts having been in different enclosures before, can end up in the same enclosure and hosts that were in the same enclosure before can end up in different enclosures after this operation.

When using drag&drop for the relocation, the host or enclosure is always placed into the lowest suitable slot of the target enclosure. This reflects our experience, that usually enclosures are simply filled from left to right and bottom to top.

But sometimes this is not the case and a host or enclosure should be in a different slot as compared to the automatic placement. In this case, the host or enclosure can be moved through the context menu. The latter shows all the free slots the host or enclosure can be relocated to and a checked mark indicates the current location. Of course the relocation is only allowed into free slots. Hence, it may require removing (drag&drop out of the enclosure) a host or enclosure temporarily to free space for moving things around.

The configuration of a host results from the assignment of different types of properties and Config Classes to it. A property is always a key + value pair. They are further divided into Generic Properties and Hardware Properties. Generic/hardware properties and config classes can be individually assigned to a host. This is the configuration level with highest priority, meaning that such an assignment will always be effective.

They can also be used to define Generic Property Sets, Hardware Property Sets and Config Sets. This is simply a means of grouping them together, so they can be used as a single entity. These sets may also be individually assigned to a host. This is the configuration level with second highest priority.

The third level of configuration are Host Templates. A Host Template consists of exactly one Config Set, one Generic Property Set and one Hardware Property Set.

The fourth and most generic level of configuration is the Global Template. It applies to all hosts in the cluster and consists of the Global Generic/Hardware Property and Config Set. In principle, the latter are just like any other set, with the one difference that they always apply to all hosts. This is useful when defining a base configuration for a cluster.

If a generic/hardware property or config is defined and assigned to a host in a particular hierarchy level, it overwrites the corresponding assignment(s)/value(s) from all lower priority levels. This introduces a lot of flexibility while retaining consistency.

For example it allows setting a property in the Global Template that is right for most hosts and then replace it in a particular Host Template being used for a few exceptional hosts. The screen-shot shows two examples of this strategy: The assigned Host Template of the chosen host replaces the configs for Net Config, Slurm Partition and Slurm Node Group of the Global Template (red box) and the number of CPU cores is directly assigned replacing the value from the Host Template (green box).

Hardware Properties

Hardware Properties are used to describe the hardware of a host. Among others, hardware properties like the amount of RAM or number of CPU cores are used to configure the Slurm workload manager, so jobs can be assigned to the desired hosts. Others, like e.g. the HW type, are purely informational and might be used for inventory management.

Hardware Properties usually don’t have to be entered manually. When a host boots, its basic hardware properties are detected and reported to the head-node. They are shown as a tool-tip in the Enclosure View when hovering over a host. Hosts that have none or only partially configured Hardware Properties will show a blue wrench, while hosts for which the configured Hardware Properties do not match the detected ones will show a red wrench. The Hardware Wizard uses the reported values to generate the correct configuration for a set of hosts and is the best way to create their initial configuration. It can also be used to correct the configuration of hosts in case their hardware was changed.

Selecting a node in the Enclosure View tree displays its configuration info on the right hand side. This includes the red or blue wrench if there is a problem with the configuration of Hardware Properties or GPUs. In the latter case, a few more details about the GPUs are included. Individual Hardware Properties that don’t match will be shown in red in the Properties and Configs tree and their tool-tip will show the detected value.

Using a host entry’s tool-tip or selecting the host to get further details about a configuration problem works only for one host at a time. With many misconfigured hosts, an easier way to get an overview of problems is by opening the Problems Report from the Manage Cluster menu. This will show the hardware and GPU configuration problems of all nodes in a list, displaying both the detected and configured values for each problematic GPU or hardware property.

At the bottom of the dialog an overall host-count of the problem is shown together with two check-boxes that allow to toggle the display of problems with Hardware Properties or GPUs selectively.

Generic Properties

A property that is not hardware related is called generic. Generic Properties can be configuration options, like OpenSM Host, or purely informational, like Paid by. While hardware properties are meant to be more rigid, typically with a configurable set of fixed values, generic properties are more flexible and can be defined at will. Generic Properties are also not necessarily unique, making it possible to assign multiple values for a single generic property. This is useful e.g. to put hosts in multiple groups for dsh/pdsh (via the 'Host tag').

Generic/Hardware properties and their values are managed by the corresponding Property Editor. It is reachable from the Generic/Hardware Property Sets windows. A new Generic/Hardware property can be created by clicking the New button. Each property has a name and a description. The name must be unique and the Ok button will only be enabled if that is the case. The description is optional and will be shown as a tool-tip when hovering over the property in other windows.

Once a property exists, values can be added to it. To accomplish this, the property must first be selected from the drop-down menu at the top of the Property Editor. Then values may be added or deleted using the context-menu.

With the exception of essential pre-defined properties, a property can be edited by clicking the Edit button. Examples of properties that can not be edited (or deleted) are the ones used to generate the slurm.conf. Changing their name would lead to errors in the generated file. Similarly, some property values can not be deleted and will be shown grayed out.

Generic/hardware sets and config sets simplify the configuration of hosts substantially. They are a means of grouping generic/hardware properties or configs, so they can be used as a single entity and be assigned to individual hosts directly or via a Host Template. There is also a global set of each type which is always assigned to every host. The generic/hardware and config sets can be managed by opening the corresponding window from the Manage Sets menu.

Hardware/Generic Property Sets

Property sets are shown in tree form with the name of each set as the top level item and the key/value pairs of assigned properties as children. The first entry is always the global property set. A new property set can be created by clicking the New button or from the context menu. An existing property set can be renamed or the description changed by selecting Edit Property Set from the context menu. Deletion of an unused property set is also possible via the context menu.

New properties can be added to a property set by opening the Add Property sub-menu in the context menu: Select the desired property and its value. Only properties and values that don’t conflict with already included properties or values are shown in the sub-menu. Already included properties may be changed in two ways: Either a different value is selected from the drop-down menu or the value is edited directly, in which case the change must be confirmed by pressing Enter. If the entered value for the property doesn’t exist yet, a new value is automatically created. Another way to create new values and the only way to delete them is through the property editor by clicking Edit Properties.

Config Sets

Config sets are managed the same way as property sets. They are displayed as a tree with the global set at the top and the key/value pairs of assigned config classes as children. Creating a new config set can be done by clicking the New button or from the context menu just like for property sets. Adding and removing config classes to/from a set also works the same way.

Unlike property sets though, the list of usable config classes is fixed and values can not be edited directly from this window. Config classes require a more complex and individual config. So for each config class there is a menu item in the Manage Configs menu. Depending on the type of config class, one or more values for that class can be assigned to a Config Set (slurm partitions are an example where multiple values are assignable).

As mentioned in the beginning of this chapter there are 4 levels of configuration. Going from lowest to highest priority these are: Global Template, Host Template, directly assigned sets and directly assigned properties and configs.

Global Template

The Global Template is the most generic way to assign configurations to hosts. It applies to all hosts in the cluster and consists of the Global Generic/Hardware Property and Config Set.

Host Templates

The next level of configuration is the Host Template. When a correct Host Template exists, a host can be configured by selecting the desired template in the Enclosure View window. For a single host, this can be done by selecting it in the tree view. This brings up the host information on the right and a template can be selected from the drop-down menu. To configure multiple hosts, you would select them in the tree view and choose a Host Template from the context menu. The check-marks in the sub-menu indicate which Host Templates are currently assigned (if any) for the selected nodes. This action will override the previous assignment for all selected hosts.

Alternatively, especially when no correct Host Template exists yet, the Hardware Wizard can be used to to create a new or modify an existing Host Template and assign it to hosts.

Directly assigned properties, configs and sets

Generic/hardware properties, configs and their corresponding sets can also be individually assigned to a host. Such assigned properties take precedence over ones of the same type selected through the Host or Global Template. This is useful when a particular (or a few) node(s) require a special property/config (set) while everything else should be set the same as for other hosts with the same template.

If per-host HW Property Sets was chosen in the previous page, the set to be used for each group must be picked here. The Wizard will try to find an existing HW Property Set that already contains the correct Hardware Properties for each group. If such a set is found, it will be pre-selected. Otherwise, the only option is to generate a new set, for which a name must be entered, before it’s possible to continue.

Various configurations managed in the QluMan GUI via Config Classes and sub-classes translate into automatically generated configuration files for the individual hosts. In the pre-systemd phase of their boot process, these files will be sent to them and written by their execd. At this stage, there is no general update mechanism concerning these files for running nodes and changes only take effect during the next boot. A preview of the generated configs can be initiated by selecting Preview config from the host’s context-menu. More than one host may be selected for this.

The Filesystem Exports dialog shows the list of exported filesystems by name and FS type. Selecting an entry will show the details for this FS export on the right. A new Filesystem Exports resource can be added by clicking the New button. This requires choosing a unique name that will be used inside QluMan to identify the resource. The Resource Name field will turn green if the entered name is unique. QluMan currently supports three types of network filesystems: NFS, Lustre and BeeGFS. The FS type of the resource can be selected from the drop-down menu.

Next the server exporting the FS has to be selected. The default is beosrv-c, the cluster-internal hostname of the head-node, as the most likely server to export a FS. Using the drop-down menu, the server can be selected from a list of servers already used for other exports. To use a new server, the name has to be entered manually. It can be any hostname known to Qluman. The Server label will turn green if the entered name is a known host. This includes all nodes configured in the Enclosure View and any cluster-external host defined in Manage Cluster  Global Settings  Network Settings.

The remaining options depend on the selected FS type. In case of NFS, the path of the FS to be exported on the server has to be entered. Because the path will later be used in a systemd mount unit file, there are some restrictions on the syntax. For example the path must start with a "/" and must not have a trailing "/". The Path label will turn green if the entered path is acceptable, otherwise it will turn red.

For a Lustre resource, the Lustre FS name has to be specified. Lustre limits this name to eight characters and again, to avoid complications in the systemd mount unit file later, only alphanumeric characters and some punctuation will be accepted.

In the case of BeeGFS, you have the option to define the TCP and UDP ports on which the management server listens for this FS resource. If the management server manages just one BeeGFS FS, the default ports are usually fine.

Once all fields are entered correctly, the OK button will be enabled and the export definition can be added. It will then appear in the Filesystem Exports window.

Qluman automatically adds the networks available on the selected server(s) to the Network priorities. Later, when a node boots and requests its config files from the head-node, the networks available on the client are checked against this list and the first common entry is used for the network path via which the FS will be mounted. Shown entries can be removed or additional networks added from the context-menu. Entries can also be moved up or down using drag&drop. This is useful e.g. to ensure that an NFS export is mounted via Infiniband/RDMA on all hosts that are connected to the IB fabric and via Ethernet on nodes without IB.

Once Filesystem Exports have been defined, they can be used to configure Network FS Mounts configs. Each config is a collection of filesystems mounts combined with their mount options. As usual, such a config can be assigned to hosts either directly or indirectly through a template. Only one Network FS Mounts config can be assigned per host, so all mounts that should be available on the booted node must be added to it. Click the New Config button to create a new Network FSMounts config.

A newly created Network FS Mounts config will be automatically selected for viewing and editing. Previously defined configs may be selected from the drop-down menu in the top left. Below that, the list of mountpoints for the selected config is shown along with the FS type for each mount. Selecting one of the mountpoints will show its configuration details on the right.

A mount definition can be deleted or a new one added to the config from the context-menu. To define a new one, enter the path where the FS should be mounted in the Mount Filesystem dialog. Also select one of the Filesystem Exports resources declared earlier from the drop-down menu. In most cases this information is already sufficient. The next time when a node assigned to this Network FS Mounts config boots, it will mount this FS.

A bind mount can be added in a similar way. However, instead of selecting an external resource to be mounted, the source path of the bind has to be specified. QluMan is unable to verify the existence of the specified path, so it is worth to double check before adding the bind mount config.

The name of a Network FS Mounts config can be changed by clicking the Rename button. Sometimes one wants to mount an additional Network FS on a particular node. But since each host can only ever have a single Network FS Mounts config, a new config must be created for this host even if all other required mounts are the same as on other nodes. To simplify this, the Clone button can be clicked to create a clone of an existing config. The new config can then be edited to include the additional Network FS.

Qluman timer configs are an additive Config Class. This means that more than one config can be assigned to a host through one of the standard methods (template/direct assignment) as long as it’s the same method. Example: Assigning a timer config to a host directly, negates any timer config defined in the host template. However, two configs may be directly assigned.

A new config can be created by clicking the New button and entering a name for the new config. An existing config can also be cloned to create a new one with copies of the timers of that config. This way they can be modified without having to recreate them all.

In general, QluMan configs may only be be deleted when they are not in use. Therefore, the Delete button will only be available if that is the case. If a config is in use, the bottom left shows the corresponding config set or hosts to which it is assigned.

A config can be selected from the drop-down menu. Once selected, the timers in that config are listed at the left. The properties of the selected timer are shown at the right and can be edited. Changes made to the timer will be highlighted in green if valid, yellow if incomplete and red if invalid. To save the changes click the Save button.

Once a config is created, timers can be added to it by clicking the New Timer button at the bottom. This will reset the input form on the right to allow entering the name of a new timer and set its properties. Click the Create button once the desired settings are made.

The time at which the command will be executed can be edited directly by typing in a date and time. When moving the cursor to any part of the date or time, the up and down cursor keys can be used to increment or decrement the selected field. The date may also be selected from a calendar by opening the drop down menu.

A timer can have a duration by checking the until checkmark and setting a time at which the timer will end. This allows to define a command at the bottom which should run at the end of the duration entered. Usage example: Run a ZFS pool scrub for a limited time every night, then pause the scrub and continue it the next night.

Besides the start and end time, a timer can also be triggered when a node boots. This allows the command to be executed on nodes that where down when a timer was supposed to be started. By selecting At boot the command will be executed during the boot process before systemd is started. At that time no services will be running and no filesystems will be mounted besides the system itself. The other option is selecting After Boot to run the command at the end of the boot process when all services are up and all filesystems are mounted.

A timer can be set to repeat hourly, daily or weekly. In the simple case of a timer with just a start time, this does what is expected: An hourly repeating timer e.g. will execute the given command at the start time and again an hour later etc. as long as the node is up.

Things get slightly more complex for repeating timers with a duration. There are two cases to distinguish, best explained with examples.

Case 1: The duration is less than the interval at which the timer will repeat as shown in the screenshot on the left. In this example, the timer is set to run for one hour on September 28th and repeat daily. This means, it will run again for one hour on the September 29th starting at 15:00h and the same every following day. The command and end command will be executed each day at 15:00h and 16:00h respectively.

Case 2: The duration is longer than the repeating interval. In the example with the screenshot on the right, the timer is set to start at 15:00h on September 28th to be run daily until it ends at 15:00h on October 28th. Here the command will be executed every day at 15:00h but the end command will only be executed once on the final day.

The commands associated with a timer are executed using the system shell. As such it can be a simple command or a shell script. Short commands can be entered directly in the mask. For longer commands and complex scripts click the Edit button at the right to open a larger editor window. The same applies to the end command.

When a timer config is unassigned from a node, one intuitively expects that all timers of this config will be deleted on the node, once the Qluman Timers section in the Write Files dialog is written. However, this is not the case due to the intrinsic nature of the Write Files mechanism. To delete the timers on this node, you should create an empty timer config (possibly called Cleanup for later reuse) and assign it to the node. After that, the explicit write of the Cleanup timer config in Write Files will remove the timer on the node. You can double-check this by inspecting the local timer directory /var/lib/qluman/monitoring on the node. The assignment of the Cleanup timer config can be removed afterwards.

Besides configuring the remote console and power management for hosts, a Remote Management config can also control their BIOS settings if the BMC has Redfish support. When a new config with Redfish support is created, the BIOS settings portion of the config is initially empty. The specific config required for a host is highly hardware specific and impossible for QluMan to create on its own. Instead, the BIOS settings may be read from a host that was configured manually using the vendor provided interface and can then be used to configure other nodes with the same hardware in the cluster.

To read the BIOS settings from a host, it has to be specified first. This can be done by selecting a host in the Enclosure View and then clicking the Select button in the Remote Management Configs dialog. Alternatively the name of the host can be entered directly.

Once a host has been specified, its BIOS settings can be loaded from the BMC by clicking the Read button. On success, this will add the settings of the host to the active Remote Management config and display them in a table as key-value pairs.

BIOS settings are classified into three categories reflected in the check column. Some settings are specific to the vendor and hardware model. They can be used to identify the hardware type and make sure the configured BIOS settings are not written to a host with different hardware. Such entries need to be marked with a checkmark in the check column.

The second class of settings are purely informational and provide information about the actual hardware detected in the system, e.g. the amount of memory. These settings can not be written to and attempting to change them will result in errors. They may be marked with a dash in the check column indicating that QluMan should check that the value matches what the host reports. Often though, these settings are meaningless for the user and just clutter up the config. Unwanted entries can therefore be selected and then be removed from the config by clicking the Remove button.

Once BIOS Settings are as desired, they can be tested by writing them to a host using the Write button. As a first test, the settings should be written back to the host they were read from. This will check whether writing BIOS settings is allowed and working at all. As a second test, the settings should be written to a different host, preferably one with different BIOS settings. This will test if modification of BIOS settings works.

When a Remote Management Config with Redfish support and BIOS Settings is assigned to a host, QluMan can probe the host during boot and validate or even update its BIOS settings. By default, it will read them out at boot time, compare them to the configured settings and then keep on booting. Even a failure during read out is ignored. This should be harmless in all cases.

The action taken during boot can be configured by assigning the generic property BIOS Settings to the host via one of the known methods. Then one of three actions can be chosen.

Currently available image versions are 11, 11.0 (all meta-versions) and 11.0.0. Note, that selecting meta-versions (like e.g. 11) has implications on the update process. They allow tracking the newest x.y (x.y.z) releases automatically. Example: If you have installed version 11 of the modules, you will currently get the 11.0 (most recent 11.y) versions, but if a 11.1 would become available, apt-get dist-upgrade will update to 11.1 versions automatically. So with this choice, updates will usually include larger changes, since new feature releases (like 11.1) will automatically be installed.

Similarly, if you have selected the 11.0 version (currently default after a fresh installation) you will currently get 11.0.0 (most recent 11.0.z version) and apt-get dist-upgrade will update the modules/images to 11.0.1 automatically once available. So this choice will update to new maintenance releases automatically. The most conservative choice would be to explicitly select a x.y.z version (currently 11.0.0), since then images will only receive bug fix updates without explicitly changing the version in Qlustar. See also the discussion in the general Qlustar Update Guide

A couple of images are pre-defined during the installation process. The dialog shows the images sorted by their names. Expanding an entry shows its configuration and allows to select a UnionFS chroot via the drop-down menu. Each image contains at least the core module. Additional modules can be added or removed using the context menu when hovering over an entry. Only modules that are not already chosen are available for selection.

New images can be added through the context menu or by pressing the New button at the bottom of the dialog. Like before, you should then enter the name for the new config, choose a UnionFS chroot and optionally provide a description for the new image. Existing images can be removed via the context menu.

After the Execute button has been pressed, all hosts will start in the Pending state. Once a host confirms that it has started its command, it will change to the Running state. When the command concludes, the state becomes one of Failed, Errors or Success. If the command exited with a return code other than 0, the host will enter the Failed state. If the command exited with a return code of 0, but produced output on stderr, it will enter the Errors state. Otherwise, it enters the Success state.

In the screenshot example, the hosts beo-201 and beo-202 were down, so they remained in the Pending state. By clicking the Pending button, a hostlist of the pending hosts is displayed. The QluMan server will start the command on those hosts, when they become online again. If you do not want that to happen, or if the command does not terminate on its own, then the Kill button allows you to stop the command. A killed command counts as failed, so beo-201 and beo-202 now enter that state. The command output also reflects, that the command was killed.

Hosts executing a command are not only grouped by their execution state, the command output produced by the different hosts is also analyzed and compared to each other. Hosts with identical output are put into a group. Their output is only displayed once, prefixed with the hostlist representing the hosts in each group. For a quick overview, the number of hosts and groups is also displayed below each state button.

In the screenshot example, two hosts (beo-201 and beo-202) have failed, because they where offline and the command was killed before starting. The output of both was identical, so they form one group. Similar, one host (beosrv-c) completed the command successfully and builds its own group.

The S buttons next to the numbers add or remove the hosts in each state to form a new hostlist for the next command. Press the button to include the corresponding hosts and press it once more to exclude them again. This is convenient, e.g. to quickly select only the hosts for which a command failed: Analyze the errors and later relaunch with an adjusted command. Another example: Select only the successful hosts to run a follow-up command etc.

Commands usually output regular text to stdout and warnings as well as errors to stderr. In the latter case, the command ends up in the Errors state, because this is usually something that needs further inspection. The screenshot example prints two lines, one to stderr and one to stdout. Unfortunately Unix does not enforce any order between output to stdout and stderr. Therefore, as in this example, it can happen, that a small delay between the command output and reading from the file descriptors causes the order to slightly change.

Some commands produce a lot of output. Error messages are then easily overseen in between the lines. Similarly a command might report a lot of harmless errors, that hide the interesting output going to stdout. To simplify an analysis of the command output for such cases, the two buttons stdout and stderr at the bottom of the window allow toggling the visibility of stdout and stderr output selectively on and off.

Searching in the command output is possible by entering text in the search field at the bottom of the execution window. The text is interpreted as a regular expression and a match is searched in the text starting at the current position in the output. If a match is found, the matching text is highlighted. The up/down buttons next to the search field allow skipping to the previous/next match respectively. If no match is found, the Search label will turn red.

Commands are kept in a tree structure, grouping similar commands together. They can be sorted freely using drag&drop. You may select one ore more commands or groups and drag them where they should appear. Items are moved if they remain inside the same top level item (cluster or user) otherwise they are copied, since the two are stored in different places. Groups can be created, renamed, moved and deleted to achieve any desired hierarchy of commands.

When a cluster is first installed, a number of pre-defined commands are added to the cluster automatically. A few more default commands can be added from the context menu by selecting Add default command and following the tree structure there. This also includes all the commands added at installation time. So in case one of these was deleted, it can always be restored again.

To define a new command, select New Command from the context menu and set its name. The new command will be created in the group, where the context menu was opened or in the root, if the mouse is outside of any group. Initially, the command will have no definitions.

To edit a command, it needs to be selected first. Then its definitions will be shown on the right. The name/alias of a command can be edited by clicking in the text box at the top and entering the new name. Press return, to save the new name and the check-box will become fully checked again. To undo editing, simply re select the command in the tree view.

A command can be executed on any host or set of hosts in the cluster. The Execute on field governs how that host or set of hosts is constructed. The default is User input. This means, the user will have to choose the hostlist, where the command will run, at the time, when it will be executed. Alternatively, the hostlist of the command can be preset by selecting one of the filters from the dropdown menu. If a filter is selected, the hostlist, it currently evaluates to, is displayed below it.

Editing the command itself may take a while. To avoid conflicts from concurrent editing attempts by different QluMan users, only one person can edit a command at a time. To start the editing process, click the Edit button at the bottom. After that, changes to the command can be entered. Commands will be interpreted/executed by the BASH shell on every host matching the hostlist. The full bash syntax is supported. Redirection of output to files and working with variables works as expected. Please refer to the bash documentation (e.g. man bash) for more details. There is one exception to this: A "%" character followed by a number specifies additional arguments for the command, as explained in more detail below.

Sometimes it is necessary, to pass some input to a pre-defined command. This can be done by clicking the Input check-box. It will bring up an input text-box, where the desired input text can be entered.

To finish editing the command, click the Save button at the bottom. This actually saves the command text and input, if any, in the database and releases the lock on the command. This also scans the command text for argument placeholders and updates the entries in the Arguments box.

The definition of command arguments use the same mechanism as detailed for the Execute on definition. They can either be left up to the user, to be filled in when the command is executed or be specified by a filter selectable from the drop-down menu. When executed, the <num> placeholders in the command text are replaced by the user specified arguments or the resulting hostlist of the filter. There are always as many arguments as there are placeholders in the command. To add an argument, edit the command text and add a placeholder there. To remove an argument, edit the command text and remove the placeholder.

In the screenshot example, the test command is defined to execute on all head-nodes (beosrv-c is the only head node in the cluster). It has some input and two extra arguments. The first one is fixed to the ONLINE NODES filter that evaluates to any host reported as online. The second one is left for the user to be specified, hence, when executing the command, only the second argument is editable. In the screenshot, the HEADNODES filter was chosen for this argument, but any other text would have been possible too. For easy verification, the command text, with all the arguments substituted, is shown together with the command input (if defined).

Editing a filter is similar to creating a new one. First select the filter from the drop-down menu to display it’s current definition. Then add, edit or remove individual filters as desired. Finally click Save as to save the altered filter, Using an existing name will replace the old filter. Using a different name will create a new filter.

A filter can be added from the context menu (right mouse click) in the top area. For a host to show up in the filtered list (bottom part), it must pass all the filters added. Each filter may narrow down the list. Any number of filters can be added and they do not have to be unique. For example you can add a Hostname filter that selects all hosts that begin with beo and a Host Template filter that selects all Demo VM nodes. A host has to pass all top-level filters to show up. Currently, QluMan provides six top-level filters: Hostname, HostTemplate, Enclosure, HEADNODE, HEADNODES and ONLINE. Additional ones will be added in the future.

Every filter, sub-filter and pattern can be inverted through the context menu. The context menu for a pattern contains menu entries for both, the pattern and the enclosing filter separated by a line. The first Invert entry will invert the specific pattern that was selected, while the second Invert will invert the whole filter.

Besides the obvious, this can also be useful in finding hosts that are not configured correctly. For example, adding an empty Host Template filter and inverting it, will show all hosts without a Host Template. Adding a second filter, that selects all switches, power controllers and other special devices (they usually don’t need a Host Template) and also inverting that, results in a list of all hosts, that are neither properly configured nodes (missing Host Template) nor special devices.

When constructing a filter, it is important to remember, that all top-level filters are subtractive. A host must pass all top-level filters to show up in the result. On the other hand, all patterns and sub-filters are additive. Matching any one of them within a top-level filter adds the host to the result of that filter. Hence, when subtractive behavior is desired for patterns or sub-filters, each pattern or sub-filter must be added to its own top-level filter. For example, to select all hosts that start with beo as well as end on "1", two Hostname filters have to be added.

To create a new user, click New User and enter the name for the

new user to create it. Then select the user from the drop-down menu and fill out the remaining fields. The changes will be saved automatically when Enter is pressed or the input field looses the focus. New users have no roles assigned to them and will have no rights to change something. They can only inspect the cluster config (read-only mode). See Managing User Roles/Permissions for how to create new roles and assign them to the user by checking the respective check-boxes. If the

New User button is not selectable, then the user lacks sufficient rights to create new users. The Roles buttons will then also be disabled, preventing an unauthorized user from giving himself or others extra roles.

A new user also lacks login credentials, so initially, he can’t connect to QluMan. Hence, the next step is to generate a one-time token for the user, by clicking New Auth Token. Generating the one-time token may take a little time to finish and happens before the New Auth Token dialog opens. The dialog shows a certificate containing the

generated one-time token, as well as the other login information required to connect to the server. The certificate is protected by an auto-generated 8 digit pin, so that it can be transferred over unencrypted communication channels like e-mail or chat programs. In such a case, the pin should be sent over a second, different, communication channel, e.g. reading it over the phone.

As a special case, when a user clicks New Auth Token for himself, the generated token is imported into his running client and replaces the current login credentials. A reconnect of the GUI client is then triggered automatically. It forces the client to generate a new random public/private key pair and use the new one-time token to authenticate itself to the server. This procedure should be used to invalidate the old keys and replace them with fresh ones, in case a user suspects the certificate safe might have been compromised by an attacker.

The New Auth Token dialog also has 3 useful buttons at the right bottom corner. The Import button allows adding the certificate directly to the running client. The use case for this is when creating a user account for oneself when working as admin. It is recommended, that for clusters with multiple users having the admin role, that every user has his own user account and the admin user is only used to initially create the new users.

The Save button allows saving the certificate into a file and the Mail button sends the certificate to the email configured for the user. In both cases, only the certificate is saved or mailed and the password needs to be send separately.

For optimal security, it is recommended to leave a new user without roles, until he has logged in using the one-time token. That way, if the certificate was intercepted, it will be useless to an attacker, since he won’t be able to perform any actions within QluMan. Also, if the attacker manages to intercept and use the certificate before the real intended user does, the real user won’t be able to use it anymore, and notice that something is wrong, most likely reporting to the main cluster administrator.

A user other than admin can be deleted by clicking the Delete User button. Just like the New User button, it is only enabled if the current user has sufficient rights.

A filter can be constructed as a positive or negative filter. This means it can hide all messages that are not specifically matched or show all messages that are not specifically chosen as hidden. The default action can be chosen at the bottom left corner of the message filter window.

A new filter row can be added by selecting Add filter from the context menu. The new filter has an action of hide and ignores all fields. It therefore hides all messages. To be useful, at least one column should be changed through the context menu, to match only some messages. The context menu in each column contains the possible values the filter can match against in that column. The Origin and Short columns can also be edited freely by double clicking them. The action for the row can be changed between Hide and Show.

The most common filter is to hide messages with the Seen flag. It is recommended, to always start a new filter by adding a row with action Hide and the seen column set to Seen. If none of the filter rows match against the Seen flag, then it will have no effect in the Messages window. The Seen filter can also be toggled between Seen and Unseen by clicking the checkmark. The column can only be disabled by selecting Ignore from the context menu.

Messages can be purely informational, warnings or errors. Informational messages include information about nodes coming online or the server being restarted. There are usually a lot of informational messages and they can be safely ignored. On the other hand, warnings and errors should be inspected more carefully. In the Log Viewer, the priority of a message is color-coded for quicker visual recognition. Informational messages are green, warnings yellow and errors red. The highest priority of any shown message is also shown in the Messages button in the lower right corner of the main window. This indicates at a single glance, if anything important happened recently.

The origin of a message shows the node or service that generated the message. When configuring the filter, the origin can also be expressed as a hostlist to match multiple hosts.

Messages fall into different categories, pooling similar messages for easier filtering. Generally information is categorized under Misc, while messages about nodes becoming online or going offline under category Online. The Licensing category includes all messages concerning changes in the license status. This could be something simple as a reminder that the license key expires soon. Or more important, a warning or error, that the cluster, as currently configured, exceeds the available license count. The last category is Exception. It usually signals a software error, that should be reported.

Messages may also be filtered by their short description. Like Origin, this column can be edited by double clicking. Short descriptions are matched using standard regular expressions. To match only part of a short description, prefix and/or suffix the text by ".*" to match any remaining characters.

The example filter shows a more involved setup: It contains five rows showing how rows can be combined to achieve the desired filtering result. The default action for this filter is set to show messages. Hence, only messages that are explicitly filtered as not wanted will be hidden.

Row 1 excludes messages with the seen flag set. Rows number 2 and 3 might look odd at first, because their action is the same as the default action: Show. But these two rows prevent any of the later rows from hiding messages with priority error or warning. In other words, warnings and errors will always be shown, no matter what additional filter rows follow. Row number 4 hides messages in the category online and row 5 hides messages that originate from hosts matching the hostlist "vm-[0-9]".

The LDAP User Management can be opened from the main menu Components  LDAP  User Management. It shows a list of all LDAP Users on the left side and information about the currently selected user on the right.

Click the New button to create a new user. The right side of the window then allows to input the necessary data for creation. Some fields are pre-set with default values. The label before each field indicates if the input is valid: It will be green if the current value is acceptable, yellow if it’s possible to complete it to an acceptable value and red for invalid input. The first field to be filled is the User Name. Changing it will also change the Home Directory field. The latter is pre-set using the Homedir Prefix with the username appended. Once all fields are green, the Create button will also turn green, indicating that it can be clicked to create the new user.

Once a new user has been created, its random initial password is shown in a popup window. The user will then appear in LDAP immediately and be usable on any node with the standard Qlustar Nameservice config. However, this is not the default setting for a node to avoid overloading the LDAP server with simultaneous requests from many compute nodes on large clusters. The default setup is to create a passwd and group file from the LDAP data which are sent to nodes during the early boot process. This way user/group data are accessible via local lookups leaving the server mostly idle. Click the Sync now button in the popup to update the passwd and group file on any node using the default synchronization method.

The user information has 2 optional parts.

The information for an existing user can be edited by first selecting the user on the left side and then modifying the relevant field. The label is color-coded again to show the validity of the input. The Save button must be clicked to save the changes. To undo them without saving, just select a different user or close the window.

A user can be added to or removed from a group by opening the context-menu in the Groups box. Selecting an entry will add or remove the user from the corresponding group. A user’s password can be reset by selecting the user on the left side and clicking the Reset Password button. A window with the new password will then come up.

Once the cluster has many users, it can become hard to find a particular one in the user list. To simplify this, the list can be filtered as follows: Select one of the available filters from the Filter drop-down menu and enter a regular expression in the text field next to it. The user list then updates as you type, showing only users that match the regular expression for the selected filter.

To delete users, select them on the left and click the Delete button.

The LDAP Group Management dialog can be opened from the main menu Components  LDAP  Group Management.

It allows creation and removal of groups as well as the management of group membership. The corresponding window is split into three parts: The left part shows a list of all LDAP groups, the middle displays details for the selected group, including a list of its members and the right has a list of users.

The list of groups can be sorted by either the group ID or the group name in ascending or descending order by clicking on the column name. It can also be filtered by specifying a regular expression for the group name in the Filter text field at the bottom. Groups can be removed in batches by selecting them in the list and clicking the Remove button.

Selecting a group on the left will display its details in the middle: At the top the group name and group ID and the description of the group underneath. At the bottom is the list of users that are members of the group. Users that have this group as their primary group are shown in bold. The context menu for the Members box allows modifying the membership of the selected users in four ways:

If a group is selected with its details shown in the middle, the right side lists users that are not members of the group. This list supports filtering to simplify finding users, just like in the LDAP User Management.

Selected users in this list can be added to the group shown in the middle by dragging them into the Members box. This also works in the other direction: Selecting users in the Members box and dragging them into the All Users box will remove them from the group, but only if this group is not their primary group.

A new LDAP group can be created by clicking the New button. A group name must then be entered. The group ID can be changed from the suggested one and a description for the group can be added in the box below the name. The labels are color-coded to show valid inputs: Green for valid inputs, yellow for partial ones and red when the group ID is already in use. If all inputs are valid, the Create button will turn green and clicking it will create the new group.

The description of the group can be edited by clicking in its box. When it is changed, the Undo and Save buttons become active. Click the Save button to save it.

The currently displayed group can be deleted by clicking the Remove button. Multiple groups can be selected and removed in batches.

Two methods for importing users are supported: a) Import of users, passwords and groups from flat files in the standard UNIX formats. b) QluMan may be configured to allow for a single sign-on setup where users can be imported from and authenticated against an Active Directory (AD) server. In such a setup, the Qlustar LDAP server uses SASL pass-through authentification against AD passwords instead of storing password hashes.

The LDAP Configuration dialog can be opened from the main menu Components  LDAP  Config. It handles a few settings relevant to the user and group management in qluman-qt.

In QluMan, the user ID for LDAP users is limited to a customizable range of IDs. By convention, user IDs below 1000 (200 for CentOS) are reserved for system accounts and daemons. Per default, QluMan uses 2000 as the minimum value for UIDs. On the other hand, there is no well-defined convention for the maximum.

In some situations, it is convenient to assign certain ranges of IDs to certain user groups. Changing the Min and Max values before user creation then helps by making qluman-qt suggest UIDs only within that range for new users.

The Next value for User ID is the UID qluman-qt will suggest the next time a user is created. Once created, the Next value is incremented to the next unused UID. When it hits the Max User ID, it will be reset to Min User ID and start all over. The reason to explicitly keep track of this instead of always using the smallest free UID is to avoid reusing UIDs of deleted users. If a UID is reused, there is always a risk of loosing data protection: Left-over files on the system belonging to the old user would be accessible by the new user. The same mechanism is implemented for the Min, Next and Max settings of Group ID.

The Default Group, Default Mode and Homedir Prefix can also be set here. They are used to fill the corresponding fields with default values when creating a new user. The homedir is constructed from the Homedir Prefix and the username of the new user.

Finally, a list of cluster nodes can be entered to set Homedir Hosts. The default setting is just beosrv-c, the head-node. A change is only needed if the home directories cannot be created on the head-node. This can be the case if home directories are on an external NFS server or a Lustre filesystem for example. If a list of hosts is entered here, they will be checked for accessibility one by one and the home directory created on the first one accessible.

A hostlist can either be entered directly or hosts can be selected in the Enclosure View and the Selected Hosts button be clicked to enter a hostlist of the selected hosts.

The QluMan Slurm Component provides extensive functionality to manage and operate most aspects and features of the Slurm workload manager. All QluMan Slurm functionality is accessible underneath the Components:  Slurm top-level menu entry.

The following management and operation sub-components are available:

The Slurm Overview window provides a summary of the utilization of the cluster. It is split into 2 parts: The Cluster Usage Overview tab and the Job Overview tab.

The Job Management window shows a continuously updated table with all current jobs of the cluster. Since a single job has about 100 properties, every QluMan user is able to customize the job properties he wants to be displayed and which ones should be hidden in the table (see Customize Columns for more detailed information).

To sort the job table entries, one just has to click on the title of the property one wants to sort for (for example Job Id). Clicking the title again reverts the sort order.

These settings can be stored in layouts. Just modify the Job Management the way you want it and hit the Save Button. You can restore a layout by selecting it in the Layout combo box and press Load. When the Job Management gets opened it always uses the last state as layout. This is the layout that was set when you closed the Job Management the last time.

If you want to change the state of a job you just have to open its context-menu and select one of the following actions:

Depending on the state of a selected job some actions might be disabled (e.g. a job cannot be released if it wasn’t on hold before). As long as there is no conflict concerning their job states, it is possible to collectively manipulate either a list of jobs selected with the mouse or all jobs of the user of the currently selected job. If you want to get more information about a job, open the context-menu and select More Information (see More Job Information for details).

The number of jobs shown in the Job Management window can be overwhelming. There are three ways to reduce the information overload for a cluster with many jobs:

The Node State Management dialog lists all hosts that are registered with Slurm. There are three different kind of views showing the existing hosts. The color of the LED in front of the hostname indicates the Slurm state a node is in. When hovering over a particular node, a tool-tip describing the state appears.

To manage one or more nodes, they have to be selected first. Use the preferred view and move the node(s) to the right list via drag&drop. One can also move a whole group of nodes, for example all nodes from a rack by dragging the name of the rack to the right tree. All nodes in this list are available for later actions. You

can also select multiple nodes for drag&drop or enter a hostlist in the Hostlist field (e.g. beo-[01-04]). The nodes will appear in the right list, if the hostlist is valid.

There are nine possible actions that may be applied to the selected nodes:

Once the desired nodes are selected, an action can be chosen and then executed by clicking the Execute button. In case the action was operating on the nodes slurmd, an RXengine window comes up, in which one can track the success of the remote slurmd operation. To clear the complete list of selected nodes, one can click the Clear button. To remove only a subset of nodes, one can select them in the right list and remove them via the context-menu.

The Slurm Reservations window shows a table of all active reservations and their most important properties. Furthermore, it allows to manipulate the reservations and create new ones.

The Manage Users dialog allows to assign accounts to a user, set and change a user’s default account, register new users and delete users. When a user is selected, the accounts he is a member of are checked in the Accounts list displayed at the right. His default account is highlighted with the specific color set for default accounts in the Preferences Dialog. By default, system users are hidden. To show them, just check the Show system users (UID < 1000) checkbox.

The Quality of Service (QoS) associated with a job will affect it in three ways:

The QluMan Manage QOS dialog allows the creation or removal of a QoS and to define all its properties. Once created, a QoS can be used in slurm partitions or assigned to accounts and users to determine the QoS their jobs will run with.

The list of existing QoS is shown on the left side of the dialog. Selecting one of them will show its properties on the right hand side and allows to configure them.

A new QoS can be created in two ways: Either by clicking the New button to create a blank QoS with just the default properties, or by clicking the Clone button to create a copy of the selected QoS with a new name. In both cases, a name for the QoS has to be entered. Clicking the Ok button will then create it.

When a QoS is selected the Used by label at the bottom will list all the entities it is currently assigned to. This can be another QoS with the Preempt property, a partition definition or an account/user association. If a QoS is not in use, it can be deleted by clicking the Delete button.

Slurm may impose limits and defaults on the jobs users are allowed to submit to slurm via the concept of associations. This can be used e.g. to limit the number of CPU cores a user can allocate or to set a priority for jobs submitted using a certain slurm account.

Association limits set for a particular slurm account are inherited by all its sub-accounts (if any) and users of that account forming a tree-based hierarchy. For each user, QluMan attempts to detect properties with identical values across all accounts the user is a member of. It will then filter out these properties from the account associations shown at the top of the window and display them in the User Associations area at the window’s lower part instead. This idea reduces the number of entries shown and tries to highlight relevant information.

The settings of account associations are shown in tree-form starting with the system-wide root account at the top of the tree (shown as an asterisk, *). Account associations have an asterisk in the User column indicating that the setting affects all users of the account. Otherwise the pair of account and user is shown. The remaining columns show the active settings of the association. Values that are inherited from the parent association are shown in gray.

For the sake of clarity, per default, entries for accounts and users are only shown if they have at least one setting that is not inherited from its parent. Furthermore, user association settings listed in the User Associations area are also not displayed in the Account Associations area per default. On the other hand, the parents of any listed entry is also shown.

There are two options to list the full association tree in the Account Associations area: Checking the All Accounts check-box will display all account associations, even if they only have inherited values. Similarly, checking the All Users check-box will do the same for user associations.

The Cluster Usage display uses the Slurm utility sreport to generate reports of job usage and cluster utilization. For detailed information about the type of reports and options read its manpage on your head-node. Select your report type (for example cluster) in the left combo box and then the report options from the combobox right to it. Per default the time period used for the report is the past day. You can change this by modifying the start and the end time. The colors used in the window are customizable in the Preferences Dialog.

The Fair Share view uses the Slurm utility sshare to display Slurm fair-share information. We provide two versions of views, a basic and a long one. The long version shows additional information that is needed less often. By default we show the basic view, but you can easily switch by checking the long checkbox at the bottom right of the window.

The Job Priorities dialog uses the Slurm utility sprio to display the values of the individual factors that are used to calculate a job’s scheduling priority when the multi-factor priority plugin is installed. This is information needed, when analyzing why certain pending jobs run earlier than others.

We provide two versions of the view, a basic and a long one. The long version shows additional information that is needed less often. By default we show the basic view, but you can easily switch by checking the long checkbox at the bottom right of the window. For more information about sprio read its manpage on your head-node.

The QluMan Slurm Settings dialog allows to customize the update intervals for information about jobs, nodes, partitions and Slurm accounting. This information flow is provided by the QluMan Slurm daemon running on the cluster and the QluMan GUI automatically subscribes to it. Shorter update intervals mean more server load and more network traffic. In most cases, the default values should be adequate.

When you click on the Fonts icon, you’ll see a list of different font identifiers, for which you can change the font settings. The relevant identifiers affecting QluMan are: General, Menu and Window Title. Changing one of the values and clicking the Apply button changes the corresponding font on the fly.

Click on the Colors icon and choose the Colors tab. There you can adjust the color of the different elements of the QluMan GUI. You can narrow down the color identifiers to the ones affecting particular GUI elements, by choosing a specific color set with the corresponding pull-down menu. Changing one of the values and clicking the Apply button, changes the corresponding color on the fly.

If you’re using KDE4 on you’re desktop, instead of configuring using the System Settings dialog, you can also move /root/.kde/share/config to /root/.kde/share/config.bak and copy your personal configured .kde/share/config directory to /root/.kde/share. As long as you’re not using any non-standard KDE themes, this should just apply the favorite desktop settings you’re familiar with to QluMan, when running it on a remote machine like the cluster head- or FE-node (restart of QluMan GUI required).

Changing the widget style can be a little more involved. First you need to start the QT configurator qtconfig and choose a GUI style (default is QtCurve). The following assumes, you’re running qluman-qt on the head- or FE-node. In case you have it installed on your workstation, just execute qtconfig there.

When you’re done, select File  Save and you’ll already see the changes. After this, you can exit qtconfig. If you want further customization of the widget style (note that only some styles are configurable, among them QtCurve), you can now go back to the Application Appearance dialog (see above), click on the Style icon, choose the style you’ve selected in qtconfig as Widget style and press the Configure…​ button. You’ll then see a large number of options for

customization. When you’re satisfied with your modifications, press the OK button and finally the Apply button of the Style  System Settings window. Note, that you will see the resulting changes only after performing some actions (pressing a button, etc.) in the QluMan GUI.

For additional widget style variants apart from the default of QtCurve, you can install additional kde-style packages (.e.g kde-style-oxygen) on the machine, where you’re executing the QluMan GUI.

Depending on the KDE theme and settings, disabled texts, buttons, icons or menu entries are not rendered as grayed out. Instead they are rendered like normal text, buttons, icons and menu entries, but can not be selected with the mouse. This is rather confusing. A quick fix we found for this issue is editing ~/.kde/share/config/kdeglobals and removing/commenting out all lines of the form "ForegroundNormal=…​". After saving and restarting qluman-qt graying out should work.