The slurm configuration module comes in four parts:

Assignment of hosts to node groups and/or partitions is possible by adding the latter to the relevant Config Sets and Host Templates or by direct assignment through the config (set) context menu in the enclosure view.

The overall slurm configuration is split into two templates, the slurm config and cgroups.conf. On write, QluMan adds the NodeName and PartitionName lines at the end of the slurm config template to generate the slurm.conf file, while the cgroup.conf file gets written as is. For the syntax of both templates, please refer to the slurm documentation (e.g. man slurm.conf). To edit one of the templates, select it, click the Edit button and start making changes. Click Save to save the changes or Undo to discard them. Use the Preview button to check changes before writing them.

Slurm node properties are configured from two sources:

a) The slurm node groups. Every host can belong to at most one such group. The membership is assigned (see Assigning Hosts to Slurm Node Groups, Partitions and Gres Groups) by adding the desired node group to the Config Set that is assigned to the node via its Host Template or via the alternative ways to assign config classes.

+ Each Node Group is a collection of slurm node properties, that will be set for the members of the group. Per default, only the MemSpecLimit property is defined, but other properties like Feature or Weight can be added by using the Slurm Property Editor.

A new node group can be created by clicking the New Node Group button or selecting New Node Group from the context menu. This opens a dialog asking for the name of the new group. An existing node group can be renamed or deleted from the context menu.

The context menu also allows to add properties to a group. Note, that some properties are unique, i.e. only one value can be selected for the property. Adding a second value of the same property will automatically replace the old value in that case. Other properties are not unique. Adding multiple values to such properties results in a comma separated list of values in the

b) The slurm.conf file. An example for this is the Feature property. Properties can also be changed directly using the pull-down menu. If a change will cause a duplicate value, the previous (duplicate) value is automatically removed.

The management of Slurm partitions works exactly the same way as that of slurm node groups. Please see Slurm Node Groups for how to create, rename and change partitions.

The Slurm property editor for node or partition properties can be opened by clicking the Properties button at the bottom of the Slurm main dialog. If the Node Groups tab is selected, the editor for node properties will be opened. If the Partitions tab is selected, the editor for partition properties will be opened.

To add a new property, enter the name of the property in the name field. If the name does not already exist, the New Property button will be

enabled. Click on it to create the property. QluMan has a white-list of known valid properties, e.g. Weight and allows adding such a property without further questions. In this case, QluMan will also set the unique flag and add all known property values automatically.

When a property is created that is not part of the white-list (Gres in the screenshot) a dialog opens up, asking for confirmation. Note that adding an unknown property can lead to a failure when trying to restart slurm. Therefore make sure to only add properties you are certain slurm will know about. A property without values can be deleted by clicking the Delete button.

To add values to a property, first select the desired property using the pull-down menu from the name. Then enter the new property using Add Value at the bottom and finally press Enter to add it. To delete a value, select Delete value from the context menu.

Currently, Slurm Gres Groups are used in Qluman mainly to handle the setup of GPUs for slurm. The GPU Wizard is the most convenient and accurate way to create such resource groups. Supplementing the wizard, the Gres Groups tab allows creating and managing any type of resource group, as well as binding GPUs to specific CPU sets, which is not possible via the wizard. To view or modify a Gres Group, select the group from the drop down menu. Use the Preview button to check the resulting config file changes before writing them.

A new Gres Group can be created by clicking the New Gres Group button. This opens a dialog asking for the type, name and description of the new group. An existing type can be selected from the drop down menu or a new type can be

entered directly. After entering a new unique group name the OK button becomes selectable. A group that is not in use can be deleted by clicking Delete Group.

A Gres Group can have multiple entries. A new entry may be added to a group by clicking on New Entry. Initially, the entry is blank and at least the type column must be filled in. For resources that can be allocated in multiple pieces, a count can be set, indicating the number of resource

pieces available. For resources that have a device file associated with it, its path can be set in the file column.

Optionally, an entry can also be associated with a set of CPUs. The CPUs to be used can be entered as a comma-separated list or, for recurring sets, selected from the drop-down menu. An entry can be deleted from the group by selecting Delete Entry. A group that is no longer in use can be deleted by selecting Delete Group.

Hosts are assigned to Slurm Node/Gres Groups and Partitions by use of the global or a Host Template (through its corresponding Config Set) or by direct assignment, i.e. the standard four levels of configuration. A Config Set may contain at most one Node Group but any number of Gres Groups or Partitions,

since a host can be member of an arbitrary number of Slurm partitions. They can all be assigned by selecting them via Add Config in the context menu of a Config Set or via the Enclosure View context menu of the host(s).

The Slurm Overview window provides a summary of the utilization of the cluster. It is split into 2 parts: The Cluster Usage Overview tab and the Job Overview tab.

Cluster Usage Overview

The Cluster Usage Overview provides continuously updated information and charts about Node, CPU Core and Memory utilization by Slurm jobs. Every information field in the tables has a tool-tip that supplies more detailed information about it.

Job Overview

The Job Overview display consists of two tables and four charts being continuously updated. The Running table provides summary information about running jobs of users. It shows the color representing the user (if his share is displayed in one of the charts), his username, the count of utilized CPU cores, the number of used nodes and the number of running jobs. The Pending table provides the total number of requested CPU cores and the number of pending jobs for the same user.

The job statistics is graphically displayed in the four pie-charts Allocated CPU Cores by User, Used Nodes by User, Pending CPU Cores by User and Pending Jobs by User. Every slice of the pie-chart has a tool-tip showing the name of the user it corresponds to together with his share in percentage of the corresponding resource. The used colors change randomly with every new invocation of the window.

The Job Management window shows a continuously updated table with all current jobs of the cluster. Since a single job has about 100 properties, every QluMan user is able to customize the job properties he wants to be displayed and which ones should be hidden in the table (see Customize Columns for more detailed information).

To sort the job table entries, one just has to click on the title of the property one wants to sort for (for example Job Id). Clicking the title again changes the sort order. You can also move a column with drag and drop and change its width.

These settings can be stored in layouts. Just modify the Job Management the way you want it and hit the Save Button. You can restore a layout by selecting it in the Layout combo box and press Load. When the Job Management gets opened it always uses the last state as layout. This is the layout that was set when you closed the Job Management the last time.

If you want to change the state of a job you just have to open its context-menu and select one of the following actions:

Depending on the state of a selected job some actions might be disabled (e.g. a job cannot be released if it wasn’t on hold before). As long as there is no conflict concerning their job states, it is possible to collectively manipulate either a list of jobs selected with the mouse or all jobs of the user of the currently selected job. If you want to get more information about a job, open the context-menu and select More Information (see More Job Information for details).

Clicking on Activate Filter at the bottom of the window, allows to activate one or more custom filters (created using the Job Filter Editor) by checking the corresponding entry. This can be useful to restrict the list of displayed jobs according to some criteria (e.g. a certain user). All currently active filters are shown in the bottom left corner of the Job Management window. They can be deactivated again by unchecking their entry in the Activate Filter sub-window.

The Customize Columns dialog displays all known columns (properties of a job) in two lists. The columns in the left list will be shown in the jobs table, the ones in the right list won’t. To show or hide columns just select them and drag them either into the left or right list. Confirm your changes with OK.

This dialog opens after you select Get more information in the context-menu of a job. It shows the properties and their corresponding values of the selected job in a table. There are two filters that may be applied: One is for hiding all properties with a value of 0, None, False or empty, the other one for hiding exotic properties which one is rarely interested in. Per default, both filters are enabled. To disable them, you have to check the corresponding entry at the bottom of the dialog.

If you created some custom filters, they will be listed here (For information about creating custom filters see Job Filter Editor). Select one or more filters to be applied to the current job table. All active filters are shown as a comma-separated list in the bottom-left corner of the Job Management window.

As mentioned before, in the Job Filter Editor dialog it is possible to create custom filters for the Job Management table. After it has been opened, a new filter may be created by clicking New Filter and then insert a name for the filter. After confirming with OK the filter is created and a new window comes up, where properties can be assigned to it. To add properties, right-click for the context-menu and select the property you want to filter with.

In the current example, we chose to filter by Job Id. A new dialog pops up. Now one can select a range of job ids to be displayed. Since a job id is always an integer, one has the option to select among the filter types between x and y, bigger than x and less than x.

Choose the filter type you want, set the values and confirm with OK. Consequently, the property is now part of the new filter. One can combine multiple properties in one custom filter. Each additional property narrows down the possible jobs to be displayed. After adding all desired properties, hit the Save button. Now the new filter can be applied in the Job Management window.

The Node State Management dialog lists all hosts that are registered with Slurm. There are three different kind of views showing the existing hosts. The color of the LED in front of the hostname indicates the Slurm state a node is in. When hovering over a particular node, a tool-tip describing the state appears.

To manage one or more nodes, they have to be selected first. Use the preferred view and move the node(s) to the right list via drag&drop. One can also move a whole group of nodes, for example all nodes from a rack by dragging the name of the rack to the right tree. All nodes in this list are available for later actions. You

can also select multiple nodes for drag&drop or enter a hostlist in the Hostlist field (e.g. beo-[01-04]). The nodes will appear in the right list, if the hostlist is valid.

There are seven possible actions that may be applied to the selected nodes:

Once the desired nodes are selected, an action can be chosen and then executed by clicking the Execute button. In case the action was operating on the nodes slurmd, an RXengine window comes up, in which one can track the success of the remote slurmd operation. To clear the complete list of selected nodes, one can click the Clear button. To remove only a subset of nodes, one can select them in the right list and remove them via the context-menu.

The Slurm Reservations window shows a table of all active reservations and their most important properties. Furthermore, it allows to manipulate the reservations and create new ones.

Creating a new Reservation

To create a new reservation, click the Add reservation button. A new dialog pops up. The following parameters can be specified:

Confirm by clicking the Add reservation button.

Updating a Reservation

To update a reservation one just has to select it and open its context-menu. Choose Update Reservation. A window pops up with all the properties set to the values of the existing reservation. To modify the reservation just make the desired changes and click the Update Reservation button.

Deleting a Reservation

To delete a reservation one just has to choose Delete Reservation from its context-menu .

The Manage Users dialog allows to assign accounts to a user, set and change a user’s default account, register new users and delete users. When a user is selected, the accounts he is a member of are checked in the Accounts list displayed at the right. His default account is highlighted with the specific color set for default accounts in the Preferences Dialog. By default, system users are hidden. To show them, just check the Show system users (UID < 1000) checkbox.

Registering a User with Slurm

To register a user with Slurm, uncollapse the Unregistered Users and select the desired user. Every user needs a default account, so this has to be defined first. To do so, select Set as Default Account in the context-menu of the account you want to be the default. By doing this, the user will be registered with this default account. If you just select some accounts for an unregistered user by checking them and then pressing the

Create button, the user will be registered with a default account set randomly among the chosen ones.

Deleting a User

To delete a user, bring up his context-menu and select Remove User.

Assigning a User to Accounts

Selecting a registered Slurm user displays the accounts he is member of in the Accounts list to the right. To add/remove him to/from an account (un)check it and hit the Activate Changes button.

Changing the Default Account of a User

To change the default account of a user, select him in the Registered Slurm Users tree and bring up the context-menu of the account you want to set as the new default. Then select Set as Default Account.