Product SiteDocumentation Site

Qlustar Cluster OS 9.1

First Steps Guide

This document describes the first steps to perform after a fresh install of a Qlustar head-node.

Qlustar Documentation Team

Q-Leap Networks GmbH

Legal Notice

Copyright ©2017 Q-Leap Networks GmbH
This material may only be copied or distributed with explicit permission from Q-Leap Networks GmbH. The Qlustar license can be found at /usr/share/qlustar/LICENSE.html on an installed Qlustar head-node.


This document describes the first steps to perform after a fresh install of a Qlustar head-node.
1. Qlustar Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. Feedback requested
1. First Boot
1.1. Running qlustar-initial-config
1.2. Final Reboot
1.3. Starting the virtual Demo Cluster
1.4. Installed Services
1.5. Adding Software
1.6. Running the Cluster Manager QluMan
1.6.1. Generating a one time token for the first admin login
1.6.2. Starting the QluMan GUI
1.6.3. Installing the QluMan GUI on a workstation
1.7. Creating Users
1.8. Compiling an MPI program
1.9. Running an MPI Job
1.10. Running the Linpack benchmark
A. Revision History


1. Qlustar Document Conventions

Qlustar manuals use the following conventions to highlight certain words and phrases and draw attention to specific pieces of information.

1.1. Typographic Conventions

Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from an individual key by the plus sign that connects each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to a virtual terminal.
The first example highlights a particular key to press. The second example highlights a key combination: a set of three keys pressed simultaneously.
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog-box text; labeled buttons; check-box and radio-button labels; menu titles and submenu titles. For example:
Choose SystemPreferencesMouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose ApplicationsAccessoriesCharacter Map from the main menu bar. Next, choose SearchFind… from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose EditPaste from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh at a shell prompt. If the remote machine is and your username on that machine is john, type ssh
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release.
Note the words in bold italics above: username,, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.

1.2. Pull-quote Conventions

Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books        Desktop   documentation  drafts  mss    photos   stuff  svn
books_tests  Desktop1  downloads      images  notes  scripts  svgs
Commands to be executed on certain nodes of a cluster or the admins workstation are indicated by using descriptive shell prompts including user and hostname. Note that by default, the shell prompt on Qlustar nodes always ends in the newline character, thus commands are typed on the line following the prompt. As mentioned above, the command itself is shown in mono-spaced bold and the output of a command in mono-spaced roman. Examples:
0 root@cl-head ~ #
echo "I'm executed by root on a head-node"
I'm executed by root on a head-node
0 root@beo-01 ~ #
echo "I'm executed by root on a compute node"
I'm executed by root on a compute node
0 root@sn-1 ~ #
echo "I'm executed by root on a storage node"
I'm executed by root on a storage node
0 user@workstation ~ $ 
echo "I'm executed by user admin on the admins workstation"
I'm executed by user admin on the admins workstation
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:

import javax.naming.InitialContext;

public class ExClient
   public static void main(String args[]) 
       throws Exception
      InitialContext iniCtx = new InitialContext();
      Object         ref    = iniCtx.lookup("EchoBean");
      EchoHome       home   = (EchoHome) ref;
      Echo           echo   = home.create();

      System.out.println("Created Echo");

      System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));

1.3. Notes and Warnings

Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.


Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.


Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled “Important” will not cause data loss but may cause irritation and frustration.


Warnings should not be ignored. Ignoring warnings will most likely cause data loss.

2. Feedback requested

Contact to report errors or missing pieces in this documentation.

Chapter 1. First Boot

1.1. Running qlustar-initial-config

After the server has booted the newly installed Qlustar OS, log in as root and start the post-install configuration process by running the command
This will first thoroughly check your network connectivity and then complete the installation by executing the remaining configuration steps as detailed below. During the package update process you might be asked whether to keep locally modified configuration files. In this case always choose the option
keep the local version currently installed.

Remaining configuration steps run-through

If your chosen hostname can't be resolved via DNS, you will see a non-fatal error message reminding you that the hostname should be registered in some (external) name service (typically DNS).
  1. Cluster name

    First, you will be asked for the name of the new Qlustar cluster. This can be any string and is used in some places like the slurm or ganglia configuration.
  2. NIS Setup

    Next is the setup of the NIS database. Just confirm the suggested NIS server by pressing Ctrl-D+Enter to proceed.
  3. Configuring ssh

    An ssh key for the root user is generated next. You can enter an optional pass-phrase for it. This key, will be used to enable login by root to any node of the cluster without specifying a password.


    Be aware, that having a non-empty pass-phrase means, that you will have to specify it any time you try to ssh to another host in the cluster. If you don't want that, work without a pass-phrase.
  4. Configuring Nagios

    The configuration of Nagios requires you to choose a password for the Nagios admin account. Please type in the password twice.
  5. Configuring QluMan

    QluMan, the Qlustar management framework (see the QluMan Guide), requires a mysql (mariaDB) database. You will be asked for the password of the QluMan DB user next. After entering it, the QluMan database and configuration settings will be initialized. This can take a while, since a number of OS images and chroots (see Section 1.5, “Adding Software”) will be generated during this step.
  6. Configuring Slurm

    If slurm was selected as the cluster resource manager, its configuration requires the generation of a munge key and the specification of a password for the slurm mysql account. Enter the chosen password twice when asked for it.


    The slurm database daemon is also being configured by this process. Hence, you will be ready to take full advantage of the accounting features of slurm.
  7. Configuring the virtual Demo Cluster

    If you have chosen to setup some virtual demo nodes during installation, you will be asked for the user name of a test account that can be used to explore the cluster. The account will be generated with the default password for the cluster (see the information on the screen).
  8. Setting the MariaDB root password

    To conclude the configuration procedure, you will be asked to set the password for the MariaDB/MySQL root account. Setting a password here is important. It prevents unauthorized access to the Qlustar or other databases on your head-node.

1.2. Final Reboot

Please reboot again once all the previous steps are complete (e.g. by typing the command shutdown -r now ). After the head-node is up and running again, test its network connectivity by pinging its public IP address (hostname). Do the same for the virtual front-end node, if you have chosen to configure one. It should have booted as well after the head-node is up and running. You can try to login to it using ssh. A test mail should have been sent to the e-mail address(es) you specified during the installation. If you didn't receive one, review your settings in /etc/aliases and/or /etc/postfix/ In case some of them are wrong, you can execute
0 root@cl-head ~ #
dpkg-reconfigure postfix
to modify them.

1.3. Starting the virtual Demo Cluster

If you have chosen to configure a virtual demo-cluster, you can start it by executing the command:
0 root@cl-head ~ #
and to stop it
0 root@cl-head ~ #
These commands use the configuration file /etc/qlustar/vm-configs/demo-system.conf. If you find that the (automatically calculated) amount of RAM per VM is not right, you can change the variable CN_MEM to some other value in that file. The consoles of the virtual nodes (and also of the virtual front-end node if you chose to set one up) are accessible in a screen session. Type
0 root@cl-head ~ #
to attach to the console session of the virtual FE node and
0 root@cl-head ~ #
to attach to the console sessions of the virtual demo cluster nodes. Note that the screen command character is Ctrl-t. To detach from the screen session, type Ctrl-t+d, to switch to the next/previous screen type Ctrl-t+n / Ctrl-t+p. More details on the usage of screen (or byobu, the Debian customized version we use) are available in the corresponding man pages. To check whether all nodes are up and running, type
0 root@cl-head ~ #
dsh -a uptime
dsh or pdsh can be used to execute arbitrary commands on groups of nodes. Check their man pages and the corresponding section in the QluMan guide for further information.

1.4. Installed Services

At this stage, the following services are configured and running on your head-node:
  • Nagios3 (monitoring/alerts) with its web interface at http://headnode/nagios3/. Login as user nagiosadmin with the password you specified previously.
  • Ganglia (monitoring) at http://headnode/ganglia/
  • DHCP/ATFTP boot services
  • NTP time server as client and server
  • NFS-Server with exports defined in /etc/exports
  • Depending on your choice of software packages: Slurm (DB + control daemon), Torque server, Corosync (HA), Munge (authentification for slurm/torque). Note that among the latter only slurm and munge are configured automatically during installation. Torque and Corosync require a manual configuration.
  • NIS server
  • Mail service Postfix
  • MariaDB server (mysql fork)
  • QluMan server (Qlustar Management)


Please note, that you shouldn't install the default Ubuntu MySQL server packages on the head-node, since QluMan requires MariaDB and packages of the latter conflict with the MySQL packages. MariaDB is a complete and compatible substitute for MySQL.

1.5. Adding Software

As explained elsewhere, the RAM-based root file-system of a Qlustar compute/storage node can be supplemented by a global NFS-exported chroot to allow access to software not already contained in the boot images themselves. During installation, one chroot per selected edge platform was automatically created. The chroots are located at /srv/apps/chroots/chroot name, where chroot name would be trusty or wheezy. Each of them contains a full-featured installation of the corresponding Qlustar edge platform. To change into a chroot, convenience bash shell aliases of the form chroot-chroot name are defined for the root user on the head-node. You may use them as follows (e.g. for Debian/Wheezy, if it was selected at install):
0 root@cl-head ~ #
Once you're inside a chroot, you can use the standard Debian/Ubuntu tools to control its software packages, e.g.
(trusty) 0 root@cl-head ~ #
apt-get update
(trusty) 0 root@cl-head ~ #
apt-get dist-upgrade
(trusty) 0 root@cl-head ~ #
apt-get install package
(trusty) 0 root@cl-head ~ #
The nice thing about this mechanism is that software from packages installed in a particular chroot will be available instantaneously on all compute/storage nodes that are configured to use that chroot.


There is usually no need to install additional packages on the head-node itself (only in the chroot). Software packages installed directly on the head-node will not be visible cluster-wide.

1.6. Running the Cluster Manager QluMan

1.6.1. Generating a one time token for the first admin login

The Qlustar management GUI qluman-qt uses public/private keys for both encryption and authentication. For this there first needs to be an exchange of public keys between the client and the server. Normally this is done by an user with admin role through the GUI. But for the first admin login this must be accomplished using a root shell on the head-node:
(trusty) 0 root@cl-head ~ #
qluman-cli --gencert
Generating one-time login token for user 'admin':
Cluster  = QL
Hostname = beosrv-c
Port     = 6001
Pubkey   = b'T)5o]@hsjB2qyY>eb:7)8@BA?idMf>kh%^cRhV/#'
Enter new pin for one-time token: 
Server infos and one-time login token for user 'admin':
---[ CUT FROM HERE ]---
---[ TO HERE ]---
The token can also be saved directly into a file using the -o <filename> option and the user the token is for can be specified by the -u <username> option like this:
(trusty) 0 root@cl-head ~ #
qluman-cli --gencert -u admin -o token
Generating one-time login token for user 'admin':
Cluster  = QL
Hostname = beosrv-c
Port     = 6001
Pubkey   = b'T)5o]@hsjB2qyY>eb:7)8@BA?idMf>kh%^cRhV/#'
Enter new pin for one-time token: 
Server infos and one-time login token for user 'admin' saved as 'token'
The server infos and one-time login token are protected by the pin you just entered. This is important when the data is sent via unencrypted channels (e.g. email or chat programs) to users or when it is stored on a shared filesystem like NFS. The pin does not need to be a strong password. It is only used to make it non-trivial to use an intercepted token. The token can also only be used once. So once you use it yourself it becomes useless to anybody else. On the other hand, if somebody intercepts the token, guesses the pin and uses the token it will no longer work for you and you know something is wrong.

1.6.2. Starting the QluMan GUI

During installation, the Qlustar management GUI qluman-qt is installed for the virtual FE node, if one has been setup, otherwise on the head-node. If you like to have it available on the head-node in any case, just install it there like any other package:
0 root@cl-head ~ #
apt-get install qluman-qt
The installation on the head-node is not done by default, since it pulls and installs a lot of other packages that qluman-qt depends on, which will slow down updates. If you have the possibility, install qluman-qt on your workstation and work from there. If not, you can launch qluman-qt remotely on the node, where it is installed (FE or head-node), per ssh (with X11 forwarding enabled / -X option) as follows:
0 user@workstation ~ $
ssh -X root@servername qluman-qt
This should bring up the Management Console. Using the one-time token generated as explained above you will now be able to add the cluster to the list of available connections. (Details about this are explained in the QluMan Guide).

1.6.3. Installing the QluMan GUI on a workstation

If your workstation runs one of the edge platforms currently supported by Qlustar, you can install the QluMan GUI directly there. This is recommended, since the responsiveness of a GUI, that is locally started, is a lot better as compared to one that is running via remote X11. To install the qluman-qt package on your workstation, you need to add the correct Qlustar repository to your apt sources list. This can be accomplished by executing the following as root on your workstation.
0 root@workstation ~ #
dpkg -l software-properties-common > /dev/null 2>&1 || apt-get install software-properties-common
0 root@workstation ~ #
gpg --no-default-keyring --primary-keyring /etc/apt/trusted.gpg --recv-keys E6BA110F3C0BC307
On a Debian/Wheezy workstation, you should use python-software-properties instead of software-properties-common in the first command. The second one should have imported the Qlustar PGP archive key, and must output a line like:
gpg: key 3C0BC307: public key "Q-Leap Networks (automatic archive key) <>" imported


The gpg command above might fail, the first time you execute it. This happens, if gpg has never been executed before for the root user. In this case, simply execute it a second time.
Then if you have Ubuntu/Trusty execute:
0 root@workstation ~ #
add-apt-repository 'deb 9.1-trusty main non-free'
0 root@workstation ~ #
add-apt-repository 'deb
9.1-trusty-proposed-updates main non-free'
Else for Debian/Wheezy:
0 root@workstation ~ #
add-apt-repository 'deb 9.1-wheezy main non-free'
0 root@workstation ~ #
add-apt-repository 'deb 9.1-wheezy-proposed-updates main non-free'
After this you can install qluman-qt the usual way:
0 root@workstation ~ #
apt-get update
0 root@workstation ~ #
apt-get install qluman-qt


On Ubuntu you need to have the universe repository enabled in your apt sources list for the above command to succeed.
Finally, the QluMan GUI can then be launched as an ordinary user in a shell on the workstation:
0 user@workstation ~ $
qluman-qt &


The versions of the QluMan packages on the workstation should be the same as on the head-node(s) to ensure correct operation. Some unequal version combinations might work too, but are usually not well tested.

1.7. Creating Users

Authenticating users in the cluster can be done in many ways, hence the creation of users depends on what method is used. The most basic method is to use NIS. If there is no requirement of keeping user authentification in sync with some external service like e.g. LDAP this is sufficient. A NIS database is setup during the initial installation process and cluster users can be authenticated against it. Creating accounts that should use other authentification mechanisms is more complex and beyond the scope of this guide. Some options are explained in the admin manual. Add a test user by executing a command like this:
0 root@cl-head ~ # -u test -n "Test User"
The behavior of the script can be customized in its configuration file /etc/qlustar/common/ It also contains the definition of the initial user password.

1.8. Compiling an MPI program

MPI (Message Passing Interface) is the de facto standard for distributed parallel programming on Linux clusters. The default MPI variant in Qlustar is OpenMpi and is automatically installed in the default chroot during installation. You can test the correct installation of MPI with two small "hello world" test programs (one in C the other one in FORTRAN90) as the test user you created earlier: Login on the front-end node as this user and execute
0 testuser@cl-front ~ $
mpicc.openmpi-gcc -o hello-world-c hello-world.c
0 testuser@cl-front ~ $
mpif90.openmpi-gcc -o hello-world-f hello-world.f90
After this you should have created two executables. Check it with
0 testuser@cl-front ~ $
ls -l hello-world-?
Now we're prepared to test the queuing system with the two programs.

1.9. Running an MPI Job

Still logged in as the test user and assuming at least two demo nodes are started, we can submit the two "hello world" programs created previously as follows (commands are given for slurm):
0 testuser@cl-front ~ $
OMPI_MCA_btl="tcp,self" salloc -N 2 --ntasks-per-node=2 -p demo \
srun hello-world-c
This will run the job interactively on 2 nodes with 2 processes each (total of 4 processes). You should obtain an output like this:
salloc: Granted job allocation 19
cpu_bind=NULL - beo-201, task  0  0 [13959]: mask 0x1
cpu_bind=NULL - beo-202, task  3  1 [13607]: mask 0x2
cpu_bind=NULL - beo-202, task  2  0 [13606]: mask 0x1
cpu_bind=NULL - beo-201, task  1  1 [13960]: mask 0x2
Hello world from process 1 of 4
Hello world from process 3 of 4
Hello world from process 0 of 4
Hello world from process 2 of 4
salloc: Relinquishing job allocation 19
salloc: Job allocation 19 has been revoked.
The lines starting with cpu_bind=NULL are new in Qlustar 9 and appear due to the fact, that we have enabled the verbose setting for the configuration of cpusets under slurm. They indicate the binding (core affinity) of the separate MPI tasks to the CPUs/cores of the compute nodes. This is done by the slurm cgroup plugin, that is now enabled by default. Binding of processes to cores guarantees data (memory) locality and thus improves performance on NUMA systems, since access to local memory is faster than to remote memory. Note that essentially all modern multi-socket (CPU) systems have a NUMA architecture these days (Intel Xeon, AMD Opteron, ...), so this is relevant.
Similarly the F90 version can be submitted as a batch job using the script (to see the output, execute cat slurm-job#.out after the job has finished):
0 testuser@cl-front ~ $
sbatch -N 2 --ntasks-per-node=2 -p demo
Note that the environment variable OMPI_MCA_btl="tcp,self" is used in the above two examples to prevent error messages from not finding an Infiniband network. The latter would otherwise occur, because we compile OpenMPI to use an IB network per default and if not found, a TCP network is used as backup. TCP can also be set as the default in the OpenMPI config file (in the chroot, typically under /srv/apps/chroots/trusty/etc/openmpi/x.y.z/openmpi-mca-params.conf) by adding the entry:
btl = tcp,self

1.10. Running the Linpack benchmark

The Linpack benchmark s used to classify supercomputers in the The Top 500 list. That's why on most clusters, it's probably run as one of the first parallel programs to check functionality, stability and performance. Qlustar comes with an optimized pre-compiled version of Linpack (using a current version of the OpenBlas library) , and a script to auto-generate the necessary input file given the number of nodes, processes per node and total amount of RAM for the run.
The test user has some pre-defined shell aliases to simplify the submission of Linpack jobs. Type alias to see what's available. They are defined in $HOME/.bash/alias. Example submission (assuming you have 4 running demo nodes):
0 testuser@cl-front ~ $
Check that the job is started (output should be similar):
0 testuser@cl-front ~ $
   27      demo linstres     test   R  2:46      4     beo-[201-204]
ssh to one of the nodes in the NODELIST and check with top that Linpack is running at full steam, like:
0 testuser@cl-front ~ $
  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                 
18307 test  20   0  354m 280m 2764 R  100 28.0   6:42.92 xhpl-openblas           
18306 test  20   0  354m 294m 2764 R   99 29.3   6:45.09 xhpl-openblas
You can check the output of each Linpack run in the files: $HOME/bench/hpl/run/job-jobid-*/openblas/job-jobid-*-run#.out where jobid is the slurm JOBID (see the squeue command above) and run# is an integer starting from 1. The way the script is designed, it will run indefinitely, restarting Linpack in an infinite loop. So to stop it, you need to cancel the job like
0 testuser@cl-front ~ $
scancel jobid

Appendix A. Revision History

Revision History
Revision 9.1-0Thu Jul 31 2015Qlustar Doc Team
Initial version



contact information for Qlustar, Feedback requested