Product SiteDocumentation Site

Qlustar Cluster OS 9.2

Installation Guide

This document describes the necessary steps to install the Qlustar Cluster Operating System.

Qlustar Documentation Team

Q-Leap Networks GmbH

Legal Notice

Copyright ©2017 Q-Leap Networks GmbH
This material may only be copied or distributed with explicit permission from Q-Leap Networks GmbH. The Qlustar license can be found at /usr/share/qlustar/LICENSE.html on an installed Qlustar head-node.

Abstract

This document describes the necessary steps to install the Qlustar operating system. The Qlustar installer has been kept simple, yet it should include all necessary installation options a HPC cluster head-node needs. A setup with redundant (HA) head-nodes is supported, but such a configuration is a lot more complex and beyond the scope of this guide. Note, that this will require (a) a carefully chosen hardware configuration, (b) quite some manual configuration and (c) a lot of testing to ensure reliable operation.
Preface
1. Qlustar Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. Feedback requested
1. Installation Requirements
2. Loading the installer to a USB flash-drive
3. Installation process
A. Revision History
Index

⁠Preface

⁠1. Qlustar Document Conventions

Qlustar manuals use the following conventions to highlight certain words and phrases and draw attention to specific pieces of information.

⁠1.1. Typographic Conventions

Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from an individual key by the plus sign that connects each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to a virtual terminal.
The first example highlights a particular key to press. The second example highlights a key combination: a set of three keys pressed simultaneously.
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog-box text; labeled buttons; check-box and radio-button labels; menu titles and submenu titles. For example:
Choose System → Preferences → Mouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose Applications → Accessories → Character Map from the main menu bar. Next, choose Search → Find… from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose Edit → Paste from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh username@domain.name at a shell prompt. If the remote machine is example.com and your username on that machine is john, type ssh john@example.com.
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release.
Note the words in bold italics above: username, domain.name, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.

⁠1.2. Pull-quote Conventions

Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus: 
books        Desktop   documentation  drafts  mss    photos   stuff  svn
books_tests  Desktop1  downloads      images  notes  scripts  svgs
Commands to be executed on certain nodes of a cluster or the admins workstation are indicated by using descriptive shell prompts including user and hostname. Note that by default, the shell prompt on Qlustar nodes always ends in the newline character, thus commands are typed on the line following the prompt. As mentioned above, the command itself is shown in mono-spaced bold and the output of a command in mono-spaced roman. Examples: 
0 root@cl-head ~ #
echo "I'm executed by root on a head-node"
I'm executed by root on a head-node

0 root@beo-01 ~ #
echo "I'm executed by root on a compute node"
I'm executed by root on a compute node

0 root@sn-1 ~ #
echo "I'm executed by root on a storage node"
I'm executed by root on a storage node

0 user@workstation ~ $ 
echo "I'm executed by user admin on the admins workstation"
I'm executed by user admin on the admins workstation
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows: 
​package org.jboss.book.jca.ex1;
​
​import javax.naming.InitialContext;
​
​public class ExClient
​{
​   public static void main(String args[]) 
​       throws Exception
​   {
​      InitialContext iniCtx = new InitialContext();
​      Object         ref    = iniCtx.lookup("EchoBean");
​      EchoHome       home   = (EchoHome) ref;
​      Echo           echo   = home.create();
​
​      System.out.println("Created Echo");
​
​      System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
​   }
​}

⁠1.3. Notes and Warnings

Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.

Note

Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.

Important

Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled “Important” will not cause data loss but may cause irritation and frustration.

Warning

Warnings should not be ignored. Ignoring warnings will most likely cause data loss.

⁠2. Feedback requested

Contact to report errors or missing pieces in this documentation.

⁠Chapter 1. Installation Requirements

To install the Qlustar OS you need:
  • A DVD or a USB flash-drive (minimum size 2GB) loaded with the Qlustar installer
  • A 64bit x86 server/PC (or virtual machine) with
    • at least two network adapters
    • at least one disk with a minimum size of 160GB
    • optionally a second (or more) disk(s) with a minimum size of 160GB
    • CPU supporting virtualization (for virtual front-end and demo nodes)
    • Internet connection

Note

The only virtual machine (VM) we test the installation process on is KVM. Hence, this is the only VM officially supported by Qlustar. Furthermore, while nesting of VMs is supported, it will be rather slow to start a virtual front-end node or a virtual demo-cluster directly from within a head-node that is itself running as a VM.

Note

The Qlustar kernel is configured for cluster server hardware. Hence, it's possible, that drivers for some network or storage adapters on Laptops or other non-server PCs are missing. If you run into such problems, please post it in the Installation forum. Qlustar also has no support for secure boot (UEFI). If your hardware uses UEFI by default, you need to configure your BIOS to allow so-called legacy boot.

⁠Chapter 2. Loading the installer to a USB flash-drive

Starting form Qlustar 9, we provide an ISO install image, that can be burned onto a DVD or be loaded onto a USB flash-drive. This section explains the USB method. If you already have an up-to-date Qlustar USB drive you can skip this step. But make sure that you have the latest installer to be found at the Qlustar website.

Important

After downloading the installer image, verify its integrity, by executing the following command: 
0 user@workstation ~ $ 
sha256sum qlustar-installer.iso.gz
Now compare the such-generated checksum with the value that is published at the above download page. If the checksums are not identical, do not proceed with the installation, since that means the installer is corrupt (either due to hardware errors or a malicious attack). In this case, you should download the image once more. If the problem persists, also after downloading from a different computer, please post the incidence in the support forum.
The instructions in this guide only apply to the latest version. With the downloaded disk image you can easily create a bootable drive under Linux as follows (note that the USB drive should have a minimum size of 2GB): Given that the downloaded installer image file is named qlustar-installer.iso.gz and that the device name of the drive is /dev/sdb, the following command (executed as root) will load the image to the flash-drive: 
0 root@workstation ~ # 
gunzip -c qlustar-installer.iso.gz | dd of=/dev/sdb
Depending on the speed of your flash-drive, this can take up to 20 minutes.

Warning

The above command will destroy all data on the USB drive or to be more precise, it will destroy the data of the device named /dev/sdb! So make sure to find out the correct device name of the drive.
This can be achieved e.g. by executing the command 
0 root@workstation ~ # 
tail -f /var/log/kern.log
at the shell prompt before you insert the drive. Just after you have inserted it, a message like 
[sdb] Attached SCSI removable disk
should appear. In this case your USB drive would have the device name /dev/sdb. If the above dd command finished successfully, you can use the drive to install the Qlustar OS on the target machine. This is described in the next paragraph.

⁠Chapter 3. Installation process

⁠

Step by step instructions

  1. Booting from the USB Drive

    Insert the drive into the server that you want to install the Qlustar OS on and boot. Make sure to select the USB drive as the boot device, either by entering the BIOS and changing the boot order or by entering a boot menu and then selecting the drive. On some BIOSes the latter is achieved by pressing the key F7 or F12.

  2. Starting the installation process

    When the server boots from your drive, it will present a menu with various alternative boot options of which the first is highlighted (see picture). If you install via a serial console (e.g. IPMI serial-over-LAN) select the correct entry (ttyS0/1/2) depending on your hardware setup. Press Enter to start the installation. After the kernel was loaded, you will see the kernel messages on the screen followed by some messages of the installation system. Finally the installer will present its welcome screen at which you can start the installer configuration process by pressing Enter.

  3. Localization settings

    In the first screen you can select the timezone and keyboard mapping. After you have made your choices, selected OK and pressed Enter, the keyboard mapping will be configured according your choice. If you have selected a wrong mapping the keyboard will not function properly. If this is still the case after changing the mapping to the correct value, you will have to reboot the server and start the installation from scratch.

  4. Disk Configuration

    In this screen you can select the disk(s) to install Qlustar on. If there is more then one disk in the system you can select multiple disks and create a RAID array (different RAID levels can be chosen depending on the number of the disks available). The chosen disk (or created RAID device) will then be used as a LVM physical volume to create a LVM volume group (the name of which can be specified). Four ext4 file-systems (root, var, tmp, apps) will automatically be created on equally named LVM logical volumes within this volume group.

  5. Home File-System

    A separate file-system is used for the home directories. If there are additional unused disks in the system you are offered the choice to use them in the next screen. If not, you can use the previously configured volume group for this additional file-system. If you choose "Other", then no configuration for a home file-system will be done. In this case, you won't be able to add any cluster users without further manual steps to setup a home file-system.

  6. Home File-System on separate Disks

    This screen only appears if you have selected "Select disks" in the previous step. You can choose the disks and possibly the raid level to use for the home file-system similar to the previous dialog.

  7. Network Settings (1)

    In this screen, you configure the IP addresses for the cluster and public network. The installer has some logic to check for valid network configuration parameters as well as suggesting reasonable values. The chosen number of compute nodes NK in the cluster determines the suggested cluster network address and other parameters. It doesn't have to be accurate, just an approximate number will do.

    Note

    Note that Qlustar requires cluster-internal IP networks to be in a non-public range. If you're unhappy with the range suggested, you can change NK to another number (e.g. NK=1000 suggests 10.x.x.x, NK=200 172.16.x.x and NK=16 192.168.x.x).
    You can also set the hostname (associated with the public IP) for the new head-node. The name of the cluster-internal interface is beosrv-c. Note that two network adapters are necessary for proper operation of your cluster.
    It's important that you correctly identify external (public/access to the Internet) and internal (cluster) network adapters. If you select DHCP as the configuration method for the public interface, make sure to add its IP address later to the NIS and ssh related header configurations in the graphical QluMan management interface. If you don't have any special reason for changing the suggested address/netmask of the internal adapter, we recommend to leave them unchanged.
    Postfix is running on the head-node as the central mail transfer agent. You need to specify a mail relay host (either by name if resolvable via DNS or by IP) in your LAN, where mail of the cluster will be accepted and redistributed. Root mails generated on the cluster, will be redirected to a list of e-mail addresses, that you can enter next (multiple addresses should be separated by a comma).

  8. Network Settings (2)

    Here you have the possibility to configure optional Infiniband (IB) and/or IPMI network parameters, if you have the corresponding hardware in your cluster.
    If there are nodes with IB in the cluster, you first need to set the option whether the head-node itself also has IB hardware or not. If it doesn't, you only need to choose the IB TCP/IP network address parameters (defaults are again recommended).
    Otherwise, you have the additional choice to run OpenSM (IB subnet manager) on the head-node. Note, that at least one node (or switch) in an IB fabric has to run a subnet manager. Given the value of the IB network address, the host part of the IB adapters IP address will be set from the host part of the Ethernet address of the node in the cluster network. Example: IP cluster network 192.168.52.100 - IP IB network 192.168.53.100.
    As with IB, if some of the nodes of your cluster have an IPMI adapter, that is connected to a switch, you can enter the network address parameters that will be used to configure its IP address. Again, the host part of the IPMI adapters IP will be derived from its cluster network IP (see above). Additional, host specific IPMI configuration options can be set in QluMan. Please note, that the configuration of IPMI adapters is hardware dependent. The method, we use, works on most systems, but we can't give any guarantee, that it will work on yours. Please contact us in case of problems.

  9. Additional Settings

    For the sake of system stability and performance, it is usually a good idea to separate user from system activities as much as possible. To achieve this, we suggest to run only system-related tasks directly on the cluster's head-node(s) and have physical or virtual front-end (FE) nodes for user access/activity. In this dialog you can choose to setup a virtual FE node as a virtual machine running on the head-node. If you decide not to do so, we advise to assign a physical machine as a FE node at a later stage. The suggestions for the network parameters of the FE VM are derived from the head-node's values. Again, if you select DHCP as the configuration method for the public interface, make sure to add its IP address later to the NIS and ssh related header configurations in the graphical QluMan management interface. The name of the FE's cluster-internal interface would be login-c.
    You also have the option to create a virtual demo-cluster on the head-node, provided the latter has enough main memory. When choosing the number of demo-nodes, keep in mind, that each one of them will be configured with 1.5GB of main memory. Even if configured, the VMs will not be started automatically though, so the head-node will be able to boot, even if an insane number of demo VMs are chosen.

    Note

    If you choose neither the option for the virtual FE node, nor the virtual demo-cluster, the necessary network pre-configurations (bridge setup, etc.) for starting VMs will not be done. This means, that you'll have to do a number of non-trivial manual steps after installation, in case you find out, that you need support for VMs later on.

    Note

    If you intend to follow all the instructions in the First Steps Guide, a virtual demo-cluster is a requirement.
    If you need a proxy to access the WWW, you can enter the relevant proxy parameters here. To speed up the installation of additional packages later on, you could copy the software packages on the USB drive to the hard-drive of the installed machine. But note that the advantage of this fades away over time, once more and more packages will have newer versions/updates and hence will have to be downloaded from the Qlustar repository on the Internet anyway. To use a serial device for console access, select the next option and enter the device name. Additional kernel parameters for the head-node can be specified next (typically, none are needed). Finally type in the password for the root user twice.

  10. Edge Platform & Package Bundle Selection

    • ⁠Edge Platform Selection

      Qlustar supports multiple so-called Edge Platforms. In 9.2, the available ones are Ubuntu/Trusty and Debian/Wheezy. Please select your favorite platform(s). Multiple choices are possible. At least one must be selected.
      In essence, this option corresponds to the choice of what Linux distribution you (or your users, if you're an administrator) like to work with/develop for. Note that you can always add support for a previously unselected edge platform later on by creating corresponding chroots and Qlustar images using QluMan.
      Technically, choosing a specific edge platform (e.g. Ubuntu/Trusty) will cause Qlustar images based on Ubuntu/Trusty to be created along with an associated chroot. The latter will hold a complete Ubuntu/Trusty installation in which you can later add packages. By assigning such an image to some nodes later on, you will have a full-featured Ubuntu/Trusty installation available on these nodes, once they have booted over the network. Since multiple edge platforms are available, it is fully supported and easily achieved to have subsets of nodes running Ubuntu/Trusty and Debian/Wheezy simultaneously in the same cluster.

      Note

      The Qlustar 9.2 installer doesn't offer Debian/Wheezy anymore, since security support for it will be terminated at the beginning of 2018. However from within QluMan, you will still be able so setup images and chroots with Debian/Wheezy.

    • ⁠Package Bundle Selection

      The default installation will only install a minimal set of basic packages on the head-node of your cluster. Here you can select additional package bundles to install. Among others, you have the choice of two workload managers: Slurm and Torque. If you don't have a particular reason why you must use Torque, you should choose Slurm which has a nice QluMan config and operation GUI component. Furthermore, if you intend to follow all the instructions in the First Steps Guide, Slurm is a requirement.

      Note

      Beware, that you should choose Lustre as an option here only, if the head-node itself is supposed to run Lustre services (MGS/MDTs/OSTs). If you merely intend to try out Lustre, it is recommended to configure an image with Lustre support in QluMan, and assign it to some nodes of the virtual demo-cluster

      Note

      Apart from Nvidia CUDA software, the packages selected here will only be installed on the head-node. Selecting and generating boot-image modules as well as the process of general package installation for compute/storage nodes booting from the network must be done elsewhere. This is discussed in the First Steps and the QluMan Guide

  11. Reviewing Installation Settings

    Before the actual installation will begin, the final screen shows a summary of the settings, that have been selected/will be used (you will have to move back with the left arrow key in order to scroll down in the summary screen). If there are any parameters, you want to change, just go back and make the required modifications.

  12. Actual Installation

    The actual installation process will take a few minutes to complete. Press Enter when requested at the end of the process. Remove the USB drive and press Enter again to reboot. Please note that the head-node has a text-only console. An X server (graphical interface) is not installed, in order to keep resource consumption low and eliminate package overhead.

⁠Appendix A. Revision History

Revision History
Revision 9.2-0Thu Apr 27 2017Qlustar Doc Team
Updates for Qlustar 9.2
Revision 9.1-0Thu Jul 24 2015Qlustar Doc Team
Initial version

⁠Index

F

feedback
contact information for Qlustar, Feedback requested