Product SiteDocumentation Site

Qlustar DataStack 9.1

Quick Start Guides

A collection of quick start guides for the components of the Qlustar DataStack.

Qlustar Documentation Team

Q-Leap Networks GmbH

Legal Notice

Copyright ©2017 Q-Leap Networks GmbH
This material may only be copied or distributed with explicit permission from Q-Leap Networks GmbH. The Qlustar license can be found at /usr/share/qlustar/LICENSE.html on an installed Qlustar head-node.


Quick start guides for the Qlustar DataStack components. Enables you to setup a basic working configuration with little effort and time.
1. Qlustar Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. Feedback requested
1. BeeGFS on Qlustar - Quick Start
1.1. Initial configuration tasks on the head-node
1.2. Setting up BeeGFS meta and storage target on the storage nodes
A. Revision History


1. Qlustar Document Conventions

Qlustar manuals use the following conventions to highlight certain words and phrases and draw attention to specific pieces of information.

1.1. Typographic Conventions

Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from an individual key by the plus sign that connects each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to a virtual terminal.
The first example highlights a particular key to press. The second example highlights a key combination: a set of three keys pressed simultaneously.
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog-box text; labeled buttons; check-box and radio-button labels; menu titles and submenu titles. For example:
Choose SystemPreferencesMouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose ApplicationsAccessoriesCharacter Map from the main menu bar. Next, choose SearchFind… from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose EditPaste from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh at a shell prompt. If the remote machine is and your username on that machine is john, type ssh
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release.
Note the words in bold italics above: username,, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.

1.2. Pull-quote Conventions

Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books        Desktop   documentation  drafts  mss    photos   stuff  svn
books_tests  Desktop1  downloads      images  notes  scripts  svgs
Commands to be executed on certain nodes of a cluster or the admins workstation are indicated by using descriptive shell prompts including user and hostname. Note that by default, the shell prompt on Qlustar nodes always ends in the newline character, thus commands are typed on the line following the prompt. As mentioned above, the command itself is shown in mono-spaced bold and the output of a command in mono-spaced roman. Examples:
0 root@cl-head ~ #
echo "I'm executed by root on a head-node"
I'm executed by root on a head-node
0 root@beo-01 ~ #
echo "I'm executed by root on a compute node"
I'm executed by root on a compute node
0 root@sn-1 ~ #
echo "I'm executed by root on a storage node"
I'm executed by root on a storage node
0 user@workstation ~ $ 
echo "I'm executed by user admin on the admins workstation"
I'm executed by user admin on the admins workstation
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:

import javax.naming.InitialContext;

public class ExClient
   public static void main(String args[]) 
       throws Exception
      InitialContext iniCtx = new InitialContext();
      Object         ref    = iniCtx.lookup("EchoBean");
      EchoHome       home   = (EchoHome) ref;
      Echo           echo   = home.create();

      System.out.println("Created Echo");

      System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));

1.3. Notes and Warnings

Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.


Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.


Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled “Important” will not cause data loss but may cause irritation and frustration.


Warnings should not be ignored. Ignoring warnings will most likely cause data loss.

2. Feedback requested

Contact to report errors or missing pieces in this documentation.

Chapter 1. BeeGFS on Qlustar - Quick Start

BeeGFS (formerly FhGFS) is a parallel cluster file system, developed and maintained by the Fraunhofer Center for High Performance Computing with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need.


BeeGFS is a non-free but fully integrated component of Qlustar. If used in any Qlustar context other than the Basic Edition, a specific BeeGFS support contract has to be purchased from Q-Leap Networks.

1.1. Initial configuration tasks on the head-node

Select BeeGFS mgmt option upon installation

  • Check that it is running on the head-node
    0 root@cl-head ~ #
    /etc/init.d/beegfs-mgmtd status

Setting up a BeeGFS server node in QluMan

  1. Create a Qlustar image with the required modules

    In QluMan, create an image (named e.g. StorNode) that contains the module beegfs-version-server (current version 2014.01). If you want to mount the BeeGFS FS also on the storage nodes additionaly include the beegfs-version-client module. On compute nodes that don't run any BeeGFS meta/storage targets, only the beegfs-version-client module is required.
  2. Create a boot config with this image

    Create a new boot config (named e.g. StorNode) and select the image just created. If necessary, add kernel parameters to the boot config as well.
  3. Create a config set that includes this boot config

    Create a new config set (named e.g. StorNode) and add the boot config (StorNode) just created.
  4. Create a disk config for the storage nodes

    Create a new disk config (named e.g. StorNode) to define the ZFS setup.
    1. Examples

      1. Create a BeeGFS FS using the local disks of compute nodes

        A single SSD in the compute nodes is to be used for a meta and storage target.
        Copy the definitions of the default disk config and add bmeta/bstor as follows:
        ZFS = var, scratch, bmeta, bstor
        zpool = SYS
        mountpoint = /beegfs/meta
        compress = lz4
        zpool = SYS
        mountpoint = /beegfs/stor
        compress = lz4
    Add the new disk config to the new config set (StorNode)
  5. Create a new host template with the above definitions

    Create a new host template (named e.g. StorNode) and select the just created config set StorNode. Use (or create) a sensible generic/hardware property set matching the properties of the storage nodes.
  6. Register and boot the storage nodes

    Register the storage nodes like any other nodes and assign the just created host template (StorNode) to them. Reboot them. After they come up again, you should see the following ZFS filesystems mounted:
    0 root@sn-1 ~ #
    df -h | grep -E '^Filesystem|beegfs'
    Filesystem                  Size  Used Avail Use% Mounted on
    SYS/bmeta                  1002T     0 1002T   0% /beegfs/meta
    SYS/bstor                  1002T     0 1002T   0% /beegfs/stor


    Note that by default, every new host has the generic QluMan property Schedule Format: always, which is required to format the disk on the first boot. After the initial boot of a node with the desired disk config in place, you should remove this property or set Schedule Format: never to prevent repeated reformatting and hence losing data.

1.2. Setting up BeeGFS meta and storage target on the storage nodes

On each storage node perform the following steps (We assume that the above disk config StorNode is initialized on each node. We further assume that the servers are named as sn-1, sn-2, ... such that ${HOSTNAME##*-} is the storage node number):

Create Metadata server instance

  1. Initialize meta target

    0 root@sn-1 ~ #
    beegfs-setup-meta -p /beegfs/meta -s ${HOSTNAME##*-} -m beosrv-c
    Preparing storage directory: /beegfs/meta
     * Creating format.conf file...
     * Creating server numeric ID file: /beegfs/meta/nodeNumID
    Updating config file: /etc/beegfs/meta/available/default.conf
     * Setting management host: beosrv-c
     * Setting storage directory in config file...
     * Disabling usage of uninitialized storage directory in config file...
     * Setting usage of extended attributes to: true
    All done.
  2. Start meta server

    0 root@sn-1 ~ #
    /etc/init.d/beegfs-meta start
  3. Check meta status

    0 root@sn-1 ~ #
    /etc/init.d/beegfs-meta status
    0 root@sn-1 ~ #
    less /var/log/beegfs/beegfs-meta.log

Create Storage server instance

  1. Initialize storage target

    0 root@sn-1 ~ #
    beegfs-setup-storage -p /beegfs/stor -s ${HOSTNAME##*-} \
    -i ${HOSTNAME##*-}01 -m beosrv-c
    Preparing storage target directory: /beegfs/storage
     * Creating format.conf file...
     * Creating chunks directory...
     * Creating mirror directory...
     * Creating target numeric ID file: /beegfs/storage/targetNumID
     * Creating server numeric ID file: /beegfs/storage/nodeNumID
    Updating config file: /etc/beegfs/storage/available/default.conf
     * Setting management host: beosrv-c
     * Appending to target directory list in config file...
     * Disabling usage of uninitialized storage targets in config file...
    All done.
  2. Start storage server

    0 root@sn-1 ~ #
    /etc/init.d/beegfs-storage start
  3. Check storage status

    0 root@sn-1 ~ #
    /etc/init.d/beegfs-storage status
    0 root@sn-1 ~ #
    less /var/log/beegfs/beegfs-storage.log

Register the config files on the head-node for automatic start on boot

  1. Retrieve config files

    Retrieve the config files from the first storage node sn-1 (the configs on different nodes are all identical):
    0 root@cl-head ~ #
    mkdir -p /etc/qlustar/common/image-files/beegfs/{meta,storage}
    0 root@cl-head ~ #
    scp sn-1:/etc/beegfs/meta/available/default.conf \
    0 root@cl-head ~ #
    scp sn-1:/etc/beegfs/storage/available/default.conf \
  2. Generate the Qlustar destination files

    0 root@cl-head ~ #
    cat <<EOF > /etc/qlustar/common/image-files/destinations.beegfs-server
    beegfs/meta/default.conf /etc/beegfs/meta/available root:root 644
    beegfs/storage/default.conf /etc/beegfs/storage/available root:root 644
    0 root@cl-head ~ #
    cat <<EOF > /etc/qlustar/common/image-files/destinations.beegfs-client
    # Link source                  target
    ../available/default.conf /etc/beegfs/client/enabled
     0 root@cl-head ~ #
    cat <<EOF > /etc/qlustar/common/image-files/softgroups
    beegfs-server: sn-[1-4]
    beegfs-client: beo-20[1-4] login-c
    Their beegfs role is assigned to the nodes in /etc/qlustar/common/image-files/softgroups. Change the node names there according to your naming scheme.
With these configurations in place, BeeGFS server and clients should start/mount automatically upon boot. The default client mount point is /beegfs/client/default.

Appendix A. Revision History

Revision History
Revision 9.1-0Thu Jul 2 2015Qlustar Doc Team
Initial version



contact information for Qlustar, Feedback requested