Quick Start Guides

1. BeeGFS on Qlustar - Quick Start

BeeGFS (formerly FhGFS) is a parallel cluster file systemCluster FSBeeGFS, developed and maintained by the Fraunhofer Center for High Performance Computing with a strong focus on performance and designed for very easy installation and management. It transparently spreads user data across multiple servers. By increasing the number of servers and disks in the system, you can simply scale performance and capacity of the file system to the level that you need.

1.1. Initial configuration tasks on the head-node

  • Check that it is running on the head-node

    0 root@cl-head ~ #
service beegfs-mgmtd@default status

    The extension @default in the systemd service name beegfs-mgmtd@default indicates the use of a so-called systemd template service file. Qlustar uses the latter to implement a clean systemd integration for multi-target BeeGFS setups. The string after @ in the extension is the name of the config file used for the particular instance of a daemon. In this case the config file is /etc/beegfs/mgmtd/enabled/default.conf. The same concept is used for meta and storage targets (see below).
    Setting up a BeeGFS server node in QluMan

    1. Create a Qlustar image with the required modules

      In QluMan, create an image (named e.g. StorNode) that contains the module beegfs-<version>-server (current version 6). If you want to mount the BeeGFS FS also on the storage nodes, additionaly include the beegfs-<version>-client module. On compute nodes that don’t run any BeeGFS meta/storage targets, only the beegfs-<version>-client module is required.

    2. Create a boot config with this image

      Create a new boot config (named e.g. StorNode) and select the image just created. If necessary, add kernel parameters to the boot config as well.

    3. Create a config set that includes this boot config

      Create a new config set (named e.g. StorNode) and add the boot config (StorNode) just created.

    4. Create a disk config for the storage nodes

      Create a new disk config (named e.g. StorNode) to define the ZFS setup.

      1. Examples:

        1. Create a BeeGFS FS using the local disks of compute nodes

          A single SSD in the compute nodes is to be used for a meta and storage target.

          Copy the definitions of the default disk config and add bmeta/bstor as follows:

          ZFS = var, scratch, bmeta, bstor
[bmeta]
zpool = SYS
mountpoint = /beegfs/meta
compress = lz4
[bstor]
zpool = SYS
mountpoint = /beegfs/stor
compress = lz4

          Add the new disk config to the new config set (StorNode)

    5. Create a new host template with the above definitions

      Create a new host template (named e.g. StorNode) and select the just created config set StorNode. Use (or create) a sensible generic/hardware property set matching the properties of the storage nodes.

    6. Register and boot the storage nodes

      Register the storage nodes like any other nodes and assign the just created host template (StorNode) to them. Reboot them. After they come up again, you should see the following ZFS filesystems mounted:

      0 root@sn-1 ~ #
df -h | grep -E '^Filesystem|beegfs'
Filesystem                  Size  Used Avail Use% Mounted on
SYS/bmeta                  1002T     0 1002T   0% /beegfs/meta
SYS/bstor                  1002T     0 1002T   0% /beegfs/stor

      Note that by default, every new host has the generic QluMan property Schedule Format: always, which is required to format the disk on the first boot. After the initial boot of a node with the desired disk config in place, you should remove this property or set Schedule Format: never to prevent repeated reformatting and hence losing data.

1.2. Setting up BeeGFS meta and storage target on the storage nodes

On each storage node perform the following steps (We assume that the above disk config StorNode is initialized on each node. We further assume that the servers are named as sn-1, sn-2, …​ such that ${HOSTNAME##*-} is the storage node number).

It is most convenient to use the QluMan RXengine to execute the steps in parallel on all storage nodes.
Create Metadata server instance

  1. Initialize meta target

    0 root@sn-1 ~ #
beegfs-setup-meta -p /beegfs/meta -s ${HOSTNAME##-} -m beosrv-c*
Preparing storage directory: /beegfs/meta
 * Creating format.conf file...
 * Creating server numeric ID file: /beegfs/meta/nodeNumID
Updating config file: /etc/beegfs/meta/available/default.conf
 * Setting management host: beosrv-c
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
 * Setting usage of extended attributes to: true
All done.

  2. Start meta server

    0 root@sn-1 ~ #
service beegfs-meta start

  3. Check meta status

    0 root@sn-1 ~ #
service beegfs-meta@default status
0 root@sn-1 ~ #
less /var/log/beegfs/beegfs-meta.log
Create Storage server instance

  1. Initialize storage target

    0 root@sn-1 ~ #
beegfs-setup-storage -p /beegfs/stor -s ${HOSTNAME##*-} \
-i ${HOSTNAME##*-}01 -m beosrv-c
Preparing storage target directory: /beegfs/storage
 * Creating format.conf file...
 * Creating chunks directory...
 * Creating mirror directory...
 * Creating target numeric ID file: /beegfs/storage/targetNumID
 * Creating server numeric ID file: /beegfs/storage/nodeNumID
Updating config file: /etc/beegfs/storage/available/default.conf
 * Setting management host: beosrv-c
 * Appending to target directory list in config file...
 * Disabling usage of uninitialized storage targets in config file...
All done.

  2. Start storage server

    0 root@sn-1 ~ #
service beegfs-storage start

  3. Check storage status

    0 root@sn-1 ~ #
service beegfs-storage@default status

0 root@sn-1 ~ #
less /var/log/beegfs/beegfs-storage.log

Register the config files on the head-node for automatic start on boot:

+ . Retrieve config files

+ Retrieve the config files from the first storage node sn-1 (the configs on different nodes are all identical):

+

0 root@cl-head ~ #
mkdir -p /etc/qlustar/common/image-files/beegfs/{meta,storage}

0 root@cl-head ~ #
scp sn-1:/etc/beegfs/meta/available/default.conf \
/etc/qlustar/common/image-files/beegfs/meta/default.conf

0 root@cl-head ~ #
scp sn-1:/etc/beegfs/storage/available/default.conf \
/etc/qlustar/common/image-files/beegfs/storage/default.conf

+ . Generate the Qlustar destination files

+

0 root@cl-head ~ #
cat <<EOF > /etc/qlustar/common/image-files/destinations.beegfs-server
beegfs/meta/default.conf /etc/beegfs/meta/available root:root 644
beegfs/storage/default.conf /etc/beegfs/storage/available root:root 644
EOF

0 root@cl-head ~ #
cat <<EOF > /etc/qlustar/common/image-files/destinations.beegfs-client
# Link source                  target
../available/default.conf /etc/beegfs/client/enabled
EOF

0 root@cl-head ~ #
cat <<EOF > /etc/qlustar/common/image-files/softgroups
[DEFAULT]

beegfs-server: sn-[1-4]
beegfs-client: beo-20[1-4] login-c
EOF

+ Their beegfs role is assigned to the nodes in /etc/qlustar/common/image-files/softgroups. Change the node names there according to your naming scheme.

With these configurations in place, BeeGFS server and clients should start/mount automatically upon boot. The default client mount point is /beegfs/client/default.