HA Admin Manual

The most elementary example for a Qlustar HPC cluster with HA capability is one that has two equivalent head-nodes. The entities/services of a single standard Qlustar head-node that must be configured in a different way to achieve HA for such a setup are the following:

Other head-node services that don’t need special treatment in HA are dnsmasq, apache (monitoring), NTP since they can run on both head-nodes without dynamic synchronization of their data/state files.

An example for an advanced Qlustar HPC cluster with HA capability is shown in the diagram to the left. It consists of two head-nodes like in the elementary case and in addition a storage cluster (serving e.g. a Lustre parallel filesystem) made up of storage-node pairs. Physically, these pairs are setup such that each node can run all the services of its partner.

The QHAS components running on the head-nodes are in charge to control and monitor the services on the storage-nodes. This entails sending the commands for start-/stopping as well as migrating resources from one storage-node to its partner in case a failure was detected. The complete HA logic of QHAS is implemented on the head-nodes and all other HA nodes are just command recipients.

The simplest action is to display the current status of the cluster using --status. This will first display the current automation settings and available automation presets. Next a list of all HA nodes, their KOTON (or power management) status, their HA status and last a list of resource groups configured for the node. That is followed by one section for each resource group listing the groups node associations and resource status. Last is a section for pending or running actions, if any. For a freshly booted cluster with the routers started on both headnodes this looks like this:

When any actions are pending or currently running they are listed at the end:

Here is an example of qluman-dhcpscanner having failed. The monitoring of the resource fails and initiates a repair action. A failure of a resource can be due to one of it’s dependencies or the underlying reason for one resource failing will affect other resources the same way. The repair action therefore checks all the resources of the resource group in order of dependencies. So while in this example only qluman-dhcpscanner was killed as a test the repair action shows all the other resources of the core resource group being probed.

In the output each line shows one action starting with the time of the actions last activity and a tree representation of the relationship between actions. Larger actions like a repair is split into separate smaller actions like the resource probes. This can happen recursively until every action is split into a single command as can be seen with the restart Event.

Each action keeps a log of the individual steps involved in processing the action. The log can be included in the status display if the --verbose option is also specified. The above example with --verbose looks like this:

The same information is also mirrored in the router log files but there it is in a flat list showing everything that happened sorted by time and picking out the relevant lines for a specific action is difficult.

Each log entry for an action starts with a severity level. In the above case all entries have the info level. Other levels are error, stderr, stdout, debug and trace. Info and error levels show what’s going on with the action on a high level. The stderr and stdout levels corresponds with the output of commands that are executed and can be helpful in understanding why a command executed for a resource failed. The debug and trace levels are basically useless without internal knowledge or following the code path in the source and it’s probably best to ignore those.

The actions to manipulate resource and resource groups are: --start. --stop, --restart and --migrate. In a way also --koton but that is covered in the next chapter about manipulating nodes. The first three action can be used on resources or resource groups while migrate always requires a resource group since all actions of a resource group must run on the same node. Following the action a list of resources or resource groups must be specified. Specifying a resource group in an action is equivalent with listing all resources of the resource group individually. As a special alias all can also be specified to, as the name says, act on all resources.

As the names imply --start is used to start resources and --stop to stop resources. This will honor the dependencies of resources, both required and after. A little more complex is --restart as it will first stop resources in order of the required dependency and then start them back up. The after dependencies are ignored. One step further is --migrate, which will stop the resource group on one node and start it again on the alternate node.

Per default actions in qluman-ha-cli initiate the action and confirm their execution ignoring the automation settings for manually actions. This is unlike the behavior of qluman-qt as it is assumed that typing in the full name of a resource or resource group is hard enough to as it is while button is too easy to click accidentally. Also this allows qluman-ha-cli to be used in scripts. Users should be aware that there is no safety net with qluman-ha-cli.

After initiating the action and confirming it’s execution qluman-ha-cli simply returns:

A successful return only means the action has been started. It’s execution is still ongoing and might still fail. The progress of the action can be checked using the --status action.

Actions are executed in order of dependencies. A waiting state signals that one or more dependencies of an action still need to complete. But actions are executed in parallel as much as the dependencies allow. So it’s not uncommon to have multiple actions in running state.

When everything works fine actions can execute and finish quickly and trying to catch them in the action as they change with the --status command is hit and miss. Instead --monitor option can be used to tell qluman-ha-cli to wait for an action to complete and to display resource and resource group changes as they happen:

There is one extra option that applies to the --start action and that is --node. Per default resources and resource groups are started on their preferred node, assuming the resource group doesn’t already have a current node because another of it’s resources is running. To start a resource or resource group on the alternate node the node must be specified using --node.

There are also actions to manipulate nodes instead of resources or resource groups. The first of those is --set-offline. This is a special action to tell the HA system that a node is unavailable and does not have any resources running on it. It is only applicable to nodes in the unknown state and saves the HA system to wait for timeouts and to KOTON unreachable nodes. Most notable this is useful when starting the cluster with only one headnode.

The next two actions are --activate and --deactivate. The first application of this is for netboot nodes. When a netboot HA node boots it enters the inactive state:

This means the node has successfully booted, the remote execution engine is up and running and the node is ready to handle resources. But in case iSCSI still needs some more time to settle or the node was booted up for diagnostic reasons it is not automatically cleared for HA use. This can be done with the --activate action. The --deactivate action does the opposite and declares the node no longer suitable for HA resources.

There is also a second use case for --activate. When a resource on a node experiences problems that are not resolved by restarting the resource then the node is put into an error state and the affected resource group is migrated to it’s alternate node. This prevents resources from migrating to the suspect node, most importantly the resource that has just been migrated away is prevented from going back and forth over and over if it fails on both nodes. The --activate command is used to clear the error state of a node, including headnodes.

The last action related to nodes is --koton. This is the big brother to migrate. This will stop all resources on the named nodes, power cycle the node and then start the resource groups on their alternate node(s). Due to the power cycle this also works when resources can’t be stopped and it’s exactly for that use case that it exists.

When using just --koton the router attempts to stop the resources running on the specified nodes. In cases just some resources can’t be stopped this still shuts down the other resources on the node cleanly. For cases where this is known to be pointless, for example when the zpool required by all resources failed and none can be stopped, the --hard option can be added. This will skip the attempt to stop resources and immediately power cycle the node.

So far all the commands have been for initiating actions like starting a resource and qluman-ha-cli automatically confirms those actions so they immediately are executed. Not so when starting actions via qluman-qt or when the automation initiates repair actions on it’s own. In both of those cases the automation settings are consulted to see if an action should wait for confirmation or execute immediately.

As previously mentioned the --status command will show pending actions:

Pending actions are in the unconfirmed state as can be seen above for the restart event and stop action. To continue executing the action must be confirmed by listing the actions ID as argument to --confirm. Children of actions are confirmed when their parent is confirmed. So in the above example action ID 709, 733 or 735 can be confirmed. The alias all can also be used to confirm all unconfirmed actions. Say we confirm action ID 733 then the action status will change to this:

Both action 733 and 735 got confirmed and then executed successfully. This then resulted in the follow up actions to start qluman-dhcpscanner. These must be confirmed again to actually start the resource again.

Confirming an action also confirms future children of an action. So in the above if action ID 709 had been confirmed instead of ID 733 then the resource start for qluman-dhcpscanner would have been implicitly confirmed and executed automatically.

Besides --confirm actions can also be aborted by using --reject. This marks the action and all it’s children as failed. Which also means their parent actions will fail. The use case for this is when one disagrees with the suggested actions for fixing a resource or when an action was initiated by accident.

At the top of the status output the automation settings and available automation presets are listed:

The above shows the default settings the router enters when freshly started. This matches the Maintenance preset that is created per default. For each of the 3 groups of automation settings (Core Resources, General Resources and Manual Actions) the actions that are immediately executed without extra confirmation are listed, in this case only manual start actions.

Individual settings can only be changed through qluman-qt but presets can be selected from qluman-ha-cli using the --preset command:

After setting a preset the new automation settings are shown in the HA status. In this example the default Automatic preset allowing the automatic free hand while manual actions are still subject to confirmation.

After a cold start of the cluster and the routers have been started the HA Status should show both headnodes with 2 green LEDs and other nodes with a red/yellow LED combination. See the Nodes chapter below for details what those mean. All Resources Groups and Resources should have blue LEDs showing them as offline.

To start the cluster the core Resource Group need to be started first. Select the core Resource Group and then click the Start button. This should then show a new Action for starting Resource Groups in the =Action Log= at the right of the Resource Groups section. Since the default automation settings have start actions on automatic the resources will be started without further user intervention as can be seen by the chaning LEDs for both the Resources and the Actions. Eventually the Resource Group and the main Event for the start action should turn green signaling that all resources are online.

Once the core Resources are online the other nodes in the cluster can boot. At this point the KOTON status of nodes should turn green and once a HA node has booted it’s status should turn green/yellow signaling it’s readyness. Again since the startup is fickle HA nodes do not automatiocally become active. Click the Activate All button to activate all ready nodes or select nodes and activate them individually.

Once the remaining HA Nodes are activated the remaining HA Resource Groups can be started. Select the resource groups and click the Start button. Again an Event will appear in the Action log, this time with multiple sub events, one per selected Resource Group. Again the status LED will change as actions are executed and Resources become online.

Hopefully after a short time all LEDs are green and the cluster is ready for use.

The Nodes section in the HAStatus Window shows the status of all HA Nodes in the cluster, the Resource Groups assigned to each node, marked in bold if the Resource Group is running on the node. The primary headnode, determined by having the core Resource Group running on it, is marked in bold and with the string "(primary)".

Each node has 2 status LEDs. Hovering over a node will show a tool tip listing the status for each LED. The left LED indicates the availability of power management via ipmi. The nodes power status is periodically checked to ensure the ipmi connection is working. This ensures KOTON can be performed when necessary. The right LED indicates the status of the node itself. A green LED indicates all is well. A yellow LED indicates that the status is unknown and the node is unreachable. A red LED indicates an error. A a dark blue LED indicates an offline node.

Unlike yellow or red an offline node is in a well defined state where it is known that none of the Resource Groups are currently running on that node. A node is marked offline after a successful KOTON of the node. Power cycling ensures that the node has stopped running any of the Resource Groups. The only other way to get a node into offline state is to manually set it offline. Select a node and click the Set Offline button. This is a promise to Qluman that none of the Resource Groups are running on that node for cases where the node should not or can not be KOTONed.

Last a yellow/green LED indicates a node that is ready but has not been set active for HA operations. When a node it first booted or reboots after being KOTONed the node is not automatically used for HA. A node is KOTONed when a Resource can not be stopped, usually due to kernel or hardware failures and a power cycle is the only way to ensure the Resource is released. As such the admin should investigate the node to make sure the error has been resolved before returning the node to active status. The active status of all nodes can be changed by clicking the Activate All or Deactivate All buttons. The same buttons can also be used to change individual nodes by first selecting the respective nodes.

When a node is selected the KOTON action can also be triggered manually. This comes in two flavours, selectable from the drop down menu on the button. The default when simply clicking the button is to first migrate resource groups away from the selected node before triggering a power cycle via IPMI. This is the preferred method if the node is still somewhat functional as it allows a cleaner transition of resources from one node to the other. But it can take a long time when resources can not be stopped due to kernel or hardware errors. The other option is to "Power off now", which does as it says and immediately trigger the IPMI power cycling without first trying to stop resources. Post KOTON the Resource Groups that were running on the selected node are still restarted on the alternate node for each Resource Group if possible.

The Resource Groups section is split in two parts, the actual resources and resource groups on the left and the Action log on the right. The resources and resource groups are arranged as a tree with the resource groups at the top level. In front of each resource is an LED showing the current status of the resource. In front of resource groups is a combined LED showing the merged status of all resources of that group. In the second column the resource type is listed for each resource while for resource groups the current node of the resource group, if any, is shown in bold.

A yellow LED denotes a resource in unknown state, where the HA system hasn’t yet determined the true state of the resource. A dark blue denotes an offline resource while green denotes online and a resource with error will be marked in red. Resources with pending actions are marked with a split LED. One half is yellow denoting that the resource is in some intermediate state while the other half is colored for the targeted state, dark blue if the resource is to be offline and green if the resource is to be online at the end of the action.

At the bottom are 4 buttons for the actions that can be initiated for resources and resource groups: Start, Stop, Restart and Migrate. Depending on the selected resources or resource groups different buttons will be enabled. A resource can only be started when it is offline and only stopped when it isn’t offline. Offline resources can’t be restarted and only resource groups with running resources can be migrated. When nothing is selected the available actions will apply to all resources. In case of the Start button the button can be simply clicked to start a resources on the current node of the resource group or the preferred node if the resource group has no current node. Alternatively the down arrow can be clicked to open a drop-down menu to select the specific node a resource or resource group should be started on.

Clicking an action will add the selected action to the Action Log on the right. Depending on the automation settings for manual actions, see next chapter for details, the action will wait for confirmation or immediately execute.

Each action in the Action Log box has a timestamp showing the time of the last activity for that action as well as a status LED showing the current status. Actions that are awaiting confirmation have a dark blue LED which turns light blue when the action is confirmed but waiting on other actions to complete first. A yellow LED is shown when an action is currently running, which turns green on success and red on failure.

To confirm an action for execution select the desired action and click the Confirm button. Confirming an action will also confirm all present and future children on an action. If an action should not be executed click the Reject button instead. Rejecting an action marks the action as failed, which means any parent action will also fail. This does not affect siblings though. When no action is selected the Confirm All and Reject All buttons will affect all actions.

Once an action has been confirmed it will start executing commands. Details of the execution can be seen by increasing the Log Level using the drop-down menu. The highest level, error, only shows critical error messages. Next lower is the info level, which also shows commands being started and their return code. The stderr and stdout levels include the output of commands in the display and can show important details of why a command failed. Last the debug and trace levels are there for debugging the code in case it misbehaves and will be useless to most users. Please ignore. Once a level has been selected the Action Log will update to show all log messages of the selected or higher level. This works retroactively, it is not necessary to select a log level before an action is executed to see more details.

Once an action has finished it can be removed from the display by clicking the Clear button. If no action is selected the Clear All button clears all finished actions. Actions that have not finished are not affected. In case of a failed action this will also remove the action from the server. Successfull actions are automatically removed from the server on completion to preserve memory but remains visible in the Action log as long as it remains open. Closing and opening the HA Status window will not show actions already removed from the server.

The point of an HA setup is to ensure availability when things go wrong. So far all actions have been initiated manually and generally required confirmation by the user. That defeats the purpose of an HA setup and is more like a cold standby. But that is because the automation settings start in maintainance mode when a cluster is started cold. It is important to realize that in the QHAS design there is no difference in how the system behaves in different automation modes. What automation means in a QHAS context is to automate the confirmation step in the logic. That and only that.

QHAS will always create actions to restart failed resources, to migrate resource groups and to KOTON nodes beyond recovery regardless of the automation settings. The automation settings will let you choose which of those proposed actions will require manual confirmation and which will be executed automatically.

Automation settings for actions are split into 3 groups: Core Resources, General Resource and Manual Actions.

Manual Actions covers any action created by the user via the qluman-qt GUI. It’s a protection against accidentally pressing the wrong button or having the wrong Resource or Resource Group selected. Per default all but the start action requires a confirmation of the action in the action log. If having to confirm every action becomes tiresome after a while this where to change that.

The other two groups only relate to repair actions done when monitoring fails to confirm a resource is working. As such it has no options for start and stop as a repair will never initiate those actions except as part of a larger operation. The Core Resources group governs any repair for the core Resource Group while all other Resource Groups are governed by the General Resources settings. The core Resource Group is essential to the working of the cluster. Without it nodes can not boot and won’t get their configuration at boot. It therefore uses separate settings from other Resource Groups.

Another important automation setting is Mail Alerts. When activated e-mails are send whenever an repair action is proposed that requires confirmation or whenever an action finishes. For the confirmation email the proposed actions are listed. For a finished email the result of the action, success or failure, is included in the subject as well all the steps taken. The main mail only includes the top level output for all actions while attachments are included listing all the output and even debug information of the internal workings. The contents match what is available in the Action Log when qluman-qt is running.

Automation settings can be changed individually by clicking the respective LED buttons. The collor will change to reflect the indended setting. But the the LEDs will be greyed out because the setting does not immediately take effect. One reason for this is that it allows editing presets as will be shown below. To make the settings take effect the Activate button must be clicked. Note that the title for the Automation section changes when changes are pending or when settings are active that do not reflect a saved preset.

Automation settings can also be set to a preset selected from the list of saved presets. After selecting a preset click the Activate to make the settings take effect.

Presets can be edited or new presets can be created simply by adjusting the LED buttons for the automation settings and finally clicking the Save button. Saving the presets under the current name will overwrite the preset and changing the name will create a new preset. The automation settings can also be restored to the saved preset by clicking the undo button. Finally presets can be removed by clikcing the Delete button.

The start of a resource can fail on a node for whatever reason. The start of the resource is then retried up to 5 times for manual actions or up to the respawn limit for automatic actions, i.e. when a restart is triggered due to monitoring errors. Before each start retry the resource is stopped to clean up any remains from the previous try. The repeated attempts can be seen in the Action Log when the log level is raise to at least /info/.

If the resource refuses to start too often the start attempt is marked as failed and a repair action is initiated, or an ongoing repair action is extended, to migrate the resource group to the alternate node. The node with start failures is also marked inactive to prevent resources from migrating to a probably damaged node. Resource groups other than the one with the failed resource are left running on the node though for as long as monitoring detects no errors.

The final result of the repair action, assuming a successful repair, will be that the affected resource group now runs on the alternate node and the action is marked as success.

Similar to the start of a resource stopping a resource can also fail and just like start the stop action is atempted multiple times and can be seen in the Action Log when the log level is raise to at least /info/. A failure to stop is more serious than start though as then there is no way to completely shut down the resource so it can be safely started on the alternate node. The only choice left is to power cycle the problematic node via KOTON action.

Before taking that step any other resources running on the same node are stopped, if possible, as a clean shutdown is preferable to simply turning of power. If the other resources also fail to stop this can cause a large delay if each stop action fails only with a timeout. This should be rare as in most cases the stop action either succeeds or returns quickly with an error.

Note that when the repair action has finished the resource group now runs on the alternate node and the resource that failed to stop is not running. This is because the stop failure occured on a manual stop, the user asked for the resource to be stopped so it remains offline after the repair. If the stop action had failed during a restart or repair action then the resource would have been brought back up on the other node as there was no intention of keeping the resource offline.

Each resource is periodically checked to assert it’s still functional. When the check reports an error or fails complete in a timely manner a repair action is created for the failed resource. In this case the monitoring has detected that the dhcp-scanner is no longer running. If the automation settings allow it the next steps run on their own. Otherwise, as shown here, the repair stops and waits for the admin to confirm the planned action.

The first step is to stop the resource. This is done to clean up any remnants of the resource that might still be running, to make sure the resource is completely offline and in a clean state. This will also honor the dependencies of resources, so resources requiring dhcp-scanner (none here) are stopped as well. After that the resource and dependencies are started again to restore the system to a functional state.

If there is no larger underlying problem the repair will finish with the resources online on the same node again.

QHAS normally requires both headnodes to be running before it allows starting the core resources. The reason for this is that resources must never be started on two hosts at the same time to prevent data corruption in filesystems or databases. So the requirement isn’t so much that both headnodes are running but that the headnodes knows no resources are running on the other headnode.

There are two ways to achieve this. The first is to tell one headnode that the other headnode is offline using

This marks the head-2 node as offline, promising that no resources are running on that node. This should not be promised lightly but ZFS is configured in multihost mode as additional check to prevent mounting the zpool core resource on both heads at the same time. Starting core twice would just fail to start.

The other way is to wait for the timeout for connecting both heads to expire. The running headnode will then propose an action to take over as primary headnode from the other headnode and to KOTON said headnode. This action can be confirmed either via qluman-ha-cli or qluman-qt.

Once the other headnode is marked offline the core Resource Group can be started like normal and the cluster will run in a degraded mode. QHAS will still try to restart resource on failure but it won’t be able to recover from an error that isn’t resolved with a simple resource restart. Without a second headnode it can’t migrate resources. This does not affect other pairs of nodes though. Each Resource Group is independent. As long as both hosts for a Resource Group are online migration and koton can happen.