🧑‍💻 agile sysadmin

by Ferenc Erki

Minimum Viable Rex

We consider enabling graceful bootstrapping as one of our main guiding principles around Rex, the friendly automation framework.

While our How to get started with Rex page provides a good initial set of concepts, I wondered about the minimal set of features that already proves useful in practice. I find this especially interesting when using Rex from a cronjob or in a CI/CD pipeline.

Let’s see what I found through this exercise in minimalism.

Basic architecture

Rex itself can run where Perl can run, and also can manage hosts both locally and over SSH. To differentiate it from managed hosts, we call the host where Rex runs as a control host.

Rex code also stays entirely compatible with Perl code, making it extendable to support arbitrary use cases, including the use of other protocols when required.

Just like the GNU Make tool uses Makefile to describe its actions, Rex uses a Rexfile to describe its configuration, inventory, authentication, and tasks.

A task bundles related management steps together, optionally with task-specific options, like the default target host for these steps to run on.

It still proves useful to keep those details in mind, though when striving for minimalism, it also turns out we may either omit most of them to let the built-in logic try likely settings, or just provide enough hints via command-line options.

Requirements

While developing Rex we strive for keeping the requirements low and light.

On the control hosts, Rex requires Perl itself and its dependencies to run. We generally aim to support Perl versions up to 10 years. We also consider many dependencies optional, especially external tools to streamline some specific features like rsync or Augeas.

On the managed hosts, Rex attempts to detect and use Perl and core modules, though it can fall back to other methods for the most common situations. In case of remotely managed hosts, Rex also expected SSH running there and a valid account to log in. For administrative tasks, Rex may use either root directly or gain extra privileges with sudo.

Since this post focuses on the minimal approach, in the examples I will use Perl-5.40.1 to run Rex-1.16.0 on Gentoo with Net::OpenSSH backend for SSH connections, and with SSH configured for key-based authentication for my user.

Documentation

I find it always nice to keep related documentation at hand.

For Rex, that means starting with the main module documentation via perldoc Rex and the main binary documentation via man rex – and both of which also points to the available built-in commands via perldoc Rex::Commands.

On top of that, rex -h also shows the built-in help, which I reduced here for the minimal set I will cover in this post:

$ rex -h
usage:
  rex [<options>] [-H <host>] [-G <group>] <task> [<task-options>]
  rex -T[m|y|v] [<string>]

  -e     Run the given code fragment
  -H     Execute a task on the given hosts (space delimited)

  -u     Username for the ssh connection

  -d     Show debug output
  -qw    Quiet mode: only output warnings and errors

  -h     Display this help message
  -v     Display (R)?ex version

Let’s check the current Rex version quickly to make sure we run what we think we run:

$ rex -v
(R)?ex 1.16.0

Command line usage

Similarly to Perl’s -e command line switch to run one-liner programs, Rex provides its own -e to run short code fragments.

Let’s print out the standard Hello, World! message with the help of Perl’s built-in say() function:

$ rex -e 'say "Hello, World!"'
[2025-03-20 12:46:52] INFO - Running task eval-line on <local>
Hello, World!
[2025-03-20 12:46:52] INFO - All tasks successful on all hosts

On top of the output we asked for, Rex also shows the INFO log lines in green, letting us know that:

  • it runs an eval-line task on the local host (the default target for tasks)
  • everything finished successfully

Some consider that an awful lot of single and double quotes next to each other, though. Since the argument of rex -e takes any Perl code, let’s use the q() quote-like operator instead:

$ rex -e 'say q(Hello, World!)'
[2025-03-20 12:53:41] INFO - Running task eval-line on <local>
Hello, World!
[2025-03-20 12:53:42] INFO - All tasks successful on all hosts

Same result, though the code may feel easier to work with. For brevity, I will also exclude some of those logs in the further examples. The help shows a few options to hide the logs and I tend to use -qw to still keep any warnings and errors visible.

$ rex -qw -e 'say q(Hello, World!)'
Hello, World!

Run commands and manage files

While printing messages sounds interesting, we consider the core capabilities of Rex as running commands and managing files.

For example, the hostname command on Linux prints the hostname, and Rex provides the run() command to, well, run commands:

$ rex -qw -e 'say run q(hostname)'
mymachine

The file() command of Rex helps manage files:

$ rex -qw -e 'file "/tmp/hello", content => q(Hello, World!)'

Hmm, no output, though no signs of any warnings or errors. I’d like to check if the /tmp/hello file does indeed contain Hello, World! in it. On Linux, I would use the cat command for that, and Rex provides the same functionality under the same name:

$ rex -qw -e 'say cat q(/tmp/hello)'
Hello, World!

Why does Rex duplicate the cat command, when we would also call run q(cat /tmp/hello)? Because we can only expect cat, the Linux command, to exist only on Linux – while Rex may have to manage other operating systems too, where the system does not have cat. On those systems Rex abstracts away the difference, and uses other methods to achieve the same results.

Manage remote hosts

While executing ad-hoc tasks on the local machine sounds useful, many use cases involve running tasks remotely. Let’s pass a host to connect to via SSH with the -H option:

$ rex -H myserver -e 'say run q(hostname)'
[2025-03-20 19:49:16] INFO - Running task eval-line on myserver
myserver
[2025-03-20 19:49:19] INFO - All tasks successful on all hosts

Note that the INFO log output now shows running the task on myserver, and the output of the hostname command changed accordingly – it did run on myserver after all.

Also note that Rex discovered the authentication details it may use to log in to myserver. It can parse (a subset of1) the SSH configuration in ~/.ssh/config, and can also fall back to use the current local username to log in. In case we need to, we can enforce a specific user, like myuser, with -u:

$ rex -qw -H myserver -u myuser -e 'say run q(hostname)'
myserver

Since I use the Net::OpenSSH backend, it can pick up and use my SSH keys too. In case it needs to fall back to password-based authentication, or the key needs a passphrase, it will prompt for it.

Debug output

These one-liner tasks may prove difficult to debug, and the built-in debug output via -d often helps a lot to understand what happens from the perspective of Rex.

$ rex -d -e 'say run q(hostname)'

I omit the full output here as it can get verbose, though it will contain information about the following:

  • the running Rex version
  • list of command line parameters
  • internal details of the whole process, including the picked up configuration, and dependencies
  • authentication information used during execution
  • task execution details
  • list of shutdown steps

Summary

For a frugal introduction to Rex, one may find the -e command line option already enough to execute ad-hoc tasks locally, and perhaps -H to execute the same remotely, along with how to find further information in the documentation.

Adding a few more options to the repertoire, like overriding authentication details, and controlling the verbosity of the output, makes the first experience with Rex viable for the most common practical uses.


  1. I work on fully transparent support of arbitrary SSH configuration ↩