The Rosie/RPL project has a reworked Makefile along with other changes to support a build process that is better for people building packages (to make installing Rosie easier). This post describes those changes.

A build directory

Currently, Rosie is distributed only in source code form. Nothing inhibits a pre-compiled distribution except for some volunteer help to create such a package.

When someone wants to package Rosie, whether in source code or compiled form, it helps to have a very simply “system install” process, namely one that merely copies files from a local build directory to some system location like /usr/local. Formerly, running make install would do other things as well. Now, the compilation step puts the Rosie executable and librosie into a local directory named build. The only actions performed by make install are to copy files from build to their destination.

Build phases

The build is now implemented in the following phases, which are accessible by their own make targets.

  • make fetch retrieves git submodules, so that all files needed to build are available locally. All remaining steps can now happen without network access.

  • make compile compiles Rosie, creating the executable, the dynamic shared library librosie, and a static version as well. Since the front-end of the RPL compiler is written in Lua, those source files are also compiled during this phase into .luac files. All compilation results are stored into the local build directory.

  • make copy_to_build copies into the build directory the files (like documentation, and the standard rpl library) that do not need compilation.

  • make check does a very basic test of the executable in the build directory. The executable must exist, and when run with the rosie version command, must produce the expected output.

All of the phases above are executed by make with no target specified. The result is a complete installation, with all files in build. You can run the executable from the build directory, and it will use only files in the build directory to run, such as RPL files, .luac files, and the shared library librosie. To run from the build directory, the rosie executable does need to know how to find librosie – see running from the build directory below.

Finally…

  • make install simply copies from the build directory to the install destination, which defaults to /usr/local. The Makefile follows the convention that using DESTDIR=<path> on the make command line lets you choose the install location. Remember that you will need the same DESTDIR=<path> argument for make uninstall.

Running from the build directory

Prior to v1.2.2, the Rosie executable was statically linked, i.e. it did not use the shared library librosie. Now it does, and moreover it loads librosie dynamically. This is the first step in making the CLI equivalent to any other “client” (user) of the shared library. (The next step will be to move some CLI-only functionality from librosie into the rosie CLI. In the long term, librosie may be split into a small run-time library and a separate library that is needed for RPL compilation.)

On Unix systems, programs that dynamically load a shared library rely by default on the operating system to find the library by looking in a “standard” set of places, which vary from one OS distribution to another. The local build directory for any software will not be on the list of places that the OS will search, however, so we must tell the OS where to find librosie when the rosie executable in the build directory requests it. The easiest and most portable way to do this is to set the LD_LIBRARY_PATH environment variable. This can be done for a single command, so that the setting will not affect the running of any other program.

We now provide for convenience a script in the Rosie distribution directory (the directory into which Rosie is downloaded, e.g. with git clone or tar -xf). If the current working directory is the Rosie distribution directory, ./rosie will run rosie out of the build directory. That script is essentially the following, but more robustly written:

LD_LIBRARY_PATH="build/lib" build/bin/rosie "$@"

A note on start-up time

We know that rosie has a slow start-up time, because it compiles all the patterns it needs from source. For instance, starting and compiling the pattern "hi" consistently takes about 0.04 seconds:

$ /usr/bin/time -p rosie match '"hi"' /dev/null
real         0.06
user         0.04
sys          0.00

A good portion of the start-up time is that the definition of RPL itself is compiled from RPL source at startup. Rosie is self-hosting, and so the grammar of the RPL language is defined in RPL. And that grammar is 113 lines of RPL source:

$ rosie match '!{re.s* {"--"/$}}' rpl/rosie/rpl_1_3.rpl | wc -l
     113

(Note that the new-ish RPL package re has definitions for many common regex abbreviations, like s for whitespace. And recall that ! means “not looking at”.)

And when compiling the all package, which in turn requires the ts, date, time, net, num, id, and word packages, the start-up time increases to a very long 0.19 seconds.

$ /usr/bin/time -p rosie -f rpl/all.rpl match '"hi"' /dev/null
real         0.23
user         0.19
sys          0.03
$ 

The next major version of Rosie will have ahead-of-time compilation, making start-up time much faster, because pre-compiled patterns can be loaded from files cached on disk. And the pre-compiled pattern for RPL itself can be “baked in” to the Rosie executable itself. Prototyping done already suggests good results.

Returning to the question of start-up time in v1.2.2, might it be longer now that the executable dynamically loads librosie? In fact, the difference is too small to matter. Even better, whatever difference there might be due to dynamic loading is overshadowed by a tiny performance gain in v1.2.2-beta when executing actual matching tasks.

See the results of a really basic test, in which parsing 5,000 lines of log files takes about 0.01 seconds less in v1.2.2-beta than previously. I did not do a test of statistical significance, nor any carefully controlled study, so the tiny improvement may be insignificant or impossible to attribute to any specific change.

But we aim to continue improving performance, and to never regress.


We welcome feedback and contributions. Please open issues (or merge requests) on GitLab, or get in touch by email. And there is now a mailing list for general questions and discussion: https://groups.io/g/rosiepattern.


Follow the Rosie project on Twitter for announcements.