Benchmark series part 2: Hardware benchmarks with Phoronix Test Suite

Phoronix Test Suite (PTS) is a benchmark orchestration tool which runs multiple configurable benchmark CLIs and extracts their results. This article explains the two most important core concepts of PTS, test profiles and test suites, and how to build them yourself. The detailed steps of installing and running tests are also explained, as well as how to build your own container image that ships with PTS and all required dependencies.

Introduction to benchmarking with PTS

The Phoronix Test Suite (PTS) is an open-source, cross-platform benchmarking software, written in PHP. PTS does not ship with any benchmarking code per se, but PTS is actually a meta-level framework. It is basically a “benchmark orchestrator” which allows running versioned, reproducible tests that build upon a huge library of third party test software (including all of those benchmark tools I presented in part1 of this series, e.g. Sysbench, fio, or Geekbench). You can also build your own customized tests.

Using PTS greatly simplifies the process of running many (reproducible) benchmarks and collecting their results in an uniform way. Without PTS, you would have to spend a lot of time building container images (that contain your desired benchmark tools) and writing custom code that collects and parses the results of the benchmark tools, which all have different output formats.

In this article, I explain PTS and its core concepts in detail. I provide tips how you can build your own tests and test suites, and how to build your own Docker image that runs a customized PTS test suite. You can find the image I personally use for my benchmarks here on GitHub.

Installation of PTS

Installing PTS is as simple as cloning the PTS Git repository and running the phoronix-test-suite Linux-Bash (or Windows-Batch) script. This script detects any missing system packages (e.g. a PHP interpreter) and suggest the commands needed to install them. Further below I show how to easily build a customized Docker image for PTS.

CLI commands of PTS

PTS comes with over 130 CLI commands, but in practice, I only needed a few of them:

• install <space-separated names of tests or test suites>: ensures that all benchmark tool binaries (and corresponding test data) required by the specified tests or test suites are available locally (the detailed steps are explained below in section “Test profiles in detail”). Running this command does not require any interaction with the user.
  • Example: ./phoronix-test-suite install pts/fio (to install the newest version of the pts/fio test)
    or ./phoronix-test-suite install pts/fio-2.1.0 (to install a pinned version)
• run <test or test suite>: executes a test (which must have been previously installed) in interactive mode: “interactive” means that PTS prompts you with various questions, e.g. configuration parameters for a test, and a name+description for the results.
• benchmark <test or test suite>: a simple alias for
  “install <test or test suite> && run <test or test suite>“
• batch-setup: interactive command which asks you what prompts (related to the test results) should be omitted, when using the batch-run command
• batch-run <test or test suite>: like run, but omits those prompts that you disabled via batch-setup
• build-suite: shows a series of interactive prompts with which you compose your own test suite (more details below)
• result-file-to-csv <result name>: assuming a certain test (suite) finished (with the results stored under the name <result name>), this command creates a CSV-formatted file with the results

Core concepts of PTS

In a nutshell, you need to understand these two core concepts of PTS:

• Test profile: defines a test which calls a specific benchmark CLI tool (e.g. FIO, which I presented in part 1. A test profile (like this FIO test profile example) defines:
  • (optional) Linux/Windows system packages to install (e.g. via apt-get)
  • (optional) file(s) to download from some URL (e.g. the benchmark tool’s source code as .tar.gz archive)
  • (optional) an installation shell script, which e.g. compiles the benchmark tool (if its sources were downloaded)
  • how to parse the results produced by the benchmark tool
  • (optional) one or more configuration options (in case of FIO, e.g. which type of I/O pattern to test, like random read or sequential write); PTS prompts the user for the concrete values at run-time
• Test suite: a collection of multiple tests, each with specific hard-coded configuration options. You can think of a suite as a “batch mode” for running one or more tests profiles, with predetermined test configuration options for each test profile. See here for an example (which calls various test profiles, e.g. pts/rodinia or pts/blender, setting configuration options in the <Arguments> block where applicable). Note that a test suite itself does not have any configuration options.

There are hundreds of readily available test profiles and suites, which you can discover and inspect on https://openbenchmarking.org/. You could also inspect and discover them using the PTS CLI (e.g. with the list-available-tests/list-available-suites CLI commands), but the openbenchmarking.org website is much easier to navigate, supports search, and offers more details.

Each test profile or suite is versioned, and you can inspect the source code either right on the https://openbenchmarking.org/ site (click on the “View source” link in the “Revision history” section of a test, e.g. fio), or on GitHub (see test profiles and test suites repos).

Let’s look at test profiles in more detail, and how to create test profiles / suites.

Test profiles in detail

A test profile is a set of XML + shell script files. While you could use the create-test-profile CLI command to create a new test profile from scratch, it is usually easier to simply make a copy of an existing test profile, and modify that copy.

Let’s take a look at a specific version of the pts/fio test to better understand the file contents of a test profile. That web page shows a list of 7 files (downloads.xml, install.sh, etc.), so let’s see what each file does:

• test-definition.xml: defines aspects like the test name, description, configuration options, etc. Some important hints:
  • <TimesToRun> configures how many times PTS will repeatedly run the test. PTS then computes the average value of all runs and reports it as result. Most test profiles I examined use 3 as value.
  • <Executable> is the name of the shell script or binary that PTS will try to call when running an installed test. It is the job of the install.sh script to ensure that this file exists.
  • <TestType> is self-explanatory – it should be either very hardware-specific (value = Disk, Graphics, Network, or Processor) or generic (value = System)
  • <ExternalDependencies>: optional list of system packages to install, e.g. using the OS-native package manager. You need to use names from a predefined catalog (see e.g. here for generic packages, here for Ubuntu-specific packages, here for Windows-specific packages).
    • Alternatively, you can use the newer <SystemDependencies> feature, which only works on select platforms. At the time of writing, the Linux distros Ubuntu/Arch/Fedora/OpenSUSE are supported, as well as Windows (to a very limited extent), see the files in this directory. In case of Linux, the idea is that you specify the names of Header or .so dynamic libraries (or binaries which should be on PATH), see the relevant code). PTS then uses tools like apt-file, pkg, dnf or zypper (depending on the Linux distro) to resolve which packages it needs to install. For Windows, the support is so limited (see relevant code) that <SystemDependencies>  is unlikely to be helpful.
  • <TestSettings> is optional, it sets the test’s configuration options:
    • Typically, you will configure an array of one or more <Option> entries, for which the user chooses concrete values when running the test. Each <Option> has a <DisplayName>, <Identifier>, optional <Message>, optional <DefaultEntry> and a <Menu> object. Each <Menu> has one or more <Entry> objects (each with a <Name> and a <Value> and optional <Message>).
    • You can optionally create an <Default><Arguments>--some-arg=value</Arguments><PostArguments>--some-arg=value</PostArguments></Default> block with CLI arguments that are always given to the underlying tool. When PTS calls the test, it provides the arguments in <Arguments> first, then the arguments from the <Option>s, then the <PostArguments> arguments.
• downloads.xml: optionally defines downloads of arbitrary files. Only the <URL> and <FileName> is required, but you can optionally provide various verification parameters, or provide multiple downloads, each one specific for a certain platform/OS
• install.sh: This script runs once, during the installation process of a test profile. Typically, its job is to extract downloads defined in the downloads.xml file, which often contain source code. After extraction, install.sh compiles the sources to a binary. Finally, install.sh typically creates an executable shell script with the name of <Executable> (defined in the test-definition.xml explained above), which is called when running a test. That shell script calls the just-compiled binary and contains literal shell variables using their numbered index ($1, $2, …), which are set to the values of the <TestSettings> <Option>s you defined in test-definition.xml.
  • Note: for tests that are compatible with Microsoft Windows, the file is instead called install_windows.sh and it is run in a Cygwin interpreter.
• pre.sh / post.sh / interim.sh: when running a test, PTS calls pre.sh before the very first test run, PTS calls post.sh after the last test run, and PTS calls interim.sh between runs (if the are multiple runs). On Windows, the files end with “_windows.sh”, e.g. pre_windows.sh. These scripts are typically used to clean up temporary files.
• results-definition.xml: contains one or more parser definitions for the results. By looking at various example test profiles, you should have no problem understanding how the parsing mechanism works.

It also helps to understand what exactly happens when you run a command like “./phoronix-test-suite install pts/fio“, so let’s look at the detailed steps. Note that I use <pts_root> as a placeholder, which could resolve to e.g. ~/.phoronix-test-suite or /var/lib/phoronix-test-suite, depending on how and where you run PTS (or whether this path is overwritten by the environment variable PTS_USER_PATH).

The steps of installing a test are:

1. PTS downloads the test profile (XML+shell scripts) to <pts_root>/test-profiles/pts/fio-x.y.z (unless already present/cached)
2. PTS installs the test profile:
   1. PTS installs <ExternalDependencies> and <SystemDependencies>, if any are defined
   2. PTS downloads files from downloads.xml to <pts_root>/installed-tests/pts/fio-x.y.z/
   3. PTS calls install.sh. The installed tests can be found in directory <pts_root>/installed-tests/pts/fio-x.y.z/. In case of pts/fio, install.sh produces a shell script named fio-run in that folder.

The following happens when you run the test, e.g. via “./phoronix-test-suite run pts/fio“:

1. Because pts/fio is a test (not a test suite) with configuration options, PTS first shows you a series of prompts in which you configure the values for the configuration options (see the <Option> entries in the test-definition.xml file of fio’s test profile). From your responses to these prompts, PTS assembles the CLI arguments.
   • For instance, if you answer the prompts with “Random Read”, “Linux AIO”, “Yes”, “4KB”, “1”, PTS will call the fio-run script (in step 3) as follows: fio-run randread libaio 1 4k 1
2. Because the run CLI command is interactive (in contrast to the batch-run command), PTS prompts you with questions regarding where to store the results of the test
3. Next, PTS actually runs the test, typically running it multiple times, storing the raw outputs in <pts_root>/test-results/<result-name-prompted-in-step-2>.
   • PTS runs the tests at least <TimesToRun> times, unless you overwrite it via environment variable FORCE_TIMES_TO_RUN. PTS may even run the test more often than <TimesToRun> times, if the DynamicRunCount setting is enabled (which is true by default), see the docs for more details.
   • As explained above, PTS calls the shell scripts in the following order:
     • pre.sh
     • <Executable with args> + interim.sh , repeating these 2 script calls for each test run (for pts/fio, <Executable with args> would e.g. be “fio-run randread libaio 1 4k 1” from our example above)
     • post.sh
4. PTS extracts the results from the raw outputs of the underlying benchmark tool, using the parser(s) defined in the results-definition.xml file
5. PTS prompts you whether to print the results to the console. If you say yes, PTS also prints a comparison of your score to benchmark results made by others, retrieved from https://openbenchmarking.org/ (if available).
6. PTS prompts you whether to upload the results to https://openbenchmarking.org/

Note that PTS only does the last 2 prompts because the run CLI command is interactive.

Creating your own test profile

In general, I recommend that you look for an existing test profile that is already quite similar to the one you want to write, and then create a copy of it. The advantage of this approach is that the existing test profile already provides many values you can re-use. Also, understanding the already-filled files by example works rather well.

You can find the existing test profiles in “<pts_root>/test-profiles/[system|pts]/<test-identifier>-<version>“. You should copy a test profile to “<pts_root>/test-profiles/local/<test-identifier>-<version>“, so that you can run it via “./phoronix-test-suite run local/<test-identifier>-<version>“.

PTS also comes with a create-test-profile CLI command. This command helped me get a better understanding of the meanings of some of the entries in test-definition.xml. However, it is not sufficient on its own to create a test profile, because the other generated files (e.g. results-definition.xml) lack details and examples.

While you are tuning your test profile files, I recommend you set the environment variable FORCE_TIMES_TO_RUN to “1”, to reduce the waiting time (which is particularly helpful while you are working on the result parsers). You might also find the debug-install or debug-result-parser CLI commands helpful.

Creating your own test suite

While you could create test suites the same approach as I presented it for test profiles (create create a copy of an existing test suite which is stored in  “<pts_root>/test-suites/[system|pts]/<test-suite-identifier>-<version>), I instead recommend the build-suite CLI command of PTS.

The build-suite CLI command first prompts you for some meta-data for the suite (such as the test suite name or description) and then lets you add one or more test profiles, prompting you for the values of possibly-existing configuration options.

If desired, you can manually edit the resulting test suite (which is stored in  “<pts_root>/test-suites/local/<test-suite-name>/suite-definition.xml) after the build-suite CLI command has completed. You can then run the suite with
“./phoronix-test-suite run <test-suite-name>“, or (more explicitly) with
“./phoronix-test-suite run local/<test-suite-name>“.

Batch mode of PTS

If you want to run many benchmarks, you need the benchmark tool (like PTS) to work in a fully unattended fashion, where it does not require any interactively prompted values from you during the benchmark. With PTS, this can be achieved via its batch mode, which allows you to disable specific prompts (or even all prompts) that PTS would otherwise show.

Batch mode configuration

PTS requires you to initially configure its batch mode, before you can use the batch-run CLI command to run PTS tests or suites in batch mode.

The configuration can actually be done in multiple ways:

• Via the batch-setup CLI command, which itself is interactive! This CLI command modifies the user configuration file, which is stored in /etc/phoronix-test-suite.xml (if you run PTS as root user) or <pts_root>/user-config.xml (non-root).
• Manually modify the entries below <BatchMode> in the user configuration file
• Run multiple commands such as “./phoronix-test-suite user-config-set UploadResults=FALSE” (see here for a complete example)

The following snippet illustrates the prompts shown by the batch-run CLI command:

Save test results when in batch mode (Y/n) 
Open the web browser automatically when in batch mode (y/N)
Auto upload the results to OpenBenchmarking.org (Y/n)
Prompt for test identifier (Y/n)
Prompt for test description (Y/n)
Prompt for saved results file-name (Y/n)
Run all test options (Y/n)Code language: plaintext (plaintext)

To achieve a fully unattended batch mode, you would have to answer the questions accordingly. Note the value you provide for “Run all test options” only matters if you batch-run a test profile (but it is ignored if you run a test suite). Since I would generally recommend to run only test suites anyway, it does effectively not matter which option you choose.

Tips for using the batch mode

Assuming you have configured batch mode, batch-running a test suite should work without problems. However, there is one small nuisance to be solved: when you specify no for the “Prompt for saved results file-name” option (shown above), PTS auto-generates the filename/folder of the saved results, using the <YYYY-MM-DD>-<HHMM> date format pattern. This is a problem if you want to automatically extract the final results as CSV, via a command such as “./phoronix-test-suite result-file-to-csv <result-name>”, since you do not know the exact <result-name>.

Fortunately, you can set the environment variable TEST_RESULTS_NAME to a value of your choice to solve this problem. The following example runs a local test suite and generates a file that contains only the final results formatted as CSV:

#!/bin/bash
set -e
export TEST_RESULTS_NAME=myresult
./phoronix-test-suite batch-run local/my-suite
./phoronix-test-suite result-file-to-csv $TEST_RESULTS_NAME  # PTS prints the path to resulting file to the consoleCode language: Bash (bash)

Building a customized PTS Docker image

There is an official Docker image of PTS (which the PTS maintainer created using this script), but I don’t recommend its usage, for the following reasons:

• It is based on an outdated Ubuntu distribution version (20.04) and it is not regularly rebuilt
• Batch mode is not configured by default
• It does not contain any installed tests, and consequently running benchmarks is slower (and requires Internet access) because PTS always needs to first install the benchmark tools that are required by the tests

Instead, I recommend that you build your own, customized Docker image which has all tests and suites already baked in that you need. I’ve created my own version here, with prebuilt Intel+ARM64 images available at ghcr.io/mshekow/pts-docker-benchmark. Feel free to use it, or base your own image on the code of my repository.

Conclusion

Once you are familiar with the core concepts of PTS and how its essential commands work, creating a test suite which benchmarks only what you need is very easy (because in most cases, you only need to create a new test suite that is based on existing test profiles). But even creating a new test profile for a not-yet-covered benchmark tool is a matter of few hours.

In the next article (part 3), I present an automation framework which tests different VMs and storages in Kubernetes.