Phoronix Test Suite (PTS) is a benchmark orchestration tool which runs multiple configurable benchmark CLIs and extracts their results. This article explains the two most important core concepts of PTS, test profiles and test suites, and how to build them yourself. The detailed steps of installing and running tests are also explained, as well as how to build your own container image that ships with PTS and all required dependencies.
Benchmark series
This article is part of a multi-part series about benchmarking virtual cloud hardware:
- Part 1: Benchmarking virtual cloud hardware using 6 great tools
- Part 2: Hardware benchmarks with Phoronix Test Suite (this article)
- Part 3: Automated hardware benchmarks in Kubernetes
There is also the related Performance benchmark of over 30 Azure VMs in AKS you might find interesting.
Introduction to benchmarking with PTS
The Phoronix Test Suite (PTS) is an open-source, cross-platform benchmarking software, written in PHP. PTS does not ship with any benchmarking code per se, but PTS is actually a meta-level framework. It is basically a “benchmark orchestrator” which allows running versioned, reproducible tests that build upon a huge library of third party test software (including all of those benchmark tools I presented in part1 of this series, e.g. Sysbench, fio, or Geekbench). You can also build your own customized tests.
Using PTS greatly simplifies the process of running many (reproducible) benchmarks and collecting their results in an uniform way. Without PTS, you would have to spend a lot of time building container images (that contain your desired benchmark tools) and writing custom code that collects and parses the results of the benchmark tools, which all have different output formats.
In this article, I explain PTS and its core concepts in detail. I provide tips how you can build your own tests and test suites, and how to build your own Docker image that runs a customized PTS test suite. You can find the image I personally use for my benchmarks here on GitHub.
Installation of PTS
Installing PTS is as simple as cloning the PTS Git repository and running the phoronix-test-suite
Linux-Bash (or Windows-Batch) script. This script detects any missing system packages (e.g. a PHP interpreter) and suggest the commands needed to install them. Further below I show how to easily build a customized Docker image for PTS.
CLI commands of PTS
PTS comes with over 130 CLI commands, but in practice, I only needed a few of them:
install <space-separated names of tests or test suites>
: ensures that all benchmark tool binaries (and corresponding test data) required by the specified tests or test suites are available locally (the detailed steps are explained below in section “Test profiles in detail”). Running this command does not require any interaction with the user.- Example:
./phoronix-test-suite install pts/fio
(to install the newest version of thepts/fio
test)
or./phoronix-test-suite install pts/fio-2.1.0
(to install a pinned version)
- Example:
run <test or test suite>
: executes a test (which must have been previously installed) in interactive mode: “interactive” means that PTS prompts you with various questions, e.g. configuration parameters for a test, and a name+description for the results.benchmark <test or test suite>
: a simple alias for
“install <test or test suite> && run <test or test suite>
“batch-setup
: interactive command which asks you what prompts (related to the test results) should be omitted, when using thebatch-run
commandbatch-run <test or test suite>
: likerun
, but omits those prompts that you disabled viabatch-setup
build-suite
: shows a series of interactive prompts with which you compose your own test suite (more details below)result-file-to-csv <result name>
: assuming a certain test (suite) finished (with the results stored under the name<result name>
), this command creates a CSV-formatted file with the results
Core concepts of PTS
In a nutshell, you need to understand these two core concepts of PTS:
- Test profile: defines a test which calls a specific benchmark CLI tool (e.g. FIO, which I presented in part 1. A test profile (like this FIO test profile example) defines:
- (optional) Linux/Windows system packages to install (e.g. via
apt-get
) - (optional) file(s) to download from some URL (e.g. the benchmark tool’s source code as .tar.gz archive)
- (optional) an installation shell script, which e.g. compiles the benchmark tool (if its sources were downloaded)
- how to parse the results produced by the benchmark tool
- (optional) one or more configuration options (in case of FIO, e.g. which type of I/O pattern to test, like random read or sequential write); PTS prompts the user for the concrete values at run-time
- (optional) Linux/Windows system packages to install (e.g. via
- Test suite: a collection of multiple tests, each with specific hard-coded configuration options. You can think of a suite as a “batch mode” for running one or more tests profiles, with predetermined test configuration options for each test profile. See here for an example (which calls various test profiles, e.g.
pts/rodinia
orpts/blender
, setting configuration options in the<Arguments>
block where applicable). Note that a test suite itself does not have any configuration options.
There are hundreds of readily available test profiles and suites, which you can discover and inspect on https://openbenchmarking.org/. You could also inspect and discover them using the PTS CLI (e.g. with the list-available-tests
/list-available-suites
CLI commands), but the openbenchmarking.org website is much easier to navigate, supports search, and offers more details.
Each test profile or suite is versioned, and you can inspect the source code either right on the https://openbenchmarking.org/ site (click on the “View source” link in the “Revision history” section of a test, e.g. fio), or on GitHub (see test profiles and test suites repos).
Let’s look at test profiles in more detail, and how to create test profiles / suites.
Test profiles in detail
A test profile is a set of XML + shell script files. While you could use the create-test-profile
CLI command to create a new test profile from scratch, it is usually easier to simply make a copy of an existing test profile, and modify that copy.
Let’s take a look at a specific version of the pts/fio
test to better understand the file contents of a test profile. That web page shows a list of 7 files (downloads.xml
, install.sh
, etc.), so let’s see what each file does:
test-definition.xml
: defines aspects like the test name, description, configuration options, etc. Some important hints:<TimesToRun>
configures how many times PTS will repeatedly run the test. PTS then computes the average value of all runs and reports it as result. Most test profiles I examined use 3 as value.<Executable>
is the name of the shell script or binary that PTS will try to call when running an installed test. It is the job of theinstall.sh
script to ensure that this file exists.<TestType>
is self-explanatory – it should be either very hardware-specific (value = Disk, Graphics, Network, or Processor) or generic (value = System)<ExternalDependencies>
: optional list of system packages to install, e.g. using the OS-native package manager. You need to use names from a predefined catalog (see e.g. here for generic packages, here for Ubuntu-specific packages, here for Windows-specific packages).- Alternatively, you can use the newer
<SystemDependencies>
feature, which only works on select platforms. At the time of writing, the Linux distros Ubuntu/Arch/Fedora/OpenSUSE are supported, as well as Windows (to a very limited extent), see the files in this directory. In case of Linux, the idea is that you specify the names of Header or .so dynamic libraries (or binaries which should be onPATH
), see the relevant code). PTS then uses tools likeapt-file
,pkg
,dnf
orzypper
(depending on the Linux distro) to resolve which packages it needs to install. For Windows, the support is so limited (see relevant code) that<SystemDependencies>
is unlikely to be helpful.
- Alternatively, you can use the newer
<TestSettings>
is optional, it sets the test’s configuration options:- Typically, you will configure an array of one or more
<Option>
entries, for which the user chooses concrete values when running the test. Each<Option>
has a<DisplayName>
,<Identifier>
, optional<Message>
, optional<DefaultEntry>
and a<Menu>
object. Each<Menu>
has one or more<Entry>
objects (each with a<Name>
and a<Value>
and optional<Message>
). - You can optionally create an
<Default><Arguments>--some-arg=value</Arguments><PostArguments>--some-arg=value</PostArguments></Default>
block with CLI arguments that are always given to the underlying tool. When PTS calls the test, it provides the arguments in<Arguments>
first, then the arguments from the<Option>
s, then the<PostArguments>
arguments.
- Typically, you will configure an array of one or more
downloads.xml
: optionally defines downloads of arbitrary files. Only the<URL>
and<FileName>
is required, but you can optionally provide various verification parameters, or provide multiple downloads, each one specific for a certain platform/OSinstall.sh
: This script runs once, during the installation process of a test profile. Typically, its job is to extract downloads defined in thedownloads.xml
file, which often contain source code. After extraction,install.sh
compiles the sources to a binary. Finally,install.sh
typically creates an executable shell script with the name of<Executable>
(defined in thetest-definition.xml
explained above), which is called when running a test. That shell script calls the just-compiled binary and contains literal shell variables using their numbered index ($1
,$2
, …), which are set to the values of the<TestSettings> <Option>
s you defined intest-definition.xml
.- Note: for tests that are compatible with Microsoft Windows, the file is instead called
install_windows.sh
and it is run in a Cygwin interpreter.
- Note: for tests that are compatible with Microsoft Windows, the file is instead called
pre.sh
/post.sh
/interim.sh
: when running a test, PTS callspre.sh
before the very first test run, PTS callspost.sh
after the last test run, and PTS callsinterim.sh
between runs (if the are multiple runs). On Windows, the files end with “_windows.sh
”, e.g.pre_windows.sh
. These scripts are typically used to clean up temporary files.results-definition.xml
: contains one or more parser definitions for the results. By looking at various example test profiles, you should have no problem understanding how the parsing mechanism works.
It also helps to understand what exactly happens when you run a command like “./phoronix-test-suite install pts/fio
“, so let’s look at the detailed steps. Note that I use <pts_root>
as a placeholder, which could resolve to e.g. ~/.phoronix-test-suite
or /var/lib/phoronix-test-suite
, depending on how and where you run PTS (or whether this path is overwritten by the environment variable PTS_USER_PATH
).
The steps of installing a test are:
- PTS downloads the test profile (XML+shell scripts) to
<pts_root>/test-profiles/pts/fio-x.y.z
(unless already present/cached) - PTS installs the test profile:
- PTS installs
<ExternalDependencies>
and<SystemDependencies>
, if any are defined - PTS downloads files from
downloads.xml
to<pts_root>/installed-tests/pts/fio-x.y.z/
- PTS calls
install.sh
. The installed tests can be found in directory<pts_root>/installed-tests/pts/fio-x.y.z/
. In case ofpts/fio
,install.sh
produces a shell script namedfio-run
in that folder.
- PTS installs
The following happens when you run the test, e.g. via “./phoronix-test-suite run pts/fio
“:
- Because
pts/fio
is a test (not a test suite) with configuration options, PTS first shows you a series of prompts in which you configure the values for the configuration options (see the<Option>
entries in thetest-definition.xml
file of fio’s test profile). From your responses to these prompts, PTS assembles the CLI arguments.- For instance, if you answer the prompts with “Random Read”, “Linux AIO”, “Yes”, “4KB”, “1”, PTS will call the
fio-run
script (in step 3) as follows:fio-run randread libaio 1 4k 1
- For instance, if you answer the prompts with “Random Read”, “Linux AIO”, “Yes”, “4KB”, “1”, PTS will call the
- Because the
run
CLI command is interactive (in contrast to thebatch-run
command), PTS prompts you with questions regarding where to store the results of the test - Next, PTS actually runs the test, typically running it multiple times, storing the raw outputs in
<pts_root>/test-results/<result-name-prompted-in-step-2>
.- PTS runs the tests at least
<TimesToRun>
times, unless you overwrite it via environment variableFORCE_TIMES_TO_RUN
. PTS may even run the test more often than<TimesToRun>
times, if theDynamicRunCount
setting is enabled (which is true by default), see the docs for more details. - As explained above, PTS calls the shell scripts in the following order:
pre.sh
<Executable with args>
+interim.sh
, repeating these 2 script calls for each test run (forpts/fio
,<Executable with args>
would e.g. be “fio-run randread libaio 1 4k 1
” from our example above)post.sh
- PTS runs the tests at least
- PTS extracts the results from the raw outputs of the underlying benchmark tool, using the parser(s) defined in the
results-definition.xml
file - PTS prompts you whether to print the results to the console. If you say yes, PTS also prints a comparison of your score to benchmark results made by others, retrieved from https://openbenchmarking.org/ (if available).
- PTS prompts you whether to upload the results to https://openbenchmarking.org/
Note that PTS only does the last 2 prompts because the run
CLI command is interactive.
Creating your own test profile
In general, I recommend that you look for an existing test profile that is already quite similar to the one you want to write, and then create a copy of it. The advantage of this approach is that the existing test profile already provides many values you can re-use. Also, understanding the already-filled files by example works rather well.
You can find the existing test profiles in “<pts_root>/test-profiles/[system|pts]/<test-identifier>-<version>
“. You should copy a test profile to “<pts_root>/test-profiles/local/<test-identifier>-<version>
“, so that you can run it via “./phoronix-test-suite run local/<test-identifier>-<version>
“.
PTS also comes with a create-test-profile
CLI command. This command helped me get a better understanding of the meanings of some of the entries in test-definition.xml
. However, it is not sufficient on its own to create a test profile, because the other generated files (e.g. results-definition.xml
) lack details and examples.
While you are tuning your test profile files, I recommend you set the environment variable FORCE_TIMES_TO_RUN
to “1”, to reduce the waiting time (which is particularly helpful while you are working on the result parsers). You might also find the debug-install
or debug-result-parser
CLI commands helpful.
Creating your own test suite
While you could create test suites the same approach as I presented it for test profiles (create create a copy of an existing test suite which is stored in “<pts_root>/test-suites/[system|pts]/<test-suite-identifier>-<version>
), I instead recommend the build-suite
CLI command of PTS.
The build-suite
CLI command first prompts you for some meta-data for the suite (such as the test suite name or description) and then lets you add one or more test profiles, prompting you for the values of possibly-existing configuration options.
If desired, you can manually edit the resulting test suite (which is stored in “<pts_root>/test-suites/local/<test-suite-name>/suite-definition.xml
) after the build-suite
CLI command has completed. You can then run the suite with
“./phoronix-test-suite run <test-suite-name>
“, or (more explicitly) with
“./phoronix-test-suite run local/<test-suite-name>
“.
Batch mode of PTS
If you want to run many benchmarks, you need the benchmark tool (like PTS) to work in a fully unattended fashion, where it does not require any interactively prompted values from you during the benchmark. With PTS, this can be achieved via its batch mode, which allows you to disable specific prompts (or even all prompts) that PTS would otherwise show.
Batch mode configuration
PTS requires you to initially configure its batch mode, before you can use the batch-run
CLI command to run PTS tests or suites in batch mode.
The configuration can actually be done in multiple ways:
- Via the
batch-setup
CLI command, which itself is interactive! This CLI command modifies the user configuration file, which is stored in/etc/phoronix-test-suite.xml
(if you run PTS as root user) or<pts_root>/user-config.xml
(non-root). - Manually modify the entries below
<BatchMode>
in the user configuration file - Run multiple commands such as “
./phoronix-test-suite user-config-set UploadResults=FALSE
” (see here for a complete example)
The following snippet illustrates the prompts shown by the batch-run
CLI command:
Save test results when in batch mode (Y/n)
Open the web browser automatically when in batch mode (y/N)
Auto upload the results to OpenBenchmarking.org (Y/n)
Prompt for test identifier (Y/n)
Prompt for test description (Y/n)
Prompt for saved results file-name (Y/n)
Run all test options (Y/n)
Code language: plaintext (plaintext)
To achieve a fully unattended batch mode, you would have to answer the questions accordingly. Note the value you provide for “Run all test options” only matters if you batch-run a test profile (but it is ignored if you run a test suite). Since I would generally recommend to run only test suites anyway, it does effectively not matter which option you choose.
Tips for using the batch mode
Assuming you have configured batch mode, batch-running a test suite should work without problems. However, there is one small nuisance to be solved: when you specify no for the “Prompt for saved results file-name” option (shown above), PTS auto-generates the filename/folder of the saved results, using the <YYYY-MM-DD>-<HHMM>
date format pattern. This is a problem if you want to automatically extract the final results as CSV, via a command such as “./phoronix-test-suite result-file-to-csv <result-name>
”, since you do not know the exact <result-name>
.
Fortunately, you can set the environment variable TEST_RESULTS_NAME
to a value of your choice to solve this problem. The following example runs a local test suite and generates a file that contains only the final results formatted as CSV:
#!/bin/bash
set -e
export TEST_RESULTS_NAME=myresult
./phoronix-test-suite batch-run local/my-suite
./phoronix-test-suite result-file-to-csv $TEST_RESULTS_NAME # PTS prints the path to resulting file to the console
Code language: Bash (bash)
Building a customized PTS Docker image
There is an official Docker image of PTS (which the PTS maintainer created using this script), but I don’t recommend its usage, for the following reasons:
- It is based on an outdated Ubuntu distribution version (20.04) and it is not regularly rebuilt
- Batch mode is not configured by default
- It does not contain any installed tests, and consequently running benchmarks is slower (and requires Internet access) because PTS always needs to first install the benchmark tools that are required by the tests
Instead, I recommend that you build your own, customized Docker image which has all tests and suites already baked in that you need. I’ve created my own version here, with prebuilt Intel+ARM64 images available at ghcr.io/mshekow/pts-docker-benchmark
. Feel free to use it, or base your own image on the code of my repository.
Conclusion
Once you are familiar with the core concepts of PTS and how its essential commands work, creating a test suite which benchmarks only what you need is very easy (because in most cases, you only need to create a new test suite that is based on existing test profiles). But even creating a new test profile for a not-yet-covered benchmark tool is a matter of few hours.
In the next article (part 3), I present an automation framework which tests different VMs and storages in Kubernetes.