Create a build runner VM with vagrant for Windows and macOS – Part 2

Build runners, which compile and package your software, are often implemented using Docker containers. However, you cannot use Docker if you need a Windows or macOS build runner. This article presents how to create a build runner VM with Vagrant, for Windows and macOS. While the instantiation of a build VM is fully automated, it is challenging to create the list of provisioning commands that install and configure your build tools within the VM. This article is part 2 of 2, presenting several tips and recipes for common provisioning steps. Make sure to read part 1 first!

Tips and recipes for how to create a build runner VM with Vagrant

Download and extract archive files

It’s not always possible to install everything with package managers. Sometimes you need to download a zip archive and extract it.

On macOS, this is very easy, because the corresponding tools come with the system. A snippet as the one below will do the job for you:


config.vm.provision "shell", privileged: false, inline: <<-SHELL
  curl https://some-url.com/some-archive.zip -L -o local-path.zip
  unzip local-path.zip
SHELLCode language: Ruby (ruby)

The -L argument ensures that curl follows redirects. In case the archive is a tar.gz archive, you will need to use
tar xvzf archive-path.tar.gz instead of unzip.

On Windows there are no built-in tools (exe binaries) which download or extract archives. However, the Powershell offers so-called cmdlets which do the job for you. Vagrant uses the Powershell anyway for any instructions placed inside an inline shell block. The following snippet illustrates how to download and extract zip archives:

config.vm.provision "shell", privileged: true, inline: <<-SHELL
  [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
  Invoke-WebRequest https://some-url.com/some-archive.zip -OutFile local-path.zip
  Expand-Archive local-path.zip -DestinationPath "C:\\libs"
SHELLCode language: Ruby (ruby)

The first line is only required once per shell-provisioning block. It enables downloading files via HTTPS, which would otherwise fail if that line was missing. The second and third line download and extract the file respectively. Note that backslashes only need to be escaped because we are using the inline mechanism of Vagrant. If you instead placed the content inside a .ps1 file, single backslashes would suffice.

Note that the Expand-Archive cmdlet may not work on older Windows versions, e.g. Windows 8.1. If you run into problems with that command, you need to upgrade Powershell to the latest version. You can use Chocolatey for this, as the following provisioning block illustrates. Place it above the block that calls Expand-Archive, so that Powershell is updated before using the cmdlet:

config.vm.provision "shell", reboot: true, inline: 'choco install powershell -y --no-progress'Code language: JavaScript (javascript)

The reboot: true part is important, as the Powershell upgrade requires a reboot. Also, don’t forget to also install Chocolatey beforehand!

Build and use C++ libraries with CMake

If you use C++ in your project, chances are that you are using CMake to build your project, and to #include and link other native libraries used as dependencies in your project. Consequently, your build VM needs to download, build and install all those dependencies. While there are projects such as vcpkg which help you do that, they may lack support for pinning specific versions of the libraries, and always install the latest version. The following snippet illustrates the manual installation process for the ZeroMQ library, pinning the version to v4.3.3. For Windows:

config.vm.provision "shell", privileged: true, inline: <<-SHELL
  mkdir C:\\libs
  [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
  Invoke-WebRequest https://github.com/zeromq/libzmq/releases/download/v4.3.3/zeromq-4.3.3.zip -OutFile C:\\libs\\zeromq.zip
  Expand-Archive C:\\libs\\zeromq.zip -DestinationPath "C:\\libs"
  cd C:\\libs\\zeromq*
  cmake -G "Visual Studio 16 2019" -A x64 -B build64 -D ZMQ_BUILD_TESTS=OFF
  cmake --build build64 --target install --config Release
SHELLCode language: Ruby (ruby)

Similarly, for macOS:

config.vm.provision "shell", privileged: true, inline: <<-SHELL
  mkdir libs
  cd libs
  curl https://github.com/zeromq/libzmq/releases/download/v4.3.3/zeromq-4.3.3.zip -L -o zeromq.zip
  unzip zeromq.zip
  cd zeromq*
  cmake -G "Unix Makefiles" -B thebuild -D ZMQ_BUILD_TESTS=OFF
  cmake --build thebuild --target install --config Release
SHELLCode language: JavaScript (javascript)

A few notes:

  • On Windows, the "Visual Studio 16 2019" generator will find the Visual Studio Build Tools I discussed in part 1. You don’t need to install the full version of Visual Studio. On macOS, the Command Line Tools will suffice in order to use the "Unix Makefiles" generator. You only need to install Xcode if you want to use the "Xcode" generator.
  • On Windows, you have to decide whether you want to build for 32 for 64-bit. In CMake, 32-bit builds are the default, unless you specify the -A x64 argument. If you need to support both 32 and 64-bit, things become more complicated. You need to run the last two commands twice (one for each bitness), but take extra care that the second install command does not overwrite the binary files of your library. I have observed the weird behavior of CMake to always install the headers and binaries somewhere below “C:\Program Files (x86)”, even for 64-bit builds. Check the CMakeCache.txt file inside the generated build directory of your 64-bit build to verify that the CMAKE_INSTALL_PREFIX is correctly pointing to “C:/Program Files”. If it is not, delete the build directory and re-run the cmake -G command, appending something like
    -D CMAKE_INSTALL_PREFIX="C:/Program Files".
    • Note: on macOS your OS and hardware and underlying build system decides the default architecture (e.g. 64-bit) that is used to during compilation. See here how to build universal binaries, which becomes necessary once you want to natively support Apple silicon!
  • The -B argument specifies the path to the build directory that CMake will create for you. I simply specified a simple name, i.e., a relative path, which is not used on the file system yet. I recommend against using “build”, as some packages already have a non-empty “build” directory.
  • The last line builds and installs the ZeroMQ library. The path given by the --build argument must match the path provided to the -B argument from the cmake -G line.

Silent Git clones and pulls

If you use Git as your Version Control System, your build VM needs to be able to clone or update (pull) your code without user interaction. For this, you need to deposit the credentials in a file and put it in the VM, and configure Git to use that file. The following snippet illustrates the necessary additions to your Vagrantfile:

config.vm.provision "file", source: ".git-credentials", destination: "/Users/vagrant/.git-credentials"

config.vm.provision "shell", privileged: false, inline: <<-SHELL
  git config --global credential.interactive never
  git config --global credential.useHttpPath true
  git config --global credential.helper store
  git clone https://yourhost.com/your-project.git
SHELLCode language: Ruby (ruby)

So, what is going on here?:

  • We configure Git to use the store credentials-helper, which expects that there is a .git-credentials file in the user’s home directory. This is a simple text file which contains the usernames, passwords and corresponding hosts / URLs, in clear-text (unencrypted)! You must not store this file in your VCS. For extra security, I recommend that you generate extra Git accounts (with read-only permissions) specifically for your build VM, e.g. using deploy tokens which systems such as GitHub or GitLab offer. This way, even if these credentials get leaked, the damage is limited, as long as you are using a unique, generated password for that Git account.
    • This .git-credentials file is placed in the VM using the file provisioner in line 1. Make sure to adapt the destination path if you are creating a Windows build VM.
    • The required format of the content in that file depends on the credential.useHttpPath setting (which is false by default). If credential.useHttpPath is set to true, it must contain lines of the form https://user:pw@yourhost.com/your-project.git, otherwise the path postfix is omitted, so each entry should have the form https://user:pw@your-host.com
  • The credential.useHttpPath setting is only needed in case you are using several distinct credentials for several repositories which are located on the same host, e.g. when using Git submodules. Otherwise you can omit that line.
  • The credential.interactive setting ensures that Git really never ever asks for credentials interactively via a GUI or CLI prompt. Especially on Windows I found that, without this setting, Git would still attempt to ask you for the credentials, even though the store helper is configured and the credentials are found in the .git-credentials file.
  • The privileged: false part is important! It avoids that the git clone command runs as root user, for which no .git-credentials file was prepared!

Working with secrets

By “secrets” I mean things like credentials (typically username+password combinations), license keys, or any other data that requires protection. Sometimes, CLI provisioning commands require these kinds of secrets. However, you cannot put these secrets into your Vagrantfile, because the Vagrantfile is part of your VCS, and secrets don’t belong in VCS, ever!

To solve this problem, I use simple text files, which contain the secrets. Let’s take a look at an example for macOS, where my build script notarizes my macOS app automatically. This requires my Apple developer account email address and a token, which I don’t want to permanently store on disk, or hard-code into the build script. Instead, I use macOS’s keychain. My provisioning script needs to add an account to the login key chain. I store the secrets, my email address with the secret token, in a text file, e.g. named notarization-credentials. I add notarization-credentials to my .gitignore file, to ensure that it won’t be accidentally committed, and I commit a file such as notarization-credentials.example instead. It contains dummy data (to illustrate the format), together with instructions on how to get this data, or who to ask, e.g.:

user.name@yourcompany.com;yourPassword

Copy this file to "notarization-credentials".
Make sure the file only contains one line with your own credentials. Create the password as described on
https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow
using an "app-specific password". See also https://support.apple.com/en-us/HT204397Code language: JavaScript (javascript)

Here is an example for the Bash script (committed to Git) that Vagrant executes to create the account, which reads the credentials from notarization-credentials:

#!/bin/bash

security unlock-keychain -p vagrant /Users/vagrant/Library/Keychains/login.keychain
EMAIL=$(awk -F ';' '{print $1 }' notarization-credentials)
PASSWORD=$(awk -F ';' '{print $2 }' notarization-credentials)
security add-generic-password -a $EMAIL -s AC_PASSWORD -T /Applications/Xcode.app/Contents/Developer/usr/bin/altool -p "$PASSWORD"Code language: JavaScript (javascript)

This example uses the command awk -F ';' '{print $1 }' <file> which reads and parses <file>, splits the content by the separator specified by the -F argument, and then outputs the first one ($1). The EMAIL=$(...) construct ensures that the output is stored in the variable EMAIL, which is then accessed in the last line, using $EMAIL. If all of this is very confusing, you will need to become familiar with basic (Bash) shell programming first.

If your file only contains a single piece of information, you can simply use $(cat <file>) instead of awk.

After the secrets have been processed by your shell provisioning scripts, make sure to delete the corresponding files from inside the VM, using another shell provisioning block!

Conclusion

Whenever you need build systems or test runners, you should rely on automation to get reliable, reproducible build or test environments. If you need to work with platforms not supported by Docker, Vagrant offers a great, VM-based alternative. This article, together with part 1, presented a whole slew of pitfalls and tips. I hope that it saves you from having to research related issues.

I find Vagrant to be the most flexible tool for creating build and test environments. The provider (“hypervisor”) is configurable completely independently from your provisioning commands. Thus, the VM could be local, e.g. using VirtualBox, or in the cloud. If you don’t like package managers like Chocolatey or Homebrew, you can also use configuration management tools such as Puppet, Chef or Ansible, which are all integrated with Vagrant. The choice is yours!

Leave a Comment