Create a build runner VM with vagrant for Windows and macOS – Part 1

Build runners, which compile and package your software, are often implemented using Docker containers. However, you cannot use Docker if you need a Windows or macOS build runner. This article presents how to create Windows and macOS build runners as virtual machines using Vagrant. While the instantiation of a build VM is fully automated, it is challenging to create the list of provisioning commands that install and configure your build tools within the VM. This article is part 1 of 2, presenting several tips and recipes for common provisioning steps.

Introduction

Build or test runners are tools which build your application and run automated tests as part of a CI pipeline. They are typically implemented using Docker, where your code is built and tested inside a Linux container. As I discussed in my previous article, Docker does not support Windows containers (on a Linux host), and generally does not support macOS containers. However, you sometimes need a macOS or Windows environment, because the software you want to build runs on it. Examples include desktop applications, or iOS apps, which can only be built on macOS.

This article is part 1 of a 2-part series that discusses how to use Vagrant to create such virtualized environments in a fully automated fashion. It presents recipe-like solutions to problems I encountered while adopting this approach.

Manual steps to create a build runner

What does it mean to create a build runner? Starting from a naked operating system, you need to perform many steps to get a ready-to-use build runner. Here is an example of steps you might execute:

• Install compiler tool chains (e.g. Xcode or Visual Studio)
• Install other build-related tools, e.g. CMake for building C++ projects
• Install tools that build a redistributable installation medium (e.g. DropDMG for macOS, or Inno Setup for Windows)
• Install a version control system (VCS) such as Git
• Configure the VCS so that checking out your code works without any user interaction
• Clone your code from VCS
• Download third party dependencies (e.g. using pip for Python, Gradle for Java, etc.)
• Install signing certificates so that your build tools can sign your binaries
• If any of the tools you installed require a license key, enter that license key
• Install a CI integration tool, e.g. a GitLab runner
• Perform a test build (smoke test), e.g. by checking out an older revision of your code that should build without errors, and run your build scripts, verifying that they complete without errors.

Downsides of manually-built environments

A common approach I often see (and have used myself in the past) is to build these environments by hand, documenting the steps somewhere, e.g. in a wiki. Unfortunately, this has several disadvantages:

Solution: to create a build runner VM with Vagrant

Instead of manually creating your build runner I suggest a fully automated approach, using Vagrant. All required steps are implemented as shell commands that you put in a Vagrantfile, or helper bash/shell scripts called by Vagrant. Once the Vagrantfile is prepared, getting a ready-to-use build VM is as easy as running vagrant up, and waiting a few hours. No interaction is necessary. This approach solves all downsides mentioned above! It saves a ton of time to get a build system up for anyone who needs it, you can quickly adapt it to newer versions of the base OS, and the recipe is much more precise than hand-made documentation. You cannot possibly forget a step without noticing it quickly.

Since there are many smaller caveats, I’ll provide recipe-like solutions and hints that address these caveats.

Tips and recipes for creating a build runner VM

Use a trustworthy base box

With Vagrant, your VM is always based on some base box – the immutable image your VM will be instantiated from. In my previous article, How to create a Vagrant box with Packer, I explained why you may not want to use boxes hosted on Vagrant Cloud for building. While using ready-made boxes is convenient, the problem is that anyone can upload boxes to this service – including bad actors who can embed malware into the box, which tracks your activities and sends information to a remote server. You certainly don’t want to use such a box only to find out that your application’s source code was leaked to an unknown third party. Fortunately, building your own base box is not very difficult when using Packer, so head over to my article to find out how to build your own box.

Provide sufficient memory to the VM

With too little memory (RAM) assigned to the VM, the VM will constantly be using the disk swap partition, which slows down the provision process (and subsequent build script executions) drastically. You need to experiment with different values. For me, Windows 10 works fine even with just 2 GB, while macOS is usually atrociously slow with less than 4 GB. YMMV.

Install software using a package manager

There is a lot of third party off-the-shelf software you may need in your build VM. There are package managers which take away the need for manually-written scripts that download the software’s installation medium and perform a “silent install”. Let’s take a look at the most popular package manager for macOS and Windows.

Windows

On Windows, I recommend chocolatey, which you can install by adding the following snippet to your Vagrantfile:

config.vm.provision "shell", inline: <<-SHELL
    powershell -NoProfile -ExecutionPolicy unrestricted -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))"
  SHELLCode language: Ruby (ruby)

The chocolatey catalog contains thousands of popular applications, compilers, programming languages and libraries, including Git, Visual Studio, Inno Setup, Python, etc. For instance, the following line will install Inno Setup for us:

choco install innosetup -y --no-progress --version=6.0.5

Use the search functionality on chocolatey.org to determine the concrete alphanumeric name (“innosetup” in the above example) for your desired package. The package page also discusses additional, package-specific arguments, and lists a full version history. In many cases, fixing the package to a specific version is highly recommended, in order to get a reproducible build VM. Here, we fixed the version using the --version argument. The -y argument makes sure that the process requires no interaction on your part, forcing Chocolatey to automatically accept any license agreements (which you should read, of course, hehehe). The --no-progress argument makes the output of choco much more silent. Without this flag, I found that installations are slowed down, as choco would print hundreds of lines of the form “Downloading: 1.5%”, “Downloading: 2%”, etc. It seems that this extremely verbose output floods the remote shell of Vagrant, which slows down the installation process.

In case you installed CLI tools using choco which need to be added to PATH, use the setx command to adapt it. The following example installs Inno Setup and fixes PATH:

choco install innosetup -y --no-progress --version=6.0.5
setx PATH "%PATH%;C:\\Program Files (x86)\\Inno Setup 6"Code language: JavaScript (javascript)

Note that you will sometimes need a refresh of the environment variables during the execution of your shell scripts. For instance, the choco command is not available right away after installing Chocolatey using the powershell command presented above. Fortunately, all you need to do to refresh your environment is to start a new config.vm.provision "shell" block. Vagrant executes these blocks from top to bottom, logging out and into the VM each time, which coincidentally refreshes the environment.

macOS

The equivalent to chocolatey on Windows is Homebrew for macOS. To install it, add the following snippet to your Vagrantfile:

config.vm.provision "shell" do |s|
    s.privileged = false
    s.inline = '/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"'
endCode language: JavaScript (javascript)

The s.privileged = false part is important, because Homebrew refuses to install as root user. Brew requires Git to function, which in turn requires Xcode or the macOS Command Line Tools to be present. If neither of them are, Homebrew attempts to silently install the Command Line Tools for you. This usually works without problems, but if not, consider the Xcode section below for more information.

When looking for packages, it is important to understand that Homebrew differentiates between formula and cask packages. Casks are binary software applications, e.g. Firefox, CMake, DropDMG, etc., which are installed via brew cask install <name>. Formulae are Ruby scripts which typically download source code and automatically compile it on your machine (essentially executing something like ./configure && make && make install). They are installed via brew install <name>. On a package’s site (e.g. DropDMG), the install command will show you which command to use, so make sure not to confuse them.

Unlike Chocolatey, Homebrew does not support installing specific (older) versions of a package, but attempts to always install the most recent version (known to Homebrew). There are a few package maintainers who offer multiple package branches, split by the major version, for instance python@3.7 and python@3.8. However, most packages don’t offer this. There are intricate ways to still get older versions installed, though.

You can also try mas-cli which provides a CLI interface to the Mac App Store.

Install Xcode or MS Visual Studio

Xcode / Command Line Tools

This section describes the manual installation of Xcode, for which you need an Apple developer account. In the browser, head over to the Downloads page and log in using your credentials. Download the Xcode xip archive file with a suitable Xcode version. Usually you will want the newest Xcode version supported by your macOS base box version. You can find the minimum required macOS version for a specific Xcode version on Wikipedia. For instance, on macOS 10.13, this would be Xcode 10.1.

To silently install Xcode given the xip file, add the following snippet to your Vagrantfile, adapting the paths as needed:

# Note: replace the source path
config.vm.provision "file", source: "Xcode_10.1.xip", destination: "/Users/vagrant/Xcode.xip"

config.vm.provision "shell", privileged: true, inline: <<-SHELL
  xip -x Xcode.xip
  rm Xcode.xip
  mv Xcode.app /Applications/
  xcode-select --switch /Applications/Xcode.app
  xcodebuild -license accept
  xcodebuild -runFirstLaunch
SHELLCode language: Ruby (ruby)

If you only need the compiler tools to build libraries using make, installing Xcode is overkill. The Command Line Tools, which contain C/C++ compilers, make, git, macOS SDKs, etc., are sufficient for this job. The easiest way to get them installed is to install Homebrew (as explained above), even if you don’t intend to use Homebrew at all. Homebrew’s installation procedure contains some magic which installs the Command Line Tools without any user interaction. You should not use the often advertised terminal command xcode-select --install, because it does require user interaction. That command triggers a popup in which you have to confirm the installation via mouse click. Thus, that command is not suitable.

In some rare cases, Homebrew may fail installing the Command Line Tools. In that case, head over to Apple’s Downloads page again, and download the corresponding Command Line Tools dmg file. An example on how to get it installed is shown below:

# Note: you may have to adapt some paths
config.vm.provision "file", source: "Command_Line_Tools_macOS_10.13_for_Xcode_10.1.dmg",
  destination: "/Users/vagrant/Command_Line_Tools_macOS_10.13_for_Xcode_10.1.dmg"
config.vm.provision "shell", inline: <<-SHELL
  hdiutil attach Command_Line_Tools_macOS_10.13_for_Xcode_10.1.dmg
  installer -pkg "/Volumes/Command Line Developer Tools/Command Line Tools (macOS High Sierra version 10.13).pkg" -target /
  hdiutil detach "/Volumes/Command Line Developer Tools"
SHELLCode language: Ruby (ruby)

The above example also illustrates how to silently install software from a .dmg or .pkg file. Disk image (dmg) files will either contain .pkg files, or contain the app bundle (.app folder). In the latter case, a simple copy command, such as
cp -R "/Volumes/Some App/Some Software.app" /Applications/
will suffice.

Microsoft Visual Studio

To compile C++, C# or other kinds of projects supported by Visual Studio, all you need are the free Visual Studio Build Tools. These are basically the compilers, without the IDE. Using Chocolatey (which I introduced above), there are corresponding packages (e.g. visualstudio2019buildtools). Using the linked documentation, you will have to figure out the arguments to pass into the installer so that it actually installs your required workloads or components. The following example installs the components necessary to build C/C++ projects using the Visual C++ 2019 (14.2) run-times :

config.vm.provision "shell", reboot: true, inline: <<-SHELL
  choco install visualstudio2019buildtools -y --no-progress --package-parameters "--add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.18362"
SHELLCode language: Ruby (ruby)

To install multiple workloads, you have to repeat the --add argument (with values taken from here) multiple times, separated by space, inside the --package-parameters argument string.

Use snapshots to speed up iterations

The hardest part of converting manual provisioning steps (which often involve clicking things in a GUI) to a list of non-interactive terminal commands is to figure out the exact commands. Since the effect of any command depends on the current state of the VM, testing different commands and different orders on different VM states yield completely different, non-reproducible results. Also, I can promise you that an approach of the sort “I just research all the commands on Google and put them all in the Vagrantfile” is destined to fail, because knowledge becomes out of date quickly. A command that worked for someone else some time in 2018 may no longer work today.

To overcome this problem, you have two options:

1. Destroy the VM via vagrant destroy, then vagrant up to get a clean slate, including all the Vagrantfile commands that are already known to work. Once the VM is up, experiment with new commands. If they work, put them in the Vagrantfile, destroy the VM again and vagrant up again to verify they actually work.
2. Run vagrant snapshot save <name> to create a snapshot of the VM, then experiment with the new commands. If they worked, put them in the Vagrantfile (providing a name for the provision section), then run vagrant snapshot restore <name>, and vagrant provision --provision-with <name> to execute only that specific provisioner section.

The second approach is much faster, saving you time while iteratively refining your Vagrantfile.

Conclusion

Whenever you need build systems or test runners, you should rely on automation to get reliable, reproducible build or test environments. If you need to work with platforms not supported by Docker, Vagrant offers a great, VM-based alternative. This article presented a whole slew of pitfalls and tips, which are complemented in part 2 of this series.