Container-based development environments

In this article I discuss container-based development, where you not only run but also develop and debug software inside a locally-running Docker container. I explain the advantages and disadvantages of this approach, and look at VS Code’s development container feature, various features of IntelliJ-based IDEs, and how to implement an IDE-independent approach.

Introduction to container-based development

For many years, developers have been developing “natively” on their host machine. They install compilers, language runtimes and frameworks on their machine directly, and use the tight integration of their IDE for things like code-completion or debugging (with break points).

I still observe this today, and found that this kind of native development causes several problems in practice:

• Onboarding of new developers takes time and expertise: developers need to be told how to install the myriad of tools (beyond the IDE), e.g. the programming language runtimes. The problem becomes worse when developers leave their area of expertise, e.g. when a frontend developer (familiar only with Node.js) wants to develop against a local Java backend: the frontend developer will need help from someone from the backend team to get the backend up and running (or follow long tutorials).
• The differences between the local and production environment cause problems: Examples for such differences are: different OS (e.g. Linux in production vs. macOS/Windows on the developer machine), or different versions of the runtimes (e.g. Java SDK 19 in production, 17 on the developer’s machine). The problems that result from these differences are costly to find, diagnose and fix. They are typically only discovered at a later point (after deploying to production), and it involves communicating between the developer and the ops/SRE/infra-team. The developer may already have moved on to developing the next feature, and now needs to context-switch to understand the old feature again. These are problems of the form “it worked on my machine”.

Recently, the shift to (Docker) containers and images happened, not only in production, but also in the development phase of the SDLC. Container-based development environments aim to solve these problems, by developing inside a (Docker) container that is very similar to the production environment, which is (presumably) also running in a Docker/OCI container. There are many different ways of doing container-based development, and the different IDEs out there support it to different degrees. These are possible approaches I identified:

• The development container stores your code, runs your application and (typically) contains a server-part of your IDE. It runs in the cloud. An editor (your IDE) runs as thin client, locally as an application, or in your browser. This is the “cloud IDE” territory about I have written about in this article. Modern solutions include JetBrains Space, Gitpod or GitHub Codespaces.
• If you run Kubernetes in production, you can run an entire Kubernetes cluster locally (or host a dedicated development cluster in the cloud / data center), develop your code locally, and use tools like Tilt or DevSpace to quickly build and push code changes to the cluster. I have written about these tools in this article.
• If running development Kubernetes clusters is overkill and consumes too many resources (or if you do not use Kubernetes in production anyway), you can also mount your code into a single local (Docker) container, build your application inside the container, and also run it there. If you have multiple apps/services that interact with each other, you can use docker-compose to simplify the orchestration.

In this article I explain the last variant: running local development environments in a Docker container. The main points of this article are:

• How to use the VS Code IDE and its development containers feature
• A brief look at the IntelliJ platform, its shortcomings when it comes to Docker-integration, and its Gateway feature as an alternative to VS Code’s development container feature.
• Building a Docker-based development environment (including hot reloading) for any IDE, with examples for a Gradle-based Spring-Boot backend and an Angular frontend application.

If you tried container-based development containers a while ago and discarded this approach, e.g. because of bad runtime performance (compared to using a native runtime installed on your host), or because of a complicated debugger setup: do not worry, things have changed and are now much better than a few years ago.

Advantages & disadvantages

Let’s start with advantages of local development containers:

• Development containers are very similar to the production environment → this avoids surprises and bugs that happen later at deployment time, which are costly to fix.
• When used by every developer, the set up is not only the same as in production, but also every developer has the same setup.
• Development containers can be customized for each project to incorporate extra tools the project needs, e.g. linters that you run in a Git pre-commit hook, or (in case of VS Code) a set of IDE plug-ins everyone in your team should use.
• When using VS Code:
  • Faster developer onboarding: you only need to tell your developers to install and start Docker before opening the project with VS Code: the entire setup happens automagically: the developer is prompted by VS Code automatically to do the right thing.
  • Less communication overhead on setup changes: whenever your runtime or other aspects of your production environment changes, your developers get up-to-date completely automatically. You no longer need to write messages such as “hey Josh, you need to update from Java 17 to 19 ASAP”. VS Code automatically detects that the development container is no longer up-to-date (due to changes in the Dockerfile), and prompts the developer to rebuild it when opening the project.

There are also a few disadvantages:

• There is a performance penalty, due to virtualization, and because file system mounting from the host into the container is slower than the native file system itself. Consequently, the dockerized app runs a bit slower. On Windows and macOS, the RAM usage is higher because Docker runs a Linux VM.
• When working on a modern Mac with M1 (ARM64) chip, the environment is still not exactly the same as in production, where the servers typically have a Intel/AMD 64-bit processor. Although Docker Desktop for Mac can emulate the Intel/AMD64 processor, there is a severe performance impact.
• You need experts who know how to maintain the Dockerfile and other configuration files, such as the devcontainer.json file, to avoid as many performance pitfalls as possible. Fear not, this article explains all relevant tips you need to know.
• The setup varies for each IDE, and not all popular IDEs support development containers. There are a few emerging / competing standards such as Development Containers (Microsoft) and Devfile (or GitPod’s proprietary gitpod.yml file), but only time will tell which ones will dominate the market. There are also voices asking JetBrains to implement support for devcontainer-like files (see ticket), but JetBrains has not reacted at all, presumably because they want to sell their own solution, JetBrains Space.

Development containers: basic approaches

Before we go into the specifics of creating & using development containers for VS Code or IntelliJ-based IDEs, let’s take a look at the general two approaches for Docker-based development containers:

• Approach #1: the IDE (which is usually a fat client) supports being split into a server and a thin client. The fat client (running on the host) starts the development container (first building its image as specified in a Dockerfile) and installs the IDE’s server-component into the container. The fat client on the host becomes a thin client that only renders the IDE windows, and it is permanently connected to the server-component inside the container. The server-component does things like finding and configuring runtimes (e.g. Java), attaching a debugger to your application, or running a language server used for code navigation and code-completion. This deep integration between the IDE and the container let’s you do typical tasks inside the containers as if you were doing them on the host, like running a debugger.
• Approach #2: the IDE is not split into client & server. Instead, the IDE uses some proprietary integration mechanism to simplify some tasks, e.g. setting up a debugger for your programming language.

Approach #1 is done by the development container feature of VS Code. The next section takes a detailed look at this approach. VS Code also has a Remote development feature (docs), which is a generalization of the development container feature, the difference being that the VS Code server-component is not specifically installed into a local Docker container, but into some (remote) Linux box (VM, physical, cloud, container, …). JetBrains’ Gateway feature is also similar to VS Code’s remote development feature.

Approach #2 is done by the Docker integration feature of the IntelliJ platform. I take a look at this further below.

VS Code: development container

The development container feature of VS Code is a special variant of VS Code’s Remote Development feature. With VS Code’s development container, VS Code’s server-component is not installed into some (generic) Linux box, but into a Docker container. The documentation of the development container feature is quite good, see here and here, but also a bit overwhelming. The general idea is that you need a Dockerfile and a .devcontainer/devcontainer.json file. The Dockerfile tells VS Code how to build the development container, and the devcontainer.json file tells VS Code which Docker image to build (by referencing the Dockerfile), and how to start and configure the development container (after VS Code built its image). The Dockerfile could be one generated by VS Code specifically for the purpose of being a development container, or you could reuse a Dockerfile you already have. VS Code also supports referencing a docker-compose.yml file (and choosing a specific service from it), instead of referencing a Dockerfile.

You could simply follow the official VS Code tutorials (see this one) to add a development container to your project, but I have a few tricks that you should definitely keep in mind.

Tip 1: Use your own Dockerfile

Microsoft advertises that you use Microsoft’s devcontainer images (images starting with mcr.microsoft.com/devcontainers/...). These are the images referenced by the Dockerfile that VS Code creates if you use its “Dev Containers: Add Dev Container Configuration Files…” function, and these are also used in all of Microsoft’s sample projects for the development container feature.

I highly recommend that you do not use these images!

Why? Because Microsoft’s pre-built images have two disadvantages:

1. Chances are that the programming language runtime you are using in production (that you reference in your own Dockerfile) does not match the runtime that comes preinstalled with the dev-container image built by Microsoft. Exemplary, for a Java application, Microsoft’s mcr.microsoft.com/vscode/devcontainers/java:[11/17] image installs Microsoft‘s OpenJDK, which might (in detail) behave differently than your chosen Java distribution (e.g. IBM Semeru, or something else). You also have less control, because you don’t know which minor version of the language runtime is installed. Essentially, this means that the behavior of the Docker image is not 100% reproducible between machines, and you are likely not even running the same language runtime as you do in production.
2. It is difficult to figure out what exactly is installed into Microsoft’s images – they are beyond your control. They may contain stuff that you don’t need or want. For instance, Microsoft installs sdkman (see Dockerfile) into their Java images, because some users might want to install Maven or Gradle (and your project might not need it because you use the Gradle wrapper). The situation reminds me of the bloated webbrowser toolbars you got back in the day, after installing “free” stuff…

Instead, I recommend that you use your own Dockerfile, design it as a multi-stage image, and simply add a dedicated stage for VS Code. Your Dockerfile could look like this:

# Contents of Dockerfile
FROM some-language-runtime:major.minor.patch as base
 
FROM base as build
# here are statements that build your application
 
FROM base as vs-code-devcontainer
# Install Git to be able to use it right in the development container
RUN apt-get update && apt-get install git -y
# Optional: install further useful tools, e.g. pre-commit.org or others
 
FROM base as production-app
COPY --from=build /some/artifact.bin /somewhere/
ENTRYPOINT somethingCode language: PHP (php)

In your devcontainer.json file, you also have to reference the stage:

// Contents of devcontainer.json
{
    ....
    "build": {
        "dockerfile": "../Dockerfile",
        "target": "vs-code-devcontainer"
    },
    ....
}Code language: JavaScript (javascript)

Note: inside a development container, VS Code should automatically make your Git credentials (configured on the host) available inside the container. If this does not work out of the box, see here for help.

Tip 2: On Windows, clone the code to WSL

On Windows, instead of cloning your project to the Windows host file system (“C:\…” or similar), you should clone it to the Linux-native file system of a WSL Linux distro. There are two reasons:

• When you change files on “C:” the “file has changed” notification mechanism does not work, and tools inside the container that rely on it (e.g. a hot reloading feature of an Angular development server) won’t detect file changes.
• The file synchronization performance is much better, because you are not crossing the boundary between Windows and the Docker-Linux-VM filesystem.

You can install WSL2 and a Ubuntu WSL2 distro as officially documented here (wsl --install). Once installed:

• Open the WSL-distro’s terminal
• Install Git, e.g. via sudo apt-get update && sudo apt-get install git -y
• Switch to the WSL distro’s home directory, via cd ~
• git clone your repository

To open the code stored in your WSL-distro’s file system from Windows (and from VS Code), use the path \\wsl$\distroname, e.g. \\wsl$\Ubuntu-22.04 – which acts as network share.

Make sure to document these steps for your developers!

Tip 3: Override file system mount points for large folders

When VS Code starts the development container, it sets up a bind mount that mounts your entire project directory into the development container. If you work on macOS, or on Windows and ignore tip #2 for some reason, you should be aware that Docker’s bind mounting mechanism is slow on these platforms, because here bind-mounting actually copies the files between your Windows/macOS file system and the Linux-based file system internally used by the Docker engine (on Windows, Docker Desktop sets up its own WSL distro, on macOS it sets up a Linux VM with HyperKit). The impact on performance can be massive, if the folder is large (measured in bytes) or contains many files. A typical example is the node_modules folder for Node.js-based applications, which can easily contain tens of thousands of files and have a size of several hundred MBs.

To overcome this problem, you can set up a dedicated Docker volume for these large folders, and set up a bind mount for these folders. Docker volumes exist in Docker’s Linux environment and thus offer fast reading and writing speed. Let’s see how to do this for the node_modules folder:

// Contents of devcontainer.json
{
    ...
    "mounts": [
        "source=my-frontend-node_modules,target=${containerWorkspaceFolder}/node_modules,type=volume"
    ],
    ....
}Code language: JavaScript (javascript)

You can read more about this trick here. You can choose the volume name (here: my-frontend-node_modules) freely. If you have several such folders, simply add multiple entries to the mounts array.

Note that this volume-bind-mount trick has the effect that the respective folders inside the container and your host are completely independent of each other: they are no longer kept in sync!

Another tip: if you suspect that some application is running much slower in the development container (compared to the native performance on the host) due to file system activity, see here for how to start the application and trace what it is doing on the file system.

Example project

I have built an example application made up of an Angular frontend and a Spring Boot (Gradle with Kotlin DSL) backend application that demonstrates the concepts and includes the 3 tips above. Feel free to take a look at it. There I also address issues related to file system ownership: if you use Linux as host (or Windows with WSL2), you need to take care that the file ownership of the files on the host are matching with the ones inside the container. If you don’t address this, it could happen that files you create in VS Code inside the container are created as root user, and they are also owned by root on the host, which is not what you’d want. If the base image you use already has an unprivileged user (as is e.g. the case for the node image), then VS Code has some automated mechanism where it creates another image layer in which it bends the user and group ID of that user to match the user/group ID of your host user. You can search the internet for “updateRemoteUserUID” to learn more about this feature. However, if your image does not have such a user, you can create one, as explained here.

Reviewing features in a development container

Another interesting use case for VS Code’s development container feature is the ability to clone any Git repository directly into a new development container, without cloning it to the host first. See here for details. This is useful to quickly review and test a feature branch of your project in isolation, so that it does not affect your already-in-use development container. It basically avoids that you need to interrupt your work in your repository’s main working tree (e.g. via “git stash“).

However, if you use tip #3 mentioned above, the isolation is somewhat limited. For instance, if you have a Node.js application (for which node_modules is mounted into a separate Docker volume) and the feature branch changes the package.json file, your node_modules Docker volume will be modified by the feature branch’s devcontainer, which affects your main development container, too! In this case, you are better off not using this feature.

IntelliJ’s Docker integration

IntelliJ-based IDEs have two ways of supporting local, Docker-based development containers. I present each one in the next two subsections.

Remote Docker runtime support

The traditional IntelliJ-based IDEs, such as PyCharm, WebStorm or IDEA, all have the same basic level of Docker support: once you configured the Docker daemon (or Docker Desktop) in the IntelliJ settings dialog, you can create Run configurations that run or build Docker containers/images (or docker-compose stacks). You can use the Services tab at the bottom of the window to see a GUI that shows running Docker (-compose) containers, and lets you control them.

In addition, some of the IDEs have a deeper level of integration, that is specific to that IDE and its primarily-supported programming language runtime. The integration is of varying quality. Let’s look at a few examples:

• PyCharm: you can create a Docker-based remote Python interpreter (docs). PyCharm then inspects this interpreter, and the packages that were installed with pip inside the container. In my experience, running and even debugging applications this way works very well.
• WebStorm: as explained in the docs, WebStorm also supports setting up a remote Node.js interpreter, just like PyCharm does for Python. While this basically works, JetBrains decided to not care about macOS and Windows users, who suffer from poor performance with the node_modules folder, which I explained above. Whenever you create a Run configuration in WebStorm for a remote Node.js interpreter (or run a npm/yarn command with the remote interpreter), the following happens: WebStorm mounts the project’s root folder from the host to some random directory in the container (at /tmp/<uuid>). The node_modules folder is mounted and synchronized to your host, which is horribly slow on macOS and Windows. There is already a ticket asking why the run configuration does not use the default bind-mount already configured in the run configuration’s dialog (/opt/project in the container). If WebStorm would use that default bind mount, we could apply the volume-mount-trick for the node_modules folder that I described above, for VS Code. But even then, another nuisance is the fact that WebStorm’s code navigation & parsing only considers the node_modules folder on the host (unlike PyCharm, which can scans the dependencies that pip installed into the container). Consequently, you still need to have another Node interpreter installed on the host, use it to run “npm install” on the host, just to be able to get code navigation in WebStorm.
• IDEA: IDEA does not support remote Java interpreters at all. It only offers a “Remote JVM Debug” run configuration, but it is up to you to figure out how to get the remote interpreter up and running.
• Rider: like IDEA, there is no support for remote run-times. However, given a Run configuration you created for a Dockerfile, you can click the Debug button, and Rider will magically set up remote debugging for you, but only if the contained C# application is actually a .NET application.

Remote development with JetBrains Gateway

Similar to VS Code’s Remote development feature, the traditional IntelliJ-based IDEs also support being run in a headless mode, as server component. The thin client you run locally is called JetBrains Gateway, which connects to a (remote) Linux machine via SSH, installs the server component (if not already present), and then connects to the server component. Contrary to VS Code, Gateway expects your code to be already located on that remote Linux machine (not on your local machine).

Conceptually, you can simply run a local Docker container (instead of a full-blown Linux machine) with a SSH server and use it as “remote” Linux machine in the Gateway client. There are scripts and projects such as this one or this one to get you started. Of course you have to modify these examples so that they use the base image of your application. And you should keep your code on the host, mounting it into the container.

The main problem I had when trying it out: it did not work, not at all. After configuring the remote SSH host, port and username/password, and selecting the remote IDE to install, the Gateway window would simply close, not showing any error message whatsoever. I sifted through the logs, but did not find anything that indicates the root cause. And even if it had worked, from what I read the stability of Gateway is not really great. Also, your Docker image would have to include many dependencies you would normally not want in your (production) image, such as the SSH daemon. Gateway is not a solution for M1-based macs either, because Docker then runs ARM64 Docker images, but the IntelliJ remote server is only compiled for Intel-based CPUs.

IDE-independent development containers

Motivated by the fact that I love container-based development and JetBrains IDEs, but their IDEs lack proper Docker-integration for many programming languages, I figured out a generic approach that works for any language and IDE, assuming that the language runtime (and IDE) supports remote debugging (typically via a TCP port).

The workflow is that the developer starts a Docker container, and can then develop “inside” the container by simply changing files on their host (in the host IDE). Your application’s source code is on the host, bind-mounted into the container. Inside the container, there is some mechanism that detects these code changes, hot-reloads the code, and lets the developer connect to a debug port.

The main caveats of this approach are:

• Here I only explain how to do this for a Spring Boot (Gradle) application and an Angular Frontend application. For any other language & framework combination, you need to figure out yourself how to open a debug port, or how to enable hot reloading.
• To get proper code-completion and -navigation, developers need to additionally install the language runtime and project dependencies (via the runtime’s package manager) on their host. However, this host-runtime is not used for compilation, only for code-completion and -navigation. Consequently, if the developer accidentally gets this wrong (and e.g. installs the wrong version of the runtime, e.g. Node 14 even though the project uses Node 18), the only side effect is that the code-completion might be wrong.

To see how to achieve this, take a look at the example application repository that contains an Angular frontend and a Spring Boot (Gradle with Kotlin DSL) backend application. The following files are important:

• Dockerfile: contains a dedicated “development” stage that starts overrides the ENTRYPOINT to start a start-dev-server-with-hotreloading.sh script.
• docker-compose.yml: builds and starts the Docker image, configuring the necessary port forwardings (for the application and its debug port) and volumes (the project directory, and others, such as the node_modules folder bind-mount-trick)
• start-dev-server-with-hotreloading.sh: contains runtime-specific tricks to speed up the start of the development server, then finally starts the development server

In your project’s README, you should document how developers can get started. For instance:

• Explain how to set up the local runtime:
  • Tell developers who use Microsoft Windows to install WSL2 with some WSL2 Linux distribution (e.g. Ubuntu), then clone the project to somewhere below /home/<username> of that WSL distribution. Otherwise (if they clone it to the native Windows file system, e.g. “C:”), changes that they make on the host are most likely not detected by the hot-reloading mechanism inside the container (due to limitations of Docker’s bind-mounting implementation).
  • Tell developers that they need to set up the runtime and configure it in their IDE. They should look at the first line of the Dockerfile to see which version of the runtime to choose. If your team uses Windows as well as IDEs that integrate well with WSL2 (e.g. IntelliJ), suggest that they install the programming language runtime (used only in the IDE for code-completion) right into WSL (vs. installing the runtime on the Windows host). There are very nice install-helper-scripts that are usually more polished for Linux, e.g. sdkman for Java, or nvm for Node.js. They install runtimes with specific versions, even several ones in parallel.
• Provide instructions for which command to use to start the Docker-based project (typically something like “docker-compose up <servicename> -d --build“).
• Provide instructions for configuring the remote-debugger, which depends on the IDE.

Conclusion

To get a local, container-based development environment up and running, VS Code offers the most streamlined experience. However, if you dislike VS Code, for whatever reason (features, look & feel, etc.), you can either use my approach presented above, or (if you have enough RAM) use VS Code and your preferred IDE in parallel: edit the code in your preferred IDE, but run and debug it in VS Code’s development container mode.

If you like the idea of container-based development environments, but dislike the aspect of running containers locally, check out solutions such as GitPod, GitHub CodeSpaces or JetBrains Space. If you are a fan of JetBrains products, check out Fleet, which is their new IDE, currently in public preview. It is architected in a distributed way, and supports local as well as remote container-based development environments natively. However, regarding its feature set, it is not yet on par with the more mature IntelliJ-based IDEs yet.

If you liked this post and want to read another perspective about the topic, check out this excellent CodeCentric blog post.

Have you been using container-based development environments in your projects? What has your experience been? Let’s discuss it in the comments!