Docker optimization guide: the 5 best tips to optimize Docker development speed

This article presents 5 tips that improve the speed of building Docker images locally. It focuses on two problems: iterating on your unfinished Dockerfile (where you still figure out configuration options and which packages to install), and iterating on your code (assuming a finished Dockerfile). I discuss tricks such as using BuildKit’s caching features, the .dockerignore file, or tweaking the RUN statement order in your Dockerfile.

Originally posted on 2022-01-09, updated on 2024-06-09.

Introduction

Docker has become an established tool for many software developers, not only to share and deploy software, but also to develop it. Under the hood, building images is done by BuildKit.

During development (on your development workstation), you will typically face two kinds of challenges that slow you down, when it comes to building images:

1. Iterating on your Dockerfile: you are working on a still unfinished Dockerfile. You are frequently changing statements in the Dockerfile, but re-building the image takes very long -> you want to reduce the build time, to iterate faster.
2. Iterating on your code: you have a finished Dockerfile, but still working on the code. Re-building the image on each code change takes very long -> here you also want to reduce the build time.

Let’s take a look at solution approaches for each problem separately. Note that you can (and should) combine them!

Dockerfile iteration

Approach 1: Splitting RUN statements

Due to Docker’s image layer cache feature (explained below in section Use a smart RUN statement order), you can improve your Dockerfile iteration speed by deliberately creating “too many” layers during the iteration phase (and aggregating them once you finished working on your Dockerfile):

1. Build a Docker image that contains those statements that are already known to work. In the most extreme case, this image is just an alias for your chosen base image, e.g. your Dockerfile might only have a statement such as
   FROM python3.11-slim.
2. Start a temporary container of that image (e.g. via “docker run --rm -it your-image bash“). Inside this container, iteratively / experimentally run the different commands you believe are necessary for the final image. For each command that worked (returning exit code 0), create a corresponding RUN command line in your Dockerfile.
3. From time to time (after adding a few commands), stop the temporary container, rebuild the image, and restart the temporary container. Thanks to Docker’s layer caching, those RUN statements whose layers were already cached won’t be re-built.

Once your Docker image works as expected, you finalize your Dockerfile: you reduce the number of layers / RUN commands, by aggregating RUN statements into fewer RUN statements that each span multiple lines. See the example below:

bla

FROM ruby:3.3.2-alpine
WORKDIR /app
RUN git clone https://some.project.git
RUN cd project
RUN bundle install
RUN rake db:migrate
Code language: Dockerfile (dockerfile)

FROM ruby:3.3.2-alpine
WORKDIR /app
SHELL ["/bin/ash", "-euo", "pipefail", "-c"]
RUN git clone https://some.project.git && \
    cd project && \
    bundle install && \
    rake db:migrate
Code language: Dockerfile (dockerfile)

To avoid that image building appears to complete successfully, even though some errors happened in a large RUN statement, you should use the Bash strict mode. There are several options:

• Add the SHELL statement, as shown above. This line basically “prefixes” every RUN command that follows in later lines with whatever you specified in the SHELL line. Unfortunately, there are several gotchas:
  • Not all image builders support this. E.g. Podman (more specifically: the underlying Buildah library) cannot build OCI-compliant images. You will see this error: WARN[0007] SHELL is not supported for OCI image format, [/bin/ash -euo pipefail -c] will be ignored. Must use `docker` format
  • You need to adapt the shell to one that actually exists in the image. In the above example, the image is Alpine-based, where /bin/bash does not exist and you need to use /bin/ash instead (ash=Almquist Shell).
• Start your aggregated line as follows: RUN set -euo pipefail && <other commands>

See here for background information about Bash strict mode.

FROM ruby:3.3.2-alpine
WORKDIR /app
RUN <<MULTI_LINE_END
    set -euo pipefail
    git clone https://some.project.git
    cd project
    bundle install
    rake db:migrate
MULTI_LINE_ENDCode language: Dockerfile (dockerfile)

Approach 2: Use BuildKit’s cache mount

You can use BuildKit’s RUN --mount=type=cache feature to speed up lines such as
“RUN apt-get install ...” . This speeds up re-building layers that involve package managers, such as apt, pip or npm.

The BuildKit documentation explains how the feature works, and it provides concrete examples for building Go packages, and using the apt package manager. You can adapt this approach to various other package managers or build tools, e.g.

• For NPM, use something like this: RUN --mount=type=cache,target=/root/.npm,id=npm npm ci
• For PNPM, use something like this (source): RUN --mount=type=cache,id=pnmcache,target=/pnpm_store && pnpm config set store-dir /pnpm_store && pnpm config set package-import-method copy && pnpm install --prefer-offline --ignore-scripts --frozen-lockfile
• For Yarn 1.x (classic): RUN --mount=type=cache,target=/root/.yarn YARN_CACHE_FOLDER=/root/.yarn yarn install [--frozen-lockfile]
• For .NET (Nuget): RUN --mount=type=cache,id=nuget,target=/root/.nuget/packages dotnet restore
  • Note: you need to provide the same --mount options for follow-up commands that also access the cached packages, e.g. “nuget build”
• For pip, use something like this: RUN --mount=type=cache,target=/root/.cache,id=pip pip install -r requirements.txt

Note: the above examples assume that you are installing packages as root user. if this is not the case, you need to change the /root/ path prefix to the home directory of the user you are running the commands with.

Code iteration

Approach 1: Use container-based development mode of your IDE

IDEs such as Visual Studio Code let you develop within running Docker containers directly. For VS Code, this feature is documented here and is called Dev containers. The basic idea is that code files exist on your host, and are mounted into a Docker container. Inside the container, not only are all the dependencies installed (as instructed in your Dockerfile), but VS Code additionally installs parts of itself into that container, e.g. the language parsing features, a debugger, or various other VS Code extensions. VS Code thus runs on the host (just the UI, basically as thin client) and in the container. Any code you change on the host will be instantly available inside the container, due to the bind mount VS Code has set up.

Other IDEs, such as JetBrains IDEAJ-based IDEs (e.g. PyCharm) also offer Dev container support since 2023.

You can find more details about container-based development environments in my blog post here.

Approach 2: Use a smart RUN statement order

One of the advantages of Docker is its layer-caching feature. In a nutshell, each statement in the Dockerfile creates a new image layer. When you repeatedly run docker build ..., the Docker daemon reuses as many cached layers as possible. Docker automatically determines whether it can reuse a layer, as explained in these official docs. Once a layer (or its respective Dockerfile statement) is considered to be modified (and Docker can no longer use its cached version), Docker must rebuild all image layers that follow after the first modified one.

The general idea is to avoid frequent re-builds of the entire layer stack, by designing the Dockerfile in such a way that statements/layers that change rarely come first, and layers that change frequently come last.

Consider the following example:

You are developing a containerized Docker application. In your Dockerfile you have the following statements:

FROM python:3.11
WORKDIR /app
COPY src src
RUN pip install -r src/requirements.txt
CMD ["python", "my_app.py"]
Code language: Dockerfile (dockerfile)

The problem here is that the frequent code changes (happening in the src folder) invalidate the cached layer for line #3. Thus, the layers built for lines 3-5 must be rebuilt on each code change, and the pip install command is rather slow. You can improve this example as follows:

FROM python:3.11
WORKDIR /app
COPY ./src/requirements.txt .
RUN pip install -r requirements.txt
COPY src src
CMD ["python", "my_app.py"]
Code language: Dockerfile (dockerfile)

Now, pip install only needs to run when the requirements.txt file changes, which should happen less often.

Approach 3: Use a .dockerignore file

Many developers incorrectly think that COPY or ADD statements always use your local directory as data source. However, this is incorrect! The source from which any ADD/COPY statement copies files/folders from is referred to as build context. The concept is explained here. While advanced users might know that you can even define multiple build contexts (as explained in this blog post), the typical scenario is that you use one build context.

To understand build contexts better, we need some background: Docker/BuildKit uses a distributed architecture to make building images flexible: on one machine you have the docker CLI installed, and your code (stored in a folder on your disk). Then, on a (potentially different) host, there is the builder component, e.g. BuildKit running as part of the Docker daemon. The builder and the docker CLI communicate over the “network” using gRPC, even if they both run on the same machine (e.g. when you installed Docker Desktop).

With this background knowledge in mind, the build context is simply the set of files and folders that the builder (BuildKit) can access, when it runs ADD/COPY statements. The build context could be a path to a Git repository (BuildKit knows how to clone Git repositories). But more commonly, you point your docker CLI to a local folder that sits on the same machine as the docker CLI, where you are running something like “docker build -t <image>:<tag> <local-folder>“.

In this common case, whenever BuildKit encounters a “ADD/COPY <src> <dest>” statement, it communicates with the docker CLI, asking it to dynamically stream the contents from its local disk over to BuildKit via gRPC (for the requested <src> path). While doing so, you can configure the docker CLI to filter out specific files and folders, by creating a .dockerignore file that needs to be located in the root of the build context folder. From BuildKit’s point of view, the build context is the set of files and folders the docker CLI will stream to it, having already applied the filtering using the .dockerignore file.

Tuning your .dockerignore file speeds up building, because:

1. You can exclude large files / folders from being copied into the image, which reduces the time of the copy process itself
2. BuildKit’s layer caching feature will work better, assuming that you use the .dockerignore file to exclude frequently-changing files that you don’t want in the image anyway

I highly recommend that you read the .dockerignore file’s official docs here.

In spirit, .dockerignore is very similar to the .gitignore file. What you put in it depends on your project (and the used programming languages), but here is a general recommendation:

• Many entries in your .gitignore file might be suitable for your .dockerignore file
• Dockerfile or docker-compose.y[a]ml files
• Meta-data from your VCS, e.g. the “.git” or “.svn” folder (or many other files/folders starting with “.”)
  • You can also use a “allow list ” approach, where you add “.*” (without the quotes) to your .dockerignore file, to exclude any file/folder starting with “.”, and then define exceptions for those specific files that you do need.
• Meta-data from your operating system, e.g. “**/.DS_Store”
• Any other large files (e.g. database backup dumps, raw data used for automated tests, etc.) that exist on the host or the repo. If you store large files via Git LFS, it makes sense to consult your .gitattributes file for lines that contain “filter=lfs” – maybe some of these entries should also belong into the .dockerignore file.
• Documentation, e.g. “docs” folders, README.md, etc.
• Tests (unless you run them as part of building the image)
• Any caches or installed packages that will be re-built or re-installed within the container anyway, e.g. the node_modules folder for JavaScript application using npm/etc. – this is highly specific to the used programming language or package-manager, so you should Google for “dockerignore <language>” etc. to find the best entries. Some examples are available here.

Conclusion

Optimizing your development workflow is very important. While memes such as this one are funny and often true, you don’t really want the “compiler” (BuildKit, in this case) forcing many micro-breaks upon you. You can’t really do anything meaningful during these breaks. Attempting to do several tasks in parallel is mentally exhausting and error-prone, at least for most of us. Learning and applying the above tips will make you more efficient, avoiding idle time. The other articles of this series discuss more optimization tips, e.g. tailored for your CI pipeline, some of which you can also apply to your local development workflow.