Detecting Docker image version tag updates easily

This article explains how to use my open-source Docker Tag Monitor website to determine how frequently the maintainers of a Docker image rebuild specific image version tags. It also explains why the image rebuild frequency matters when choosing the best (base) image.

Introduction

Choosing a secure Docker image to run (or base your software on) is a difficult problem for application development teams, and a wrong choice leads to security problems and lots of “busy work”.

When building software, many teams are already aware that they need to carefully choose and vet the libraries and frameworks they build their application on. For instance, a team working on a Java-based backend service will (hopefully) choose actively maintained libraries, such as Spring Boot or Quarkus, which receive security patches regularly.

However, once teams package their code as a (Docker) container image, they often just use the next best Docker image they can find during a quick internet search, recommended in a random person’s blog post. They use such images without any selection criteria or considering different alternatives. In the long run, such poorly chosen container images cause headaches, because they likely contain exploitable components, and because image security scanners will find many potential vulnerabilities that teams might be obliged to fix (due to company policy).

In this article, I address this problem:

• I provide background on the components contained in an image and explain the concept of image vulnerability
• I address an important shortcoming of vulnerability scanners
• I offer selection criteria for images
• One of the selection criteria is the “image rebuild frequency” for which I built a website you can use to determine it, the Docker Tag Monitor

Background: Docker image components and vulnerabilities

When building Docker/OCI images, you typically use “off the shelf” base images, e.g. from Docker Hub. This could be a bare OS image, such as Debian, or images that contain programming language runtimes (e.g. Node or Python). Or you might simply run off-the-shelf images without modifying them, e.g. Grafana or PostgreSQL, e.g., in Kubernetes.

Regardless of the scenario (build or run), one important aspect is the security of the image. Images typically contain a large number of software components (e.g. binaries and libraries), some of which are vulnerable to exploits. A vulnerability analysis will often look frightening. Here is an example of an analysis of postgres:17.1, which (at the time of writing) was just pushed a few days ago. I performed the analysis with Grype, using the command “grype postgres:17.1 --only-fixed“:

 ✔ 8 critical, 40 high, 37 medium, 5 low, 97 negligible (5 unknown)

At the same time, Grype finds 0 vulnerabilities in the Bitnami version (determined via “grype bitnami/postgresql:17.1.0 --only-fixed“). In this example, the official postgres:17.1 image is not necessarily less secure than the Bitnami one, as this article explains. The fact that an image contains components with CVEs is not problematic per se, because the found CVEs could be false positives, or could be inexploitable. An image is only actually insecure if it contains exploitable, true positive CVEs.

Having 0 CVEs means that you can relax, but if you have more than 0 CVEs, you need to double-check each CVE for false positives and exploitability and maintain ignore lists, which can become burdensome. 

In any case, once an image is built, because its content is static/immutable, its security will deteriorate over time. Typical counter-measures include:

• Image maintainers can regularly rebuild and push the images (ideally even for older releases), to ensure that the base components (e.g. Linux distro libraries) have as few vulnerabilities as possible
• You can run (or base your builds on) minimal or distroless images which have much fewer components, and thus typically much fewer CVEs. See this section of my Docker optimization guide series for further details
• If the vulnerability affects the software of the image itself (e.g. if in a Python 3.12.1 image, the vulnerability is specific to Python 3.12.1), then you, the operator who runs the image, need to update the image version tag to a newer software version where the vulnerability is fixed. Keep in mind that image vulnerability scanners, such as Trivy or Grype, will likely not find vulnerabilities in the contained software itself. For instance, “grype postgres:16.4 --only-fixed” will find many CVEs, but none of them target PostgreSQL itself, even though 16.4 is affected by plenty of vulnerabilities (see here)

Choosing the best Docker (base) image – the best criteria

Before building or running an image, it makes sense to gather a list of alternative image distributions and compare them. For instance, should you use the official build or the Bitnami build of Postgres? What about python:3.12 vs. the minimal ubuntu/python:3.12 image as a base image?

Useful comparison criteria are:

• Image size: the smaller, the better (faster start-up, lower storage costs)
• Minimality of contained components (i.e., is the image “distroless-like” or not?): this typically correlates with the image size criterion. Minimal images are typically better suited for production
• Number and severity of vulnerabilities affecting the image: check the most recent version/build, but also older builds. The fewer vulnerabilities, the better.
• Frequency of image rebuilds (also for older version tags): it’s OK if rebuilds happen rarely for minimal images, but for non-minimal ones, you want frequent image rebuilds (~weekly or more often)

Assuming that the images are hosted on Docker Hub, you can use the Docker Hub website to answer the first three criteria. But what about the last one? It’s not possible! Docker Hub (or any image registry, for that matter) only shows the age of an image tag, but not how often it has been rebuilt in the past. Image registries generally don’t seem to store the history of digests (pointing to the concrete image builds) of a version tag.

To solve that problem, I built a tool that answers that question: the Docker Tag Monitor.

Docker Tag Monitor (DTM)

DTM is an open-source web service. The source is available on GitHub. It tells you how often a Docker/OCI image changes over time. For this to work, DTM must have been monitoring the image/tag for a while. You can access a public instance for free here: https://docker-tag-monitor.augmentedmind.de. Or you can follow the README instructions to host your own instance (with Docker Compose or on Kubernetes).

DTM has a 3 tier architecture:

• Database: A PostgreSQL database that stores a (growing) list of monitored Docker/OCI images (and their version tags) and when they were updated
• Scraper: A Python script that iterates over the monitored images/tags and resolves the exact image digest (SHA265 checksum) via a HEAD request made to the image registry. This happens once per hour. Potential changes are recorded in the database. This scraper also cleans outdated entries from the database
• Reflex web app: A web app written in Python using Reflex, which generates a React-based frontend (that talks to a FastAPI-based backend) visualizing the data, allowing users to add more images and tags to the monitoring database

To make it useful, DTM comes with a prepopulated database, so that information is available for generally “popular” images from the beginning. To achieve this, the scraper enriches the database of monitored images/tags once per day, by adding/updating the 50 most “popular” Docker Hub images (taken from here), retrieving the 25 most recently updated tags for each of these images.

Features of DTM

Toplist / Overview

The start page (called “Overview”) shows a table of all monitored images and tags, sorting them by the number of recorded version tag updates (descending)

Image and tag search

On the Overview page, use the search box to find images/tags you are interested in. The image does not need to be on Docker Hub, other registries are also supported.

If the image or tag cannot be found, you can easily add it. Just type the full name of the image, then click the “here” link to open the details page, which will show you that the image and tag have been added to the list of monitored images/tags.

You can bookmark the deep link of the tag’s Details page in your browser and open the link in a few days or weeks, once more data has become available.

Image/tag details page

The details page of a specific tag of an image shows all digests as a table, together with either a weekly or monthly aggregate of the number of tag updates as a vertical bar chart:

Dark Mode

Some users prefer dark mode, so here you go.

Statistics

To get an idea of how well the regular scraping of image tag digests is going, I built this small monitoring page. I yet have to build internal monitoring that notifies me once scraping gets stuck completely.

Future work

DTM is at an early stage. There are still quite a few tasks left to be implemented. These are listed as open issues on GitHub.

Conclusion

With DTM you can finally determine how “alive” the Docker images of a project are. You can compare different vendors (of, say Java JRE images). And since DTM also stores all previous digests, you can even compare how the Docker images (for a specific tag) have changed over time, e.g. using CLIs such as docker scout compare or diffoci.

Let me know if DTM is useful to you, and feel free to report any issues (or even fix them with a PR).

I can definitely say that I learned quite a bit from developing this tool, which I’ll elaborate on in another article.