Secure health checks for minimal container images

Secure health checks for minimal container images

by

This article explains how to add the “lprobe” CLI tool to your container image, which executes health checks/probes triggered by the Docker daemon. In contrast to curl or other alternatives, lprobe is safe because it can only connect to localhost. It supports TCP and HTTP health checks.

Table Of Contents
  1. Introduction
  2. Problem with Docker health checks
  3. Why adding curl to an image is a bad idea
  4. Introducing LProbe
  5. How to use LProbe
  6. Conclusion

Introduction

Minimal (a.k.a. distroless / chiseled / hardened) container images greatly improve container security: fewer packages mean fewer vulnerabilities, less CVE noise, and fewer tools an attacker can abuse in case of an successful attack. I’ve covered the motivation, trade-offs, and available minimal images in my previous blog posts.

But once you actually use minimal images, you’ll hit the following operational problem:

Problem with Docker health checks

In Kubernetes, there is no issue. If your app exposes a /health HTTP endpoint, you can use an httpGet probe and the kubelet will perform the HTTP request for you. No tool like curl is required in the container image.

Docker’s HEALTHCHECK, on the other hand, runs inside the container and supports only “run a command” (CMD ...). That means the probing tool must be present in the image.

In non-minimal images, people reach for curl or wget and call it a day, because it works locally, in CI and in production. But in minimal images, those tools are intentionally missing, for good reasons.

Why adding curl to an image is a bad idea

Copying curl/wget into a minimal image only to make HEALTHCHECK work undermines the security goal of minimal images:

  • Those tools can connect to any host on the network/Internet, not just your application.
  • If an attacker gets code execution in your container, curl/wget become “download and run payload” helpers.
  • These tools expand your runtime surface area (more code, more dependencies, more CVEs, more triage).

Introducing LProbe

To solve this, CLIs exist that are like curl, but limited to connecting only to localhost. My research found the following two currently-maintained ones, both of which have problems:

  •  https://github.com/fivexl/lprobe, a Go CLI that supports HTTP, HTTPS and gRPC. To support those, go.mod lists third-party dependencies (such as golang.org/x/exp or google.golang.org/grpc) which need to be continuously updated. The author does not do that automatically (no Renovate Bot present!). Similarly, the author proposes to use their binary builds in your image. This is problematic, because binaries accumulate CVEs very quickly (even just Go itself gets new CVEs on a monthly basis).
  • https://github.com/tarampampam/microcheck, a CLI written in ~1500 lines of C code. I’m not gonna vet that code base. Using a (probably) vibe-coded C code base and hoping it to be secure is… what’s the word … reckless?

That’s why I decided to fork the first tool, lprobe, and apply the following modifications:

  • Strip support for HTTPS and gRPC, since they seem to be “niche” use cases, and because stripping those gets rid of all external Go dependencies
  • Add support for generic TCP port checks
  • Advertise “from source” builds rather than binary builds

You find my fork on https://github.com/MShekow/local-health-check. Like lprobe, it’s designed to be copied into minimal Docker/OCI images as a more secure health checks replacement for curl/wget/netcat:

  • Supports HTTP or TCP checks:
    • HTTP probe performs a GET http://localhost:<port>/<configurable-endpoint>
    • TCP probe checks whether localhost:<port> is open
  • Exits with 0 on success, 1 on error (and prints what went wrong)
  • Simple Go codebase (no external Go dependencies), fewer than 200 lines of code (easy to vet yourself), 8 MB static binary
  • Allows customizing the port number, the expected HTTP response status codes, the User-Agent header, or the timeouts
  • Supports IPv6

How to use LProbe

Instead of downloading a prebuilt binary (which gets stale quickly and should be rebuilt when Go receives security fixes), you should build LProbe from source directly in your own multi-stage Dockerfile:

# Multi-stage build example for lprobe with web server
# Stage 1: Build the lprobe binary
FROM golang:1-alpine AS lprobe
WORKDIR /build
ADD https://github.com/MShekow/local-health-check.git#main .
RUN CGO_ENABLED=0 go build -o lprobe .

# Final stage: Setup example web server with health check
FROM nginx:alpine
COPY --from=lprobe /build/lprobe /bin/lprobe
HEALTHCHECK --interval=15s --timeout=5s --start-period=5s --retries=3 \
    CMD [ "lprobe", "-port=80", "-endpoint=/" ]Code language: Dockerfile (dockerfile)

This approach uses the most current Go 1.x version to compile. So if your CVE scanner finds vulnerabilities in lprobe, just rebuild the image. We don’t need Renovate, and we don’t need to constantly rebuild binaries (nor do we need any build pipeline).

Conclusion

Minimal images are worth it, but they force you to be intentional about what you include at runtime. LProbe is a small missing piece that restores Docker health checks without reintroducing heavyweight networking tools into your container.

Thanks to GitHub Copilot and the Claude Sonnet models, changing the existing lprobe code base was quite effortless.

Please give LProbe a try and let me know what you think.

Leave a Comment