This article takes a detailed look at image signatures created by Notation, which is one of several tools to create and verify Docker images.
Docker Image signing and attestation series
This article is part of a multi-part series:
- Overview: explains concepts and tool landscape, provides high-level recommendations
- (This article) Docker Image signing with Notation: detailed analysis of the Notation tool and its signatures
- Docker Image signing and attestation with Cosign: detailed analysis of Cosign, for image signatures and attestations
- Docker Image attestation with GitHub attestations: detailed analysis of GitHub attestations and its pros&cons
- Docker Image attestations with BuildKit: detailed analysis of BuildKit’s attestations
Introduction
Notation (formerly known as “Notary V2”) is one of the CLI tools under the Notary Project umbrella. Notary Project is a set of specifications and tools to sign and verify container images and other OCI artifacts. Today, the most important outcome of Notary Project that you should know of is Notation and its specification for image signatures. AFAICT, no other projects/tools (other than Notation CLI) create signatures using that specification.
Notation uses a key/certificate-based approach, where Notation generates a self-signed key and stores it in a cloud-based Key Management Service (KMS) – or you generate it using the KMS. Notation supports the KMS of AWS, Azure, and Venafi CodeSign. Notation can store and retrieve the certificates and keys (and use them for signing/verifying) directly from the KMS (at least with some of them). You could also create certificates as local files (see quick start), but then you would have to manually make sure to correctly distribute them to new machines (e.g. an ephemeral CI/CD runner, or a Kubernetes Pod).
Notation can sign (and verify) both regular container images and non-images, that is, arbitrary OCI artifacts. “OCI” is the Open Container Initiative, a working group that defines open standards for image registries (to abstract away from proprietary vendors like Docker). The OCI defined the image registry API endpoints (“distribution spec”) and file formats (“image spec”) that let you use “image registries” as storage for arbitrary (binary) file “artifacts” (that are not Docker images), e.g. SBOMs, vulnerability reports, or Helm charts. If you are unfamiliar with OCI artifacts, please read this excellent introduction.
Signing with Notation
These are the steps to sign a container image with Notation:
- On a new/fresh machine, you either create a self-signed key and certificate once (for the first time) and store the key somewhere safe, e.g. in a KMS, or you configure Notation to use a previously generated key/certificate from a KMS. In both cases, refer to Notation’s KMS-specific guides (see AWS guide, Azure KeyVault guide, or Venafi CodeSign). The concrete instructions differ between each KMS.
- You build and push your container image to your registry, using any tool (e.g. “docker build” or kaniko). The build tool returns the SHA256 digest of the pushed image manifest
- You run “
notation sign [--signature-format cose] <image-with-digest>
”. Notation creates a signature for the image manifest and pushes the signature as OCI artifact to the same image registry, making it discoverable under the new image tag named “sha256-<image-digest>
”. Notation uses “OCI references” which are explained in the box below. The basic idea is that when you tell Notation to verify an image, Notation can automatically find its corresponding signature(s). You don’t need to tell Notation where to download the signature from, nor do you have to download the signature manually first and provide it as a file (these are steps you traditionally need to do when verifying files with other tools).- Note: Notation supports two signature formats. By default, a textual (JSON-based) format is used (JWS), but you can instead use COSE, which is a binary format producing smaller signature file sizes.
- Since you most likely want to sign in a CI/CD pipeline, there are wrappers for the Notation CLI for Azure DevOps (see here) and GitHub Actions (see here).
Let’s examine a concrete example of the created manifests (note: Notation does not alter the manifest of the image it signs).
Automated verification
To automatically verify image signatures, you have the following options:
- A Kubernetes operator, configured to reject deploying images that are not signed (or signed with unknown certificates). Each solution uses a different syntax or approach to define the “trust policy”, which specifies which signing certificate(s) you trust:
- FluxCD (see announcement)
- Kyverno (docs)
- OPA Gatekeeper with Ratify (docs)
- Configure Harbor (see docs) to refuse pulls of unsigned images. Unfortunately, you cannot configure a trust policy in Harbor, such that you cannot configure the specific signing identities you deem trustworthy. This makes the approach rather useless.
- Verification in your deployment pipeline, e.g. using the wrappers for Azure DevOps (see here) or GitHub Actions (see here). Here, you also need to define a trust policy.
See also the adoptions documentation page for a few more options.
Conclusion
Notation never took off on a larger scale for open-source projects. One exception is Bitnami, who sign their images with Notation, see here. However, Notation can be a good fit for signing and verifying your (closed-source) images in an enterprise/corporate setting. Here, the most viable alternative, Cosign keyless signing, would either force you to use the public Rekor instance (publishing build details to the whole world) or require hosting a private Rekor instance.
Even though Notation requires you to take care of handling certificates and private signing keys, this handling is much simpler than other solutions, such as Docker Content Trust.