renovate bot introduction feature

Automatic dependency updates – a Renovate Bot introduction

by

Renovate bot is a tool that automatically updates third-party dependencies declared in your Git repository via pull requests. In this Renovate bot introduction I explain how the bot works, how to operate/run the bot yourself (if necessary), and how you can configure the bot’s behavior for each repository.

Table Of Contents
  1. Renovate bot introduction
  2. Supported SCM platforms and dependency types
  3. Using Renovate Bot
  4. Understanding the configuration file
  5. Conclusion

Renovate bot introduction

Renovate (Bot) is a CLI tool that regularly scans your Git repositories for outdated dependencies. If it finds any, it automatically creates a pull request (PR) in which the dependency is updated (for GitLab, the terminology is Merge Request). Renovate is highly configurable. You can tell it to leave the PR alone (so that you always have to approve the merge yourself), or even tell Renovate to fully automatically merge the PR (after the tests run by the CI pipeline for the PR have successfully passed). There are tons of other configuration options, including the ability to customize the configuration for specific dependencies – e.g. telling Renovate to ignore a specific dependency, because you are using an outdated version on purpose, for good reasons.

Supported SCM platforms and dependency types

Because interacting with the SCM (e.g. to authenticate and create or merge PRs), the Renovate bot needs to support the SCM-proprietary APIs. Consequently, Renovate only supports a selected set of SCMs, including GitLab, GitHub, Bitbucket, Azure DevOps and Gitea (including the SaaS and self-hosted alternatives). These are probably the most popular SCM platforms supporting PRs anyway.

Similarly, for the dependency types, Renovate supports most of the popular dependency systems (see here for a complete list), such as Gradle/Maven (Java) or Cargo (Rust). You could argue that those (popular) dependency types that are not supported (e.g. CMake for C/C++) also don’t offer a central repository that Renovate could query, to detect the most current version of a dependency.

Using Renovate Bot

The way Renovate is configured to run highly depends on the SCM platform. If you happen to use GitHub.com (SaaS), then you don’t need to configure any operations-related aspects. As documented here, the makers of Renovate (WhiteSource) provide (and maintain) the Renovate GitHub App, which you can install (for your GitHub account) with a single click, instructing it which GitHub repo(s) it should regularly visit. In other words: you don’t have to worry about things like making sure that the most recent Renovate CLI version is run, and that it runs regularly – WhiteSource does all that for you.

However, if you use any other SCM platform, e.g. GitLab (self-hosted or SaaS), you have to take care of these aspects yourself!

The following sections go into further detail about how you can operate the Renovate bot yourself (Operations section), and how developers can configure what the bot does when visiting a repo (Development section), including the ability to use a different configuration for each repository.

Operations (self-hosting)

The bot can be run in a few ways (see docs), such as:

  • Run as NPM CLI tool (installed via “npm install -g renovate“, in an environment where npm is available), e.g. triggered regularly via a cron job
  • Run as a Docker image: for instance, you install Docker on a Linux box, and have a cron job create and run a Docker container for the ready-to-use renovate/renovate image in regular intervals.
  • Run in Kubernetes, as CronJob
  • Run as pipeline in GitLab CI, which you can regularly trigger using a schedule (docs)

Irrespective of the concrete environment you choose, you need to configure the bot so that it knows which SCM instance to connect to, and which repos to scan. The configuration can be either done as a config.js / config.json file, or via CLI arguments or environment variables. The configuration options include:

  • The type of SCM you are targeting with the bot (e.g. gitlab vs. gitea)
  • The SCM server URL (e.g. https://github.company.com/api/v3 for GitHub Enterprise, or https://gitlab.company.com/api/v4 for a self-hosted GitLab instance)
  • Authentication credentials: the typical approach is that you create a separate (Bitbucket / GitLab / GitHub / Gitea / Azure DevOps) account for the bot, and then create some kind of personal access token for that account. You configure the bot to use this personal access token, and thus any actions it performs (like creating a branch or PR) will be done with that user account.
  • A GitHub.com personal access token: even though your chosen SCM platform is not GitHub.com (since you are choosing the self-hosted approach), the Renovate bot still needs a GitHub.com personal access token, to avoid that you hit the rate limit from the github.com API (which the bot queries often, to find new releases). I recommend you create a new GitHub account for this, and create a personal access token as described here. You don’t need to check any checkboxes in the scopes / permissions dialog.
  • The Git identity the bot should use for commits it creates (e.g. “Renovate Bot <bot@your-org.com>“)
  • Which repositories the bot should scan: you can either configure the autodiscover mode, where Renovate will run on every repository that the bot’s SCM account can access, or a list of repositories you explicitly declare
    • Note: if you use repositories, the syntax varies depending on whether you use a config.json / config.ts file, vs. using environment variables. With json/ts files, you use the usual JSON notation (e.g. ["abc/def","abc/xyz"]), but when using environment variables, you need to configure it as follows: RENOVATE_REPOSITORIES=abc/def,abc/xyz
  • Various other options, as explained in the Self Hosted Configuration documentation page.

Once the bot is running (e.g. being triggered once per hour via cron or a GitLab CI schedule), the development teams need to tell it to visit their repositories. This either happens when a developer invites the Renovate Bot SCM account into their repo (if the bot was configured in autodiscover mode), or the developer asks the bot operator to add the repo’s name explicitly (when the bot was configured with a list of repositories).

If you are interested in running your own Renovate bot instance in GitLab, take a look at my demo repository that illustrates how the bot can automatically update itself fully automatically (without any user intervention).

Development

Whenever the Renovate bot scans your repository, it uses a configuration file such as renovate.json5, located at the root of your repo. It tells the bot how it should behave specifically for this repository. The reference for what you can put into this file is found on the Repository configuration options documentation page. Don’t confuse this renovate.json5 file with the config.json file that you (may have) used to configure the bot’s operational aspects (in case you are self-hosting the bot, which I explained above)!

To avoid that the Renovate bot wreaks havoc the first time it visits your repo (due to an incorrect configuration), the bot’s default behavior is to first initiate an onboarding phase (as described in the docs). The bot basically runs the following logic:

  • Is there a renovate.json5 file (or one of its alternatives) in the root of the repo’s default branch (typically master / main)?
    • If yes: the bot assumes onboarding has completed, and it will scan your repo for outdated dependencies
    • If not:
      • Is there already a PR named “Configure Renovate” in the repo?
        • If yes:
          • If it is still open, or already closed (but not merged): don’t further scan the repo – wait for the user to merge that PR first
          • If it is merged: don’t further scan the repo – there used to be a renovate.json5 file in the past (created when merging the onboarding PR) in the default branch. But since then, the devs have deleted the file on purpose, indicating that they no longer want the repository to be scanned.
        • If not: the bot creates a new onboarding PR titled “Configure Renovate” (and a corresponding Git branch, typically named “renovate/configure“) that contains a commit with a new renovate.json[5] file.

The description text of the onboarding PR contains extensive details, including the detected dependency types (e.g. pip, docker, etc.), a summary of the configuration in understandable plain English (i.e., not just a print-out of the config array), and what other PRs the bot will create on its next visit, given that you merged the onboarding PR by then.

The initial content of the renovate.json5 file will be a very basic configuration. Typically, the developer will tweak its content (prior to merging the onboarding PR), by checking out the “renovate/configure” branch and updating the file’s content with a new commit. Note that the developer may want to use the renovate-config-validator CLI tool (see docs) to validate the renovate.json5 file, prior to committing. The next time the Renovate bot visits the repo (which may take a little while), it will detect that you changed the renovate.json5 file in the (still-open) onboarding PR, and update the PR’s description accordingly.

Of course, you can still update the renovate.json5 file even after the onboarding PR was merged. The Renovate bot will always use the most recent renovate.json5 file it finds in the repository’s default branch.

Understanding the configuration file

A developer who wants to refine the Renovate configuration for their repo for the first time will be overwhelmed by Renovate’s configuration flexibility. On my first attempt, I had questions like these:

Understanding default values

In general, it is important to understand that many of the options explained in the Configuration Options page are already summarized / grouped into presets. To understand how this works, I highly recommend reading the first two sections of the Shareable Config Presets page carefully, because it does not only explain how to share presets (which you might not care about), but also explains how presets work in general. Once you read it, it should be evident to you what happens if your renovate.json5 looks as simple as this one:


{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"]
}
Code language: JSON / JSON with Comments (json)

First, the config:base preset (see Full Config Presets docs) is itself expanded to many other presets (such as ":ignoreModulesAndTests", which is a shorthand for "default:ignoreModulesAndTests", and therefore that preset is explained on the Default Presets page). Next, these more concrete presets are translated to actual configuration options, as found on the Configuration Options page.

If you want to dive deeper into the topic, it makes sense to study these presets referenced by config:base more closely, to understand which of these default settings may not be suitable for your project. But you don’t have to go down this route – you can just leave the defaults, and then tweak the rules over time, whenever you discover that the bot’s behavior is not suitable.

The main problem with the default config:base preset is that the bot creates PRs (which ask you to update some dependency) immediately after the bot discovers an outdated dependency. If that dependency is updated often (e.g. multiple times per day), this also results in many PRs in a short amount of time, and a lot of (email) spam and merging-work. The Noise reduction page has pointers on how to handle this problem.

Understanding packageRules to tweak your config

The packageRules element of the renovate.json5 file allows you to select one specific dependency (by name), or multiple dependencies (via regular expressions), and fine-tune / override the configuration options for these dependencies. packageRules is an array of (JSON) objects. The allowed keys in each object can be one or more of the sub-keys explained in the packagesRules documentation (e.g. matchPackageNames), but can also be (most of the) other keys documented on the Configuration Options page – e.g. automerge.

You should realize that all rule objects in the array are valid at the same time! The matching-logic does not work as you know it from many other systems, where rules #3 and #4 are no longer evaluated if rule #2 was a match. However, the order of rule objects in the packageRules array does matter: rules declared later (towards the end of the array) overwrite values of an also-matching rule declared earlier.

Let’s look at a concrete example:

{
...  // other stuff here, e.g. "extends": ["config:base"],
"packageRules": [
    {
      // Whenever Renovate finds updates for any (*) package (except those starting with
      // @material-ui, @types, react, or typescript), it will create a single(!) branch+PR for
      // all the detected changes, instead of creating multiple branches/PRs (one per package)
      "matchPackagePatterns": [ "*" ],
      "excludePackagePatterns": [
        "^@material-ui",
        "^@types",
        "^react",
        "typescript"
      ],
      "groupName": "deps" // note: "deps" is just a self-chosen name without any meaning
    },
    {
      // Renovate will create a single branch/PR for any changes detected for NPM/Yarn
      // dependencies of type "dependencies" and "devDependencies" which start with @material-ui
      "matchPackagePatterns": [ "^@material-ui" ],
      "groupName": "Material-UI",
      "matchDepTypes": [
        "dependencies",
        "devDependencies"
      ]
    },
    ...
  ],
...
}Code language: JSON / JSON with Comments (json)

I provide a list of recommended configuration options in the cheat sheet article.

Other selectors useful to tweak the configuration

Apart from packageRules there are also other ways to override configuration options for a specific use case:

  • Programming languages: you can create an entry such as golang {"addLabels": "go"} in your renovate.json5 to tell Renovate bot to add a “go” label to all those PRs it created because of a Go dependency being outdated. Instead of golang, there are other keywords for the different languages, i.e., python, ruby, php, java, node or js.
  • Update types (more details below): something like major {automerge: false} tells Renovate to never automatically merge major versions.

Understanding update types

When the author of a third-party dependency offers an update, there are different types of updates. Renovate distinguishes between the following different types, which you should understand properly:

  • major: when the dependency version changed from e.g. 1.0.3 to 2.0.0
  • minor: when the dependency version changed from e.g. 1.0.3 to 1.1.0
  • patch: when the dependency version changed from e.g. 1.0.3 to 1.0.4
  • digest: when the dependency’s version did not change, but the checksum of the binaries have updated. At the time of writing, Renovate bot only seems to be detect this update type for Docker images. As an example, consider a specific Python Docker image you use in your software, e.g. python:3.9.2 – this particular image tag still changes regularly over time, because the Python release team has a CI pipeline set up which rebuilds the python:3.9.2 image regularly (e.g. to keep up with the latest security patches in the base OS image). The digest will be different for each built image.
  • rollback: applies when Renovate detected that the concrete version of the dependency referenced in your Git repo is no longer available in the corresponding artifact repository, e.g. because the dependency was removed by the author as it was incorrectly built or buggy. By default, Renovate bot will leave such dependencies untouched, but you can enable the rollbackPrs feature to change this behavior.

These update types are used in different places in the renovate.json5 file, e.g. as value in the matchUpdateTypes element (within packageRules), in bumpVersion, or as a key on the root level of the configuration JSON object (as I just illustrated above, e.g. major {automerge: true}).

Conclusion

Renovate is the perfect tool to automate the maintenance of third-party dependencies. It offers a bit more configuration flexibility, compared to other solutions such as GitHub’s dependabot. This is, however, also a bit of a downside! You should expect increased workload and “annoyed” developers at the beginning (during the onboarding phase), because it takes time to configure the Renovate bot appropriately. The configuration is a bit tricky due to the huge flexibility.

I recommend that whoever introduces Renovate bot reads the docs carefully and builds a cheat sheet for the rest of your company (your developers), which includes the most common configuration options. Assume that your developers don’t want to spend a huge amount of time learning Renovate’s configuration syntax, but the default values (of the config:base preset) are often unusable in practice, leading to a massive amount of spam. Take a look at the cheat sheet I created in this article!

By the way: if you are operating the bot yourself, I recommend you change the configuration file name created as part of the onboarding from renovate.json to renovate.json5. This lets your developers add explanatory comments to their JSON file. You can make this change in the bot’s configuration (see onboardingConfigFileName).

Leave a Comment