Kubernetes Development tools – the best 10 tools to speed up your daily development

In this article I discuss three types of Kubernetes development tools that speed up your daily development and debugging tasks: tools like Tilt that speed up development iteration cycles, general approaches for remote debugging with a debugger, and tools like Inspektor Gadget that help you diagnose problems in production.

Introduction

The increasing complexity of Kubernetes has made software development more difficult. To me, it seems that developing and debugging Kubernetes-based apps is an afterthought by the makers of Kubernetes. Kubernetes is designed to be a production system: it expects to be given production-ready OCI images, discouraging that you modify running containers.

In practice, production-ready software does not fall from the sky. To support you during your daily development and debugging tasks, I examine three categories of tools or problems:

• Tools to speed up development iteration cycles
• Debugging with a debugger (during development)
• Tools for debugging problems in production

I’m not going into too great details for each of the presented tools. This article is more of an overview to get you started.

Speed up development iteration cycles

To keep the differences between the development and production environment small, you should make the environments as similar as possible, which means that you should at least test your current code state in a Kubernetes cluster (vs. running it in a completely different set up, e.g. Docker-compose). This could be a remote staging or development cluster offered by your organization, or a local Kubernetes cluster, which I discussed in the first article of this series.

The term development iteration cycle refers to the time it takes between changing a line of code, and running it inside a container, in a pod, in the cluster. The manual approach, where you first build a new image (with a new tag), then update the referenced image in the Deployment or StatefulSet, would be very slow. Fortunately, there are a few tools to support you:

• Tools such as DevSpace, Tilt and Skaffold automate the just-mentioned manual steps of updating containers. They monitor your local source code folders. Whenever they detect a change, they automatically rebuild the (Docker) image, push it into some registry and update the Kubernetes manifests in the cluster. Alternatively (for interpreted languages that support hot-reloading), the tools can synchronize (as in: copy) your code into the running containers, without restarting them, which lets you almost instantly execute the new code. All these tools have in common that you configure them using some tool-specific configuration file, e.g. the Tiltfile for Tilt, or devspace.yaml for DevSpace. However, under the hood these tools all work differently, and you will have to try them out to determine which one works best for you.
• If you only need fast development iteration cycles for a specific microservice of your app, take a look at Telepresence. The idea is that all other microservices (that you are not currently working on) are installed “normally” into the cluster, and you have the Telepresence tool replace a selected microservice (the one for which you are currently writing code) with a locally running application (which you typically run natively on your host). Telepresence injects a proxy pod into the cluster which replaces the actual microservice, forwarding the traffic to your local application. Telepresence also redirects outbound traffic back into the cluster. The Bridge to Kubernetes feature of the VS Code (or Visual Studio) IDE is similar to Telepresence.

Debugging with a debugger

A big disadvantage of working in Kubernetes (over local / native development) is that you cannot easily debug containerized code with a debugger. Fortunately, it is possible to use debuggers using remote debugging, but you will have to find a concrete solution that works for your programming language and the IDE or debugger of your choice.

Generally speaking, remote debugging is split into two components: the server and the client. There are two approaches for establishing the connection between them:

• Server-on-host: on the host machine (e.g. in the IDE) you start a debug server. You have to add the debug client to your application’s code, so that it connects to the debug server on your host (e.g. behind a feature flag that should be disabled by default, for security reasons). Since applications or ports on your host are not reachable from within a cluster, you need a solution that sets up a proxy (a.k.a. reverse tunnel). Some solutions (like Telepresence, or VS Code’s Bridge to Kubernetes, discussed above) have this proxy feature built-in. But there are also dedicated solutions, such as ktunnel.
  • A concrete example that uses this approach is the PyCharm IDE for Python, see this example using ktunnel.
• Server-in-container: You augment your application code (which runs in a container in a Pod) with a debugging server, running on port X. Then you use kubectl port-forward to punch a hole from your host to port X, and connect to it from your debugger/IDE (which is the debugger client).
  • Examples for this approach:
    • Debugging NodeJS applications, by starting Node with the --inspect switch (see docs) to which many debuggers on the market can connect
    • Debugging Python applications with VS Code: use debugpy and follow instructions in its README

Debugging problems in production

Sometimes things start going wrong in production, and you need to find the culprit fast. These tools have your back:

• Inspektor Gadget: A collection of tools to debug and inspect Kubernetes applications – e.g. inspecting TCP traffic, start/stop of processes in a pod, or tracing file access.
• mizu: a graphical API traffic sniffer for Kubernetes that shows you all API communication between microservices – useful to debug and troubleshoot problems. Similar to TCPDump and Wireshark, but for Kubernetes.
• koolkits: if you can use the ephemeral debug container feature of Kubernetes, koolkits offers prebuilt OCI images you can use for your debug containers, for languages such as Python, Node or Java. These images contain typical tools you need for debugging apps written in those respective languages.

Miscellaneous tools

There are many more tools that make developing applications for Kubernetes easier:

• Cluster introspection tools, which I discussed in this article; they visually present your Kubernetes workloads
• Leapp: a GUI app that handles the management and security of your cloud credentials
• Monokle: a dedicated Desktop IDE for Kubernetes / Kustomize / Helm manifests. Has many features, such as refactoring, rendering Helm charts as preview, using forms with auto-complete instead of manually typing YAML, linting, OPA policy integration, or diffing your local manifest with those deployed in a cluster.

Conclusion

With the help of the Kubernetes development tools mentioned in this article, you can greatly simplify your development workflow for applications running in a Kubernetes cluster. However, you should expect a steep learning curve, initially, while you experiment with all the tools and get them to work. Make sure to consult your team, because everyone should be comfortable with your chosen tool stack.

Which Kubernetes-related development tools do you use? Let me know in the comments!