The below picture is from GitHub displaying the comprehensive stats for Kubernetes Project known for Production-Grade Container Scheduling and Management.
The project is rated with over 62K stars (as on the publishing date of this blog) and is skillfully developed by more than 2400 contributors from around the globe. Kubernetes has become the de-facto standard for container orchestration. The way Kubernetes has evolved and been adopted at a fierce rate since its announcement is phenomenal. On the other hand, while solving some of the critical challenges, it did introduce new ones.
Monitoring is inhibiting the Kubernetes use
It’s not surprising anymore when engineers list monitoring as one of the crucial obstacles for adopting Kubernetes. No matter what, monitoring for the distributed environments has never been easy, and Kubernetes is tagging more complexity with it. Kubernetes belongs to the open-source community. So, that open-source community has developed various tools to simplify the complexity that Kubernetes brings along with it.
The tools help bring different things under control. Such as some help with logs, others with metrics, some are data collectors while others serve the user with an interface for operating Kubernetes from a birds-eye view. The spectrum of such tools that too open-source is essential for the new modern ecosystem. These tools add to the abilities of Kubernetes.
If you are using Google Kubernetes Engine, here’s the perfect read for you: Ok! Google…. What are the Kubernetes best practices?
After honest reviews and taking opinions from the real-life users of the tools, we’ve compiled the list of top 10 open-source tools to monitor Kubernetes.
Prometheus is a monitoring and alerting toolkit, originally built at SoundCloud. Now, it is governed by the CNCF (Cloud Native Computing Foundation). It joined CNCF in 2016 as the second-ever hosted project, after Kubernetes. It provides the following monitoring capabilities:
- Users can access a multi-dimensional data model with time series data identified by “name” and “key/value pairs.”
- The dimensionality mentioned above can be leveraged via PromQL (a flexible query language).
- No need to rely on distributed storage, get the single server nodes are autonomous.
- It is developed to collect time-series data via a pull model over HTTP.
- Multiple modes are available for graph and dashboard support.
Grafana allows you to monitor your Kubernetes cluster’s performance. There are several reasons why there is so much fan following for Grafana. The user gets separate dashboards each for Cluster, Node, Pod/Container, and Deployment. The monitoring capabilities of Grafana include the following metrics:
- Pod Capacity/Usage
- Memory Capacity/Usage
- CPU Capacity/Usage
- Disk Capacity/Usage
- Overview of Nodes, Pods, and Containers
- Memory Available
- Load per CPU
- Read IOPS
- Write IOPS
- Network Traffic/second, Packets/second, Errors/second
- Memory Usage
- Network Traffic
- CPU Usage
- Read IOPS
- Write IOPS
3. ELK (Elastic Stack)
When asked about the most popular open-source solution for logging Kubernetes, everybody answered ELK Stack.
The ELK Stack consists of Elasticsearch, Logstash, Kibana, and Beats. Each component in the stack performs dedicated tasks in the logging pipeline.
- Elasticsearch: designed to be scalable and perform well, even handling the storage and search across millions of docs.
- Logstash: capable of aggregating and processing logs and finally send the processed data for storage.
- Kibana: an analysis interface to help users make sense of data.
- Beats: these are the lightweight data shippers.
4. Fluentd configured with FluentBit
Yet another couple of projects governed by the Cloud Native Computing Foundation (CNCF). Fluentd is a CNCF graduated project.
Fluentd is a specialized data collector that unifies the data collection and consumption for better use and understanding of data. On the other hand, FluentBit provides built-in metrics and general-purpose output interfaces for centralized collectors such as Fluentd. That’s why “Fluentd configured with FluentBit.“
They are designed for seamless integration with Kubernetes, enrich data with compatible pod and container metadata with minimum resource footprint. This is the best integration of open-source tools for monitoring Kubernetes.
cAdvisor is one of the recommended tools (also the basic tool yet very useful) for monitoring running containers. It is built into Kubernetes and integrated into the Kubelet binary.
The technical difference between cAdvisor and other agents is that it is not deployed per pod but on the node level. It automatically discovers every single running container on a machine and collects system metrics such as memory, CPU, network, etc.
One of the advantages of cAdvisor is that it supports multiple endpoints despite having limited functionality. This is the reason why the cAdvisor is on our top 10 list.
kubewatch is a simple tool to track specific events for Kubernetes containers. It also gives the ability to push notifications on tracked events to different endpoints such as Slack and PagerDuty.
In simple words, kubewatch is the go-to tool to receive push notifications on available collaboration apps or notification channels. Users need to run it in their k8s cluster, and the event notifications will be received via webhooks.
Suggested Read: Kubernetes vs Docker | A Complete breakdown (Note: this article is dedicated to clear out the confusion perceived by container community between Kubernetes and Docker. Both of the platforms have different use cases.)
It’s not a monitoring solution for Kubernetes. (as per the official documentation) Yeah, you read that right then why it is even on the list?
Well, it provides a common operational picture for multiple k8s clusters. It shows the capacity and resource usage for nodes and individual pods with animation, like shown below.
It’s a read-only operational view for k8s clusters. One thing that needs to be clear is that it is neither a replacement for the Kubernetes Dashboard nor a monitoring solution. It helps users to quickly figure out the usage metrics without doing too much hustle. The simple pictorial representations are necessary to cut through too much complexity, so Kub-ops-view recommended.
kube-state-metrics is designed specifically to listen to the Kubernetes API server and expose the cluster-level metrics. It currently exposes about 29 metrics (at the time of publishing this blog), which covers almost every metric for which users need to gather data.
It generates metrics without any modification whatsoever to Kubernetes API. So, users can rely on the same grade of stability as offered by Kubernetes API. Users get only the raw data unmodified from API and need to perform heuristics to be able to get the exact values, as shown by kubectl. Real-life Kubernetes practitioners recommend kube-state-metrics if someone is struggling to use Prometheus for monitoring k8s.
The Kubernetes community loves Jaeger, so it has to be on our list. Why? Out of the available tracing tools, Jaeger is the best tool for distributed tracing, which is widely acknowledged by Kubernetes practitioners as monitoring and troubleshooting best practices.
It was developed by Uber, inspired by OpenZipkin and Dapper (other tracing tools), and donated to CNCF. The project got graduated in 2019.
Users can monitor the distributed transaction, perform root cause analysis, service dependency analysis, and performance optimization. Jaeger is easy to use and provides a seamless user interface along with various instrumentation options, but there are limited backend integrations.
10. Weave Scope
Weave Scope is a troubleshooting and monitoring tool for Docker and Kubernetes, developed by folks at Weaveworks. It automatically generates a map of user applications to understand, monitor, and control containerized applications.
Unlike kub-ops-view, Weave Scope, apart from providing operational views, also provides the ability to manage containers and run diagnostic commands that, too, within the interface.
The user interface of the tool is nice and clean. Users will be able to see the application, monitor the infrastructure it is deployed on, and the relationships between the different components. The only aspect that Weave Scope lacks as a tool is the analytical depth.
The list mentioned above is the effort of the highly engaged and dedicated open-source community working towards the betterment of Kubernetes. If you are looking for tools to design your monitoring stack for Kubernetes, then this list is the perfect place to start.
Even if you are skeptical about using open-source monitoring tools, then don’t worry, these tools are easy to test and deploy. Set up a small sandbox environment, try to start small, and understand whether you got what you were looking for in the first place.
As we have already praised the active community behind the development of Kubernetes, we will be able to see additional developments ready to change the way we leverage microservices.
Suggested Reads: Why Kubernetes? – 11 benefits that simplify container management