[Prometheus](https://prometheus.io/) is a high-scalable open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes [container orchestration platform](https://devopscube.com/docker-container-clustering-tools/). Also, In the [observability](https://devopscube.com/what-is-observability/) space, it is gaining huge popularity as it helps with metrics and alerts.
The **prometheus.yaml** contains all the configurations to discover pods and services running in the Kubernetes cluster dynamically.
We have the following [scrape jobs](https://prometheus.io/docs/concepts/jobs_instances/) in our Prometheus scrape configuration.
- kubernetes-apiservers: It gets all the metrics from the API servers.
- kubernetes-nodes: It collects all the kubernetes node metrics.
- kubernetes-pods: All the pod metrics get discovered if the pod metadata is annotated with prometheus.io/scrape and prometheus.io/port annotations.
- kubernetes-cadvisor: Collects all cAdvisor metrics.
- kubernetes-service-endpoints: All the Service endpoints are scrapped if the service metadata is annotated with prometheus.io/scrape and prometheus.io/port annotations. It can be used for black-box monitoring.
**prometheus.rules** contains all the alert rules for sending alerts to the Alertmanager.
**Setting Up Kube State Metrics**
Kube state metrics service will provide many metrics which is not available by default. Please make sure you deploy Kube state metrics to monitor all your kubernetes API objects like deployments, pods, jobs, cronjobs etc.
- kubectl apply -f kube-state-metrics-configs/
**Setting Up Alertmanager**
Alertmanager handles all the alerting mechanisms for Prometheus metrics. There are many integrations available to receive alerts from the Alertmanager (Slack, email, API endpoints, etc)
- kubectl apply -f kubernetes-alert-manager/
**Setting Up Grafana**
Using Grafana you can create dashboards from Prometheus metrics to monitor the kubernetes cluster.
The best part is, you don’t have to write all the PromQL queries for the dashboards. There are many community dashboard templates available for Kubernetes. You can import it and modify it as per your needs. I have covered it in the article.
- kubectl apply -f kubernetes-grafana/
**Setting Up Node Exporter**
Node Exporter will provide all the Linux system-level metrics of all Kubernetes nodes.
The scrape config for node-exporter is part of the Prometheus config map. Once you deploy the node-exporter, you should see node-exporter targets and metrics in Prometheus.
Creating a Kubernetes dashboard from the Grafana template is pretty easy. There are many prebuilt Grafana templates available for Kubernetes. You can easily have prebuilt dashboards for ingress controllers, volumes, API servers, Prometheus metrics, and much more.
It’s not recommended that you use your personal password for this, so you should create an App Password. To do that, go to Account Settings -> Security -> Signing in to Google -> App password (if you don’t see App password as an option, you probably haven’t set up 2-Step Verification and will need to do that first). Copy the newly-created password
If you want to receive notifications via Slack, you should be part of a Slack workspace. To set up alerting in your Slack workspace, you’re going to need a Slack API URL. Go to Slack -> Administration -> Manage apps.
Next, specify in which channel you’d like to receive notifications from Alertmanager. (I’ve created #random channel). After you confirm and add Incoming WebHooks integration, webhook URL (which is your Slack API URL) is displayed. Copy it.
