Skip to content
Snippets Groups Projects
Commit 63119aeb authored by suraksha.shetty's avatar suraksha.shetty
Browse files

update

parent 871b1ff6
Branches
No related tags found
No related merge requests found
Showing
with 0 additions and 374 deletions
**Infrastructure Monitoring**
[Prometheus](https://prometheus.io/) is a high-scalable open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes [container orchestration platform](https://devopscube.com/docker-container-clustering-tools/). Also, In the [observability](https://devopscube.com/what-is-observability/) space, it is gaining huge popularity as it helps with metrics and alerts.
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.001.png)
**Connect to the Kubernetes Cluster**
- git clone <https://github.com/mohsinniveus/monitorining>
- cd monitorining/Infra-Monitoring
**Prometheus Monitoring Setup on Kubernetes**
Connect to your Kubernetes cluster and make sure you have admin privileges to create cluster roles.
- kubectl create namespace monitoring
- kubectl create clusterrolebinding YOURNAME-cluster-admin-binding --clusterrole=cluster-admin --user=123[@niveussolutions.com](mailto:suraksha.shetty@niveussolutions.com)
- kubectl apply -f kubernetes-prometheus/
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.002.png)
The **prometheus.yaml** contains all the configurations to discover pods and services running in the Kubernetes cluster dynamically.
We have the following [scrape jobs](https://prometheus.io/docs/concepts/jobs_instances/) in our Prometheus scrape configuration.
- kubernetes-apiservers: It gets all the metrics from the API servers.
- kubernetes-nodes: It collects all the kubernetes node metrics.
- kubernetes-pods: All the pod metrics get discovered if the pod metadata is annotated with prometheus.io/scrape and prometheus.io/port annotations.
- kubernetes-cadvisor: Collects all cAdvisor metrics.
- kubernetes-service-endpoints: All the Service endpoints are scrapped if the service metadata is annotated with prometheus.io/scrape and prometheus.io/port annotations. It can be used for black-box monitoring.
**prometheus.rules** contains all the alert rules for sending alerts to the Alertmanager.
**Setting Up Kube State Metrics**
Kube state metrics service will provide many metrics which is not available by default. Please make sure you deploy Kube state metrics to monitor all your kubernetes API objects like deployments, pods, jobs, cronjobs etc.
- kubectl apply -f kube-state-metrics-configs/
**Setting Up Alertmanager**
Alertmanager handles all the alerting mechanisms for Prometheus metrics. There are many integrations available to receive alerts from the Alertmanager (Slack, email, API endpoints, etc)
- kubectl apply -f kubernetes-alert-manager/
**Setting Up Grafana**
Using Grafana you can create dashboards from Prometheus metrics to monitor the kubernetes cluster.
The best part is, you don’t have to write all the PromQL queries for the dashboards. There are many community dashboard templates available for Kubernetes. You can import it and modify it as per your needs. I have covered it in the article.
- kubectl apply -f kubernetes-grafana/
**Setting Up Node Exporter**
Node Exporter will provide all the Linux system-level metrics of all Kubernetes nodes.
The scrape config for node-exporter is part of the Prometheus config map. Once you deploy the node-exporter, you should see node-exporter targets and metrics in Prometheus.
- kubectl apply -f kubernetes-node-exporter/
Check all deployemnts
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.003.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.004.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.005.png)
**Prometheus dashboard**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.006.png)
**Node exporter dashboard**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.007.png)
**Grafana dashboard**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.008.png)
Username - admin
Password - admin
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.009.png)
Add data source- create new data source-select prometheus and add prometheus url-save
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.010.png)
Creating a Kubernetes dashboard from the Grafana template is pretty easy. There are many prebuilt Grafana templates available for Kubernetes. You can easily have prebuilt dashboards for ingress controllers, volumes, API servers, Prometheus metrics, and much more.
Link- <https://grafana.com/grafana/dashboards/?search=kubernetes>
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.011.png)
Search for kubernetes deployment statefulset daemonset metrics
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.012.png)
Copy id to clipboard - 8588
Then go back to garafan dashboard and click on dashboards - import
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.013.png)
Paste id here
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.014.png)
Select prometheus datasource which already added
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.015.png)
Now you can see dashboard for kubernetes deployment statefulset daemonset metrics
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.016.png)
Similarly for node exporter
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.017.png)
Copy id- 1860
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.018.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.019.png)
Similarly you can create custom dashboard- go to dashboard- new- add new panel
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.020.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.021.png)
Select datasource-prometheus
Metric ex- node\_cpu\_secods\_total – apply and save
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.022.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.023.png)
So all saved dashboard can be seen here
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.024.png)
**Alertmanager dashboard**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.025.png)
Now check prometheus configurations are up in dashboard, which defined in prometheus.yml
Go to - status-target
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.026.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.027.png)
Prometheus.rules
Goto -status-rules
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.028.png)
Run some metric to check output
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.029.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.030.png)
Check available metrics
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.031.png)
Check alerts
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.032.png)
alert firing
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.033.png)
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.034.png)
**Email alert**
It’s not recommended that you use your personal password for this, so you should create an App Password. To do that, go to Account Settings -> Security -> Signing in to Google -> App password (if you don’t see App password as an option, you probably haven’t set up 2-Step Verification and will need to do that first). Copy the newly-created password
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.035.png)
kind: ConfigMap
apiVersion: v1
metadata:
` `name: alertmanager-config
` `namespace: monitoring
data:
` `config.yml: |-
` `global:
` `templates:
` `- '/etc/alertmanager/\*.tmpl'
` `route:
` `receiver: alert-emailer
` `group\_by: ['alertname', 'priority']
` `group\_wait: 10s
` `group\_interval: 10s
` `repeat\_interval: 30m
` `receivers:
` `- name: alert-emailer
` `email\_configs:
` `- to: 321@gmail.com
` `from: 123@gmail.com
` `smarthost: smtp.gmail.com:587
` `auth\_username: 123@gmail.com
` `auth\_identity: 123@gmail.com
` `auth\_password: qxzjcdnfagashmso
` `send\_resolved: true
` `- name: slack\_demo
` `slack\_configs:
` `- api\_url: https://hooks.slack.com/services/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
` `channel: '#random'
` `send\_resolved: true
**Email Alert**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.036.png)
**Slack alert**
If you want to receive notifications via Slack, you should be part of a Slack workspace. To set up alerting in your Slack workspace, you’re going to need a Slack API URL. Go to Slack -> Administration -> Manage apps.
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.037.png)
In the Manage apps directory, search for Incoming WebHooks and add it to your Slack workspace.
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.038.png)
Next, specify in which channel you’d like to receive notifications from Alertmanager. (I’ve created #random channel). After you confirm and add Incoming WebHooks integration, webhook URL (which is your Slack API URL) is displayed. Copy it.
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.039.png)
Copy the Slack Webhook URL into the api\_url field and specify a Slack channel
kind: ConfigMap
apiVersion: v1
metadata:
` `name: alertmanager-config
` `namespace: monitoring
data:
` `config.yml: |-
` `global:
` `templates:
` `- '/etc/alertmanager/\*.tmpl'
` `route:
` `receiver: alert-emailer
` `group\_by: ['alertname', 'priority']
` `group\_wait: 10s
` `group\_interval: 10s
` `repeat\_interval: 30m
` `routes:
` `- receiver: slack\_demo
` `# Send severity=slack alerts to slack.
` `match:
` `severity: slack
` `group\_wait: 10s
` `repeat\_interval: 1m
` `receivers:
` `- name: alert-emailer
` `email\_configs:
` `- to: 321@gmail.com
` `from: 123@gmail.com
` `smarthost: smtp.gmail.com:587
` `auth\_username: 123@gmail.com
` `auth\_identity: 123@gmail.com
` `auth\_password: qxzjcdnfagashmso
` `send\_resolved: true
` `- name: slack\_demo
` `slack\_configs:
` `- api\_url: https://hooks.slack.com/services/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
` `channel: '#random'
` `send\_resolved: true
**Slack Alert**
![](Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.040.png)
README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.001.png

25.7 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.002.png

61.8 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.003.png

148 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.004.png

29.5 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.005.png

31.1 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.006.png

12.1 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.007.png

10.5 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.008.png

71.1 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.009.png

73.7 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.010.png

27.7 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.011.png

66 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.012.png

48.1 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.013.png

14.6 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.014.png

13.8 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.015.png

24.7 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.016.png

23.6 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.017.png

34 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.018.png

57.9 KiB

README1/Aspose.Words.055c6051-7ce2-4064-8427-e99e0e998c11.019.png

46.5 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment