About Prometheus<\/h2>\n
Prometheus<\/a> is a high-scalable open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform<\/a>. <\/p>\n Also, In the observability<\/a> space, it is gaining huge popularity as it helps with metrics and alerts.<\/p>\n Explaining Prometheus is out of the scope of this article. If you want to know more about Prometheus, You can watch all the Prometheus-related videos from here<\/a>.<\/p>\n However, there are a few key concepts I would like to list for your reference.<\/p>\n Here is the high-level architecture of Prometheus.<\/strong> <\/p>\n The Kubernetes Prometheus monitoring stack has the following components.<\/p>\n In a nutshell, the following image depicts the high-level Prometheus kubernetes architecture<\/strong> that we are going to build. We have separate blogs for each component setup.<\/p>\n I assume that you have a Kubernetes cluster up and running with kubectl setup on your workstation.<\/p>\n The latest Prometheus is available as a docker image<\/a> in its official docker hub account. We will use that image for the setup.<\/p>\n Connect to your Kubernetes cluster and make sure you have admin privileges to create cluster roles.<\/p>\n All the configuration files I mentioned in this guide are hosted on Github<\/strong><\/a>. You can clone the repo using the following command.<\/p>\n Please don’t hesitate to contribute to the repo for adding features.<\/p>\n You can use the GitHub repo config files or create the files on the go for a better understanding, as mentioned in the steps.<\/p>\n Let’s get started with the setup.<\/p>\n First, we will create a Kubernetes namespace for all our monitoring components. If you don’t create a dedicated namespace, all the Prometheus kubernetes deployment objects get deployed on the default namespace.<\/p>\n Execute the following command to create a new namespace named monitoring<\/strong>.<\/p>\n Prometheus uses Kubernetes APIs to read all the available metrics from Nodes, Pods, Deployments, etc. For this reason, we need to create an RBAC policy with Step 1:<\/strong> Create a file named Step 2:<\/strong> Create the role using the following command.<\/p>\n All configurations for Prometheus are part of By externalizing Prometheus configs to a Kubernetes config map, you don’t have to build the Prometheus image whenever you need to add or remove a configuration. You just need to update the Configmap and restart the Prometheus pods to apply the new configuration.<\/p>\n The config map with all the Prometheus scrape config<\/a> and alerting rules gets mounted to the Prometheus container in Lets create the configmap.<\/p>\n Execute the following command to create the config map in Kubernetes.<\/p>\n It creates two files inside the container.<\/p>\n The Step 1:<\/strong> Create a file named Step 2:<\/strong> Create a deployment on monitoring namespace using the above file.<\/p>\n Step 3:<\/strong> You can check the created deployment using the following command.<\/p>\n You can also get details from the kubernetes dashboard as shown below.<\/p>\n You can view the deployed Prometheus dashboard in three different ways.<\/p>\n Let’s have a look at all three options.<\/p>\n Using kubectl port forwarding, you can access a pod from your local workstation using a selected port on your Step 1:<\/strong> First, get the Prometheus pod name.<\/p>\n The output will look like the following.<\/p>\n Step 2:<\/strong> Execute the following command with your pod name to access Prometheus from localhost port 8080.<\/p>\n Step 3:<\/strong> Now, if you access To access the Prometheus dashboard over a Step 1:<\/strong> Create a file named The Step 2:<\/strong> Create the service using the following command.<\/p>\n Step 3:<\/strong> Once created, you can access the Prometheus dashboard using any of the Kubernetes node’s IP on port Step 4:<\/strong> Now, if you browse to The kube-state-metrics down is expected and I’ll discuss it shortly.<\/p>\n Step 5:<\/strong> You can head over to the homepage and select the metrics you need from the drop-down, and get the graph for the time range you mention. An example graph for If you have an existing ingress controller setup<\/a>, you can create an ingress object to route the Prometheus DNS to the Prometheus backend service.<\/p>\n Also, you can add SSL for Prometheus in the ingress layer. You can refer to the Kubernetes ingress TLS\/SSL Certificate guide<\/a> for more details.<\/p>\n Here is a sample ingress object. Please refer to this GitHub link <\/a>for a sample ingress object with SSL<\/p>\n Kube state metrics<\/strong> service will provide many metrics that are not available by default. <\/p>\n Please make sure you deploy Kube state metrics to monitor all your Kubernetes API objects like Please follow this article to setup Kube state metrics on Kubernetes ==> How To Setup Kube State Metrics on Kubernetes<\/a><\/p>\n Alertmanager handles all the alerting mechanisms for Prometheus metrics. There are many integrations available to receive alerts from the Alertmanager (Slack, email, API endpoints, etc)<\/p>\n I have covered the Alert Manager setup in a separate article. Please follow ==> Alert Manager Setup on Kubernetes<\/a><\/p>\n Using Grafana, you can create dashboards from Prometheus metrics to monitor the Kubernetes cluster.<\/p>\n The best part is, you don’t have to write all the PromQL queries for the dashboards.<\/p>\n There are many community dashboard templates available for Kubernetes. You can import it and modify it as per your needs. I have covered it in the article.<\/p>\n Please follow this article for the Grafana setup ==> How To Setup Grafana On Kubernetes<\/a><\/p>\n Node Exporter will provide all the Linux system-level metrics of all Kubernetes nodes.<\/p>\n I have written a separate step-by-step guide on node-exporter daemonset deployment. Please follow Setting up Node Exporter on Kubernetes<\/a><\/p>\n The scrape config for node-exporter is part of the Prometheus config map. Once you deploy the node-exporter, you should see node-exporter targets and metrics in Prometheus.<\/p>\n For the production Prometheus setup<\/strong>, there are more configurations and parameters that need to be considered for scaling, high availability, and storage. It all depends on your environment and data volume.<\/p>\n For example, Prometheus Operator<\/a> project makes it easy to automate Prometheus setup and its configurations.<\/p>\n If you have multiple production clusters, you can use the CNCF project Thanos<\/a> to aggregate metrics from multiple Kubernetes Prometheus sources.<\/p>\n Thanos provides features like multi-tenancy, horizontal scalability, and disaster recovery, making it possible to operate Prometheus at scale with high availability.<\/p>\n With Thanos, you can query data from multiple Prometheus instances running in different kubernetes clusters<\/a> in a single place, making it easier to aggregate metrics and run complex queries.<\/p>\n Additionally, Thanos can store Prometheus data in an object storage backend, such as Amazon S3 or Google Cloud Storage, which provides an efficient and cost-effective way to retain long-term metric data.<\/p>\n In this comprehensive Prometheus kubernetes tutorial<\/strong>, I have covered the setup of important monitoring components to understand Kubernetes monitoring.<\/p>\n Also, If you are learning Kubernetes, you can check out my Kubernetes beginner tutorials<\/a> where I have 60+ comprehensive guides.<\/p>\n Let me know what you think about the Prometheus monitoring setup by leaving a comment.<\/p>\n You can also use this setup to prepare for the Prometheus Certified Associate Certification.<\/p>\n\n
Pushgateway<\/code><\/a> for use cases where Prometheus cannot Scrape the metrics. One such example is collecting custom metrics from short-lived kubernetes jobs & Cronjobs<\/a><\/li>\n\/metrics<\/code> endpoint. Prometheus uses this endpoint to pull the metrics in regular intervals.<\/li>\nPromQL<\/code><\/a>, a very flexible query language that can be used to query the metrics in the Prometheus dashboard. Also, the PromQL query will be used by Prometheus UI and Grafana to visualize metrics.<\/a><\/li>\nPrometheus Architecture<\/h2>\n
![\"Prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/prometheus-architecture-3.gif\")
\n
![\"Prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/kubernetes-1.png\")
<\/figure>\nPrometheus Monitoring Setup on Kubernetes<\/h2>\n
Connect to the Kubernetes Cluster<\/h2>\n
ACCOUNT=$(gcloud info --format='value(config.account)')\nkubectl create clusterrolebinding owner-cluster-admin-binding \\\n --clusterrole cluster-admin \\\n --user $ACCOUNT<\/code><\/pre>\nPrometheus Kubernetes Manifest Files<\/h2>\n
git clone https:\/\/github.com\/techiescamp\/kubernetes-prometheus<\/code><\/pre>\nCreate a Namespace & ClusterRole<\/h2>\n
kubectl create namespace monitoring<\/code><\/pre>\nread access<\/code> to required API groups and bind the policy to the monitoring<\/code> namespace.<\/p>\nclusterRole.yaml<\/code> and copy the following RBAC role.<\/p>\nget<\/code>, list<\/code>, and watch<\/code> permissions to nodes, services endpoints, pods, and ingresses. The role binding is bound to the monitoring namespace. If you have any use case to retrieve metrics from any other object, you need to add that in this cluster role.<\/div>\n<\/div>\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRole\nmetadata:\n name: prometheus\nrules:\n- apiGroups: [\"\"]\n resources:\n - nodes\n - nodes\/proxy\n - services\n - endpoints\n - pods\n verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups:\n - extensions\n resources:\n - ingresses\n verbs: [\"get\", \"list\", \"watch\"]\n- nonResourceURLs: [\"\/metrics\"]\n verbs: [\"get\"]\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n name: prometheus\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: prometheus\nsubjects:\n- kind: ServiceAccount\n name: default\n namespace: monitoring<\/code><\/pre>\nkubectl create -f clusterRole.yaml<\/code><\/pre>\nCreate a Config Map To Externalize Prometheus Configurations<\/h2>\n
prometheus.yaml<\/code> file and all the alert rules for Alertmanager are configured in prometheus.rules<\/code>.<\/p>\n\n
prometheus.yaml<\/code>: This is the main Prometheus configuration which holds all the scrape configs, service discovery details, storage locations, data retention configs, etc)<\/li>\nprometheus.rules<\/code>: This file contains all the Prometheus alerting rules<\/li>\n<\/ol>\n\/etc\/prometheus<\/code> location as prometheus.yaml<\/code> and prometheus.rules<\/code> files.<\/p>\nkubectl create -f https:\/\/raw.githubusercontent.com\/bibinwilson\/kubernetes-prometheus\/master\/config-map.yaml<\/code><\/pre>\njob<\/code>.<\/div>\n<\/div>\nprometheus.yaml<\/code> contains all the configurations to discover pods and services running in the Kubernetes cluster dynamically. We have the following scrape jobs<\/a> in our Prometheus scrape configuration.<\/p>\n\n
kubernetes-apiservers<\/code>: It gets all the metrics from the API servers.<\/li>\nkubernetes-nodes<\/code>: It collects all the kubernetes node metrics.<\/li>\nkubernetes-pods<\/code>: All the pod metrics get discovered if the pod metadata is annotated with prometheus.io\/scrape<\/code> and prometheus.io\/port<\/code> annotations.<\/li>\nkubernetes-cadvisor<\/code>: Collects all cAdvisor metrics.<\/li>\nkubernetes-service-endpoints<\/code>: All the Service endpoints are scrapped if the service metadata is annotated with prometheus.io\/scrape and prometheus.io\/port annotations. It can be used for black-box monitoring.<\/li>\n<\/ol>\nprometheus.rules<\/code> contains all the alert rules for sending alerts to the Alertmanager.<\/p>\nCreate a Prometheus Deployment<\/h2>\n
prometheus-deployment.yaml<\/code> and copy the following contents onto the file. In this configuration, we are mounting the Prometheus config map as a file inside \/etc\/prometheus<\/code> as explained in the previous section.<\/p>\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: prometheus-deployment\n namespace: monitoring\n labels:\n app: prometheus-server\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: prometheus-server\n template:\n metadata:\n labels:\n app: prometheus-server\n spec:\n containers:\n - name: prometheus\n image: prom\/prometheus\n args:\n - \"--storage.tsdb.retention.time=12h\"\n - \"--config.file=\/etc\/prometheus\/prometheus.yml\"\n - \"--storage.tsdb.path=\/prometheus\/\"\n ports:\n - containerPort: 9090\n resources:\n requests:\n cpu: 500m\n memory: 500M\n limits:\n cpu: 1\n memory: 1Gi\n volumeMounts:\n - name: prometheus-config-volume\n mountPath: \/etc\/prometheus\/\n - name: prometheus-storage-volume\n mountPath: \/prometheus\/\n volumes:\n - name: prometheus-config-volume\n configMap:\n defaultMode: 420\n name: prometheus-server-conf\n \n - name: prometheus-storage-volume\n emptyDir: {}<\/code><\/pre>\nkubectl create -f prometheus-deployment.yaml <\/code><\/pre>\nkubectl get deployments --namespace=monitoring<\/code><\/pre>\n![\"prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/screen-shot-2017-10-11-at-11-26-15-am-3.jpg\")
<\/figure>\nConnecting To Prometheus Dashboard<\/h2>\n
\n
Method 1: Using Kubectl port forwarding<\/h3>\n
localhost<\/code>. This method is primarily used for debugging purposes.<\/p>\nkubectl get pods --namespace=monitoring<\/code><\/pre>\n\u279c kubectl get pods --namespace=monitoring\nNAME READY STATUS RESTARTS AGE\nprometheus-monitoring-3331088907-hm5n1 1\/1 Running 0 5m<\/code><\/pre>\nkubectl port-forward prometheus-monitoring-3331088907-hm5n1 8080:9090 -n monitoring<\/code><\/pre>\nhttp:\/\/localhost:8080<\/code> on your browser, you will get the Prometheus home page.<\/p>\nMethod 2: Exposing Prometheus as a Service [NodePort & LoadBalancer]<\/h3>\n
IP<\/code> or a DNS<\/code> name, you need to expose it as a Kubernetes service.<\/p>\nprometheus-service.yaml<\/code> and copy the following contents. We will expose Prometheus on all kubernetes node IP’s on port 30000<\/code>.<\/p>\napiVersion: v1\nkind: Service\nmetadata:\n name: prometheus-service\n namespace: monitoring\n annotations:\n prometheus.io\/scrape: 'true'\n prometheus.io\/port: '9090'\nspec:\n selector: \n app: prometheus-server\n type: NodePort \n ports:\n - port: 8080\n targetPort: 9090 \n nodePort: 30000<\/code><\/pre>\nannotations<\/code> in the above service YAML<\/code> makes sure that the service endpoint is scrapped by Prometheus. The prometheus.io\/port<\/code> should always be the target port mentioned in service YAML<\/p>\nkubectl create -f prometheus-service.yaml --namespace=monitoring<\/code><\/pre>\n30000<\/code>. If you are on the cloud, make sure you have the right firewall rules to access port 30000<\/code> from your workstation.<\/p>\n![\"Prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/screen-shot-2017-10-11-at-12-15-57-pm-3.jpg\")
<\/figure>\nstatus --> Targets<\/code>, you will see all the Kubernetes endpoints connected to Prometheus automatically using service discovery as shown below.<\/p>\n![\"Prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-14-43.png\")
<\/figure>\ncontainer_cpu_usage_seconds_total<\/code> is shown below.<\/p>\n![\"Prometheus](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-13-30.png\")
<\/figure>\nMethod 3: Exposing Prometheus Using Ingress<\/h3>\n
apiVersion: extensions\/v1beta1\nkind: Ingress\nmetadata:\n name: prometheus-ui\n namespace: monitoring\n annotations:\n kubernetes.io\/ingress.class: nginx\nspec:\n rules:\n # Use the host you used in your kubernetes Ingress Configurations\n - host: prometheus.example.com\n http:\n paths:\n - backend:\n serviceName: prometheus-service\n servicePort: 8080<\/code><\/pre>\nSetting Up Kube State Metrics<\/h2>\n
deployments<\/code>, pods<\/code>, jobs<\/code>, cronjobs<\/code> etc.<\/p>\nSetting Up Alertmanager<\/h2>\n
Setting Up Grafana<\/h2>\n
Setting Up Node Exporter<\/h2>\n
Prometheus Production Setup Considerations<\/h2>\n
Conclusion<\/h2>\n
\n