{"id":434,"date":"2024-04-24T04:27:30","date_gmt":"2024-04-24T04:27:30","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=434"},"modified":"2024-04-24T04:27:30","modified_gmt":"2024-04-24T04:27:30","slug":"setup-prometheus-pushgateway-on-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=434","title":{"rendered":"How To Setup Prometheus Pushgateway On Kubernetes"},"content":{"rendered":"

In this article, we will look at an easy way to set up a Prometheus pushgateway on Kubernetes with a practical example of pushing metrics from an app.<\/p>\n

There are scenarios where Prometheus can’t pull the metrics from the endpoint, for example, batch jobs or short-lived jobs.<\/p>\n

In that kind of situation, we can use the service or application itself to push the metrics <\/strong>to Prometheus.<\/p>\n

As we know, Prometheus can only pull the metrics; however, there is an option called Pushgateway<\/strong> where applications can push the metrics. Prometheus<\/strong> will then pull the retained metrics from the Pushgateway.<\/p>\n

If you want to know more about the push gateway, I suggest you read the push gateways section in the detailed Prometheus Architecture<\/a>.<\/p>\n

Setting Up the Prometheus Pushgateway on Kubernetes<\/strong><\/h2>\n

I am implementing this into an existing monitoring setup so I will use the Namespace monitoring<\/strong> in each configuration file.<\/p>\n

If you want to set up the Prometheus on your cluster, click here.<\/a><\/p>\n

Step 1: Create a Pushgateway Deployment Manifest<\/h3>\n

We don’t need the ConfigMap for the Pushgateway, because most of our configuration will be on the target side and in the Prometheus configuration.<\/p>\n

Create a Deployment file pushgateway-deployment.yaml <\/strong>and add the following contents.<\/p>\n

apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: pushgateway\n  namespace: monitoring\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: pushgateway\n  template:\n    metadata:\n      labels:\n        app: pushgateway\n    spec:\n      containers:\n      - name: pushgateway\n        image: prom\/pushgateway:latest\n        ports:\n        - containerPort: 9091<\/code><\/pre>\n
Here, I am using the latest Pushgateway image, you can specify the particular version if you want and the default port number of the Pushgateway is 9091.<\/strong><\/p>\n
To apply this configuration on a cluster, use the following command.<\/p>\n
kubectl apply -f pushgateway-deployment.yaml<\/code><\/pre>\n
To view the list of deployments in the monitoring Namespace.<\/p>\n
kubectl get deployments -n monitoring -o wide<\/code><\/pre>\n
From the output, you can ensure the Deployment is properly placed.<\/p>\n
Step 2: Create a Pushgateway Service<\/h3>\n
Create a service configuration file push gateway-service.yaml<\/strong> and add the contents to it.<\/p>\n
apiVersion: v1\nkind: Service\nmetadata:\n  name: pushgateway\n  namespace: monitoring\nspec:\n  selector:\n    app: pushgateway\n  type: NodePort\n  ports:\n    - port: 9091\n      targetPort: 9091\n      nodePort: 31500<\/code><\/pre>\n
I want to access the Pushgateway dashboard over the Internet, for that, I am using the spec.type: NodePort<\/code>,<\/strong> and also I have specified the port number, which is 31500<\/strong>.<\/p>\n
To apply this configuration on Kubernetes, use the following command.<\/p>\n
kubectl apply -f pushgateway-service.yaml<\/code><\/pre>\n
To view the list of available Services in the monitoring Namespace.<\/p>\n
kubectl get svc -n monitoring -o wide<\/code><\/pre>\n
Once the configuration is completed, we can able to access the Pushgateway dashboard<\/strong> over the internet, for that, we need one of the instance public IPs and the node port number.<\/p>\n
\"prometheus<\/figure>\n
If the configuration is properly done, you will get this dashboard, now in the Metrics<\/strong> section, no data will be available, also Prometheus will not scrape the metrics.<\/p>\n
First, we need to add a job to the existing Prometheus configuration.<\/p>\n
Step 3: Add Pushgateway Job in Prometheus<\/h3>\n
Find the Prometheus ConfigMap<\/strong> Object or the YAML manifest and open it with a text editor.<\/p>\n
vim config-map.yaml <\/code><\/pre>\n
Your Prometheus ConfigMap<\/strong> file name would be different, so find and provide the correct name.<\/p>\n
Add the following job configuration under the scrape configs<\/strong> section.<\/p>\n
- job_name: \"pushgateway\"\n        honor_labels: true\n        static_configs:\n        - targets: [pushgateway.monitoring.svc:9091]<\/code><\/pre>\n
Ensure the target value, here, pushgateway<\/strong> is my service name and monitoring<\/strong> is the Namespace name, if you have given a different name, replace it with your values or provide the IP of the the Pushgateway.<\/p>\n
To apply the configuration and reflect the changes in the Prometheus, use the following command.<\/p>\n
kubectl apply -f config-map.yaml<\/code><\/pre>\n
Check your Prometheus dashboard target section to verify that the configuration is properly done.<\/p>\n
\"prometheus<\/figure>\n
If you can’t see the updates in the target, then perform the rollout restart to reflect the changes in the dashboard.<\/p>\n
kubectl rollout restart deployment prometheus-deployment -n monitoring<\/code><\/pre>\n
Your Prometheus deployment and the Namespace name would be different, so ensure before executing the command.<\/p>\n
This process might interrupt the scrapping for a few seconds, be aware of that otherwise your certain metrics would be skipped from scrapping.<\/p>\n
How to Test the Prometheus Pushgateway?<\/h2>\n
For testing, I am using a containerized Python script.<\/p>\n
Create a deployment file app-deployment.yaml<\/code> and add the contents.<\/p>\n
apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: python-app\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: pushgateway-python-app\n  template:\n    metadata:\n      labels:\n        app: pushgateway-python-app\n    spec:\n      containers:\n      - name: pushgateway-python-app\n        image: techiescamp\/pushgateway-python-app:latest\n        ports:\n        - containerPort: 8085<\/code><\/pre>\n
This testing image is available in the Dockerhub<\/strong> public repository so that you can use the same image for testing purposes.<\/p>\n
If you use the same image, the container will generate some random values and push those as metrics in the Pushgateway.<\/p>\n
To deploy this in the cluster, use the following command.<\/p>\n
kubectl apply -f app-deployment.yaml -n monitoring<\/code><\/pre>\n
To list the deployments, use the following command.<\/p>\n
kubectl get deployments -n monitoring<\/code><\/pre>\n
If the deployment is successfully done, we can see metrics in the Pushgateway<\/strong> dashboard.<\/p>\n
\"prometheus<\/figure>\n
Let’s try to make a query with this same metric random_metrics<\/strong> in the Prometheus, if we are getting values there, we can ensure the Prometheus is scraping the metrics from the Pushgateway<\/strong><\/p>\n
\"prometheus<\/figure>\n
Now, we know Prometheus <\/strong>is successfully scraping so <\/strong>visualization is also possible in Grafana.<\/strong><\/p>\n
\"prometheus<\/figure>\n
If you want to install Grafana on your Kubernetes cluster, please refer to this blog<\/a>.<\/p>\n
Why is Pushgateway important?<\/strong><\/h2>\n
We often set cronjobs in Kubernetes, these jobs will start in the defined time and exit when the job is completed. we can’t be sure that the Prometheus will scrape the metrics before it exits.<\/p>\n
With the help of Pushgateway, when the job executes, the metrics will be sent to the Pushgateway.<\/p>\n
For some jobs we can’t even know when it executes and when it ends, for example, we can say services, are activated when there is necessary, in that situation if we need metrics then the Pushgateway would be a good option.<\/p>\n
Conclusion<\/strong><\/h2>\n
Pushgateway is always a custom setup because you will decide what metrics you have to push from the target, so it is very useful for that kind of specific situation.<\/p>\n
I believe this setup gives you an overall idea of how to set the Pushgateway on your cluster.<\/p>\n
\n
Ngu\u1ed3n:<\/strong> How To Setup Prometheus Pushgateway On Kubernetes \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Source: https:\/\/devopscube.com\/setup-prometheus-pushgateway-on-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":435,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-434","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/434","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=434"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/434\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/435"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=434"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=434"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=434"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}