{"id":757,"date":"2024-04-04T15:32:45","date_gmt":"2024-04-04T15:32:45","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=757"},"modified":"2024-04-04T15:32:45","modified_gmt":"2024-04-04T15:32:45","slug":"blackbox-exporter-on-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=757","title":{"rendered":"Setup Blackbox Exporter on Kubernetes: A Step-by-Step Guide"},"content":{"rendered":"

In this blog, let’s explore how to set up Blackbox Exporter on Kubernetes. By the end, you’ll have a clear understanding of the process.<\/p>\n

If a requirement comes to monitor a server’s or application’s endpoint, then Blackbox Exporter<\/strong> will fulfill your requirements.<\/p>\n

This is also one of the Exporter that Prometheus<\/strong><\/a> has, and this will collect the probe metrics using the endpoint type such as HTTP, TCP, ICMP, etc.<\/p>\n

Why Blackbox Exporter is Important?<\/strong><\/h2>\n

Sometimes Kubernetes needs to access external resources such as databases and APIs, If those resources have any issue then Kubernetes<\/a> can’t able to access them, so we have to monitor their endpoint so that we can identify the issue earlier and fix it.<\/p>\n

Blackbox Exporter<\/a> not only helps to troubleshoot, but it is also useful to improve the performance of the Kubernetes with external resources.<\/p>\n

It can track the request and response time between the resources, based on the data, we can identify any kind of latency issue present and we can make adjustments to improve the performance.<\/p>\n

How to Setup Blackbox Exporter On Kubernetes?<\/strong><\/h2>\n

I have created a repo to store the configuration YAML files, if you are interested, use them for this setup.<\/p>\n

https:\/\/github.com\/arunlalp\/kubernetes-blackbox-exporter.git<\/code><\/pre>\n
This setup is a part of the Prometheus<\/strong> monitoring setup on our EKS cluster so I am using the existing namespace, which is monitoring<\/strong>.<\/p>\n
If you don’t have a dedicated namespace for your monitoring resources, create one before starting the setup.<\/p>\n
Step 1: Create a ConfigMap<\/h3>\n
Create a file blackbox-configmap.yaml<\/strong> and add the following contents.<\/p>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: blackbox-exporter-config\n  namespace: monitoring\ndata:\n  config.yml: |\n    modules:\n      http_endpoint:\n        prober: http\n        timeout: 5s\n        http:\n          valid_http_versions: [\"HTTP\/1.1\", \"HTTP\/2.0\"]\n          valid_status_codes: [200, 204]\n          no_follow_redirects: false\n          preferred_ip_protocol: \"ip4\"<\/code><\/pre>\n
Here, you can see the metadata<\/strong> section, I have mentioned the namespace monitoring<\/strong>, you can modify it with your namespace and replace the name if you want.<\/p>\n
Now, focus on the data<\/code> section, where you will add the modules. The Blackbox Exporter works based on modules.<\/p>\n
For this, I have mentioned only one HTTP<\/strong> endpoint-based module, but you can add multiple modules with different endpoint types.<\/p>\n
To apply this configmap<\/strong> to your cluster, use the following command.<\/p>\n
kubectl apply -f blackbox-configmap.yaml<\/code><\/pre>\n
If you want to see the list configmaps<\/strong> in the monitoring<\/strong> namespace.<\/p>\n
kubectl get configmaps -n monitoring<\/code><\/pre>\n
Step 2: Create the Blackbox Deployment<\/h3>\n
Create the configuration file blackbox-deployment.yaml<\/strong> and add the contents.<\/p>\n
---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: blackbox-exporter\n  namespace: monitoring\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: blackbox-exporter\n  template:\n    metadata:\n      labels:\n        app: blackbox-exporter\n    spec:\n      containers:\n      - name: blackbox-exporter\n        image: prom\/blackbox-exporter:latest\n        args:\n          - \"--config.file=\/etc\/blackbox-exporter\/config.yml\"\n        ports:\n        - containerPort: 9115\n        volumeMounts:\n        - name: config-volume\n          mountPath: \/etc\/blackbox-exporter\n      volumes:\n      - name: config-volume\n        configMap:\n          name: blackbox-exporter-config<\/code><\/pre>\n
Don’t forget to add the namespace name in the metadata<\/strong> section, in the spec section, you can see that I am using the latest Blackbox image prom\/blackbox-exporter:latest<\/strong>.<\/p>\n
you can give a specific version if you want, and the default port of the Blackbox is 9115, which I have mentioned as the value of containerPort.<\/strong><\/p>\n
To apply this configuration, use the following command.<\/p>\n
kubectl apply -f blackbox-deployment.yaml<\/code><\/pre>\n
To view the list of deployments<\/strong> in the monitoring<\/strong> namespace<\/p>\n
kubectl get deployments -n monitoring -o wide<\/code><\/pre>\n
Step 3: Create a Service for Blackbox Exporter<\/h3>\n
Create a configuration file blackbox-service.yaml<\/strong> and add the following contents.<\/p>\n
---\napiVersion: v1\nkind: Service\nmetadata:\n  name: blackbox-exporter\n  namespace: monitoring\nspec:\n  selector:\n    app: blackbox-exporter\n  type: NodePort\n  ports:\n    - port: 9115\n      targetPort: 9115\n      nodePort: 30500\n\n<\/code><\/pre>\n
In this configuration also you have to provide the namespace name and the main thing you have to consider is the spec<\/strong> section type<\/code> parameter.<\/p>\n
I am using NodePort <\/strong>type so that I can access the Blackbox Exporter<\/strong> over the Internet, you can use other types based on your requirements.<\/p>\n
The nodePort<\/strong> I have mentioned here is 30500<\/strong>, otherwise, it will choose some random port from between <\/strong>the 30000-32767<\/strong> range.<\/p>\n
To apply this configuration, use the following command.<\/p>\n
kubectl apply -f blackbox-service.yaml<\/code><\/pre>\n
To view the list of services in the monitoring<\/strong> namespace.<\/p>\n
kubectl get svc -n monitoring -o wide<\/code><\/pre>\n
Now, you can able to access the Blackbox Exporter<\/strong> dashboard over the internet, for that you need the public IP of one of your nodes and the port number 30500<\/strong>.<\/p>\n
The output you will get is<\/p>\n
\"blackbox<\/figure>\n
This time you can’t see any results in this dashboard because we haven’t mentioned the target anywhere.<\/p>\n
We can add the targets, only in the Prometheus<\/strong> configuration file, so for that, we have to do some modifications in the Prometheus<\/strong><\/a> configuration file.<\/p>\n
Step 4: Add Blackbox Job in Prometheus<\/h3>\n
You can directly make modifications to the Prometheus configmap<\/strong> file, for that first, you need to identify the configmap<\/strong> name.<\/p>\n
To list them configmaps<\/strong> in the monitoring<\/strong> namespace, use the following command.<\/p>\n
kubectl get configmaps -n monitoring<\/code><\/pre>\n
Once find the configmap name, use the below command to edit it.<\/p>\n
kubectl edit configmap prometheus-server-conf -n monitoring<\/code><\/pre>\n
There is an alternative method to edit the ConfigMap, for that, you need to find the ConfigMap YAML manifest.<\/p>\n
If you find the file name, open it with any text editor utility. In my case, the Prometheus configmap file name is config-map.yaml<\/strong>.<\/p>\n
vim config-map.yaml<\/code><\/pre>\n
Add this content inside this file, under the scrape_configs<\/strong> section.<\/p>\n
- job_name: 'blackbox-exporter-metrics'\n        metrics_path: \/probe\n        params:\n          module: [http_endpoint]\n        static_configs:\n          - targets:\n            - https:\/\/www.google.com\n            - devopscube.com\n            - https:\/\/prometheus.io\n        relabel_configs:\n         - source_labels: [__address__]\n           target_label: __param_target\n         - source_labels: [__param_target]\n           target_label: instance\n         - target_label: __address__\n           replacement: \"blackbox-exporter.monitoring.svc:9115\"<\/code><\/pre>\n
In the params<\/strong> section, http_endpoint<\/strong> is the module name, if you don’t remember, verify your Blackbox configmap yaml file.<\/p>\n
Under the static_configs<\/strong>, targets should be given, for testing purposes.<\/p>\n
I have given three websites so that the Blackbox Exporter will monitor these website endpoints and collect the metrics based on their health and availability.<\/p>\n
The replacement of the value indicates that from where the Prometheus has to scrape the metrics.<\/p>\n
Once the modifications are completed, you can deploy this configuration in Prometheus<\/strong>.<\/p>\n
kubectl apply -f config-map.yaml<\/code><\/pre>\n
Sometimes it would take a few minutes to reflect the changes in the Prometheus<\/strong> dashboard.<\/p>\n
\"prometheus<\/figure>\n
If you are not able to find the changes in the Prometheus<\/strong> dashboard, then you can perform the rollout restart deployment.<\/p>\n
To restart the rollout, use the following command.<\/p>\n
kubectl rollout restart deployment prometheus-deployment -n monitoring<\/code><\/pre>\n
Replace the prometheus-deployment<\/strong> with your deployment name.<\/p>\n
You have to really aware when you perform rollout restart, because until the restart completed, the Prometheus won’t scrape the metrics.<\/p><\/blockquote>\n
Step 5: Make Queries from Prometheus<\/h3>\n
Once the configuration is completed, Prometheus<\/a> will scrape the metrics from the Blackbox Exporter based on the scrape interval.<\/p>\n
I am using the probe_http_status_code<\/strong> metric to see the results.<\/p>\n
Prometheus output<\/strong><\/p>\n
\"prometheus<\/figure>\n
Step 6: Visualize the Metrics using the Grafana<\/h3>\n
This time I am not going to make a query using the Grafana<\/strong>. Instead of that, I am using a dashboard template to visualize the important probe metrics of all the endpoints.<\/p>\n
Grafana output<\/strong><\/p>\n
\"grafana<\/figure>\n
Conclusion<\/strong><\/h2>\n
I have already done this test setup using Linux virtual machines, this time I implemented the setup in Kubernetes and I didn’t want to make this test setup more complicated so I just used only one module to probe the metrics from the endpoint.<\/p>\n
When you test or deploy this setup, adjust the values for your needs and try to use other modules and endpoints to know the difference and use cases.<\/p>\n
\n
Ngu\u1ed3n:<\/strong> Setup Blackbox Exporter on Kubernetes: A Step-by-Step Guide \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Source: https:\/\/devopscube.com\/blackbox-exporter-on-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":758,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-757","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/757","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=757"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/757\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/758"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=757"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=757"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=757"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}