{"id":753,"date":"2024-04-04T15:28:56","date_gmt":"2024-04-04T15:28:56","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=753"},"modified":"2024-04-04T15:28:56","modified_gmt":"2024-04-04T15:28:56","slug":"set-up-blackbox-exporter-on-vm","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=753","title":{"rendered":"How To Set Up Blackbox Exporter On VM – Ultimate Guide"},"content":{"rendered":"

In this blog, let’s explore how to set up Black Box Exporter on VM(virtual machine). By the end, you’ll have a clear understanding of the process.<\/p>\n

The working nature of the Black Box exporter<\/a> is a little different than other exporters, if you require to monitor the endpoint of a server then the Blackbox exporter is useful.<\/p>\n

This exporter will probe the metrics based on monitoring the health of the server and whether the endpoint is available or not, using protocols such as HTTP, HTTPS, ICMP, TCP, gRPC, and DNS.<\/p>\n

Blackbox Exporter Workflow<\/strong><\/h2>\n
\"set<\/figure>\n

In this setup, we use two separate servers for Prometheus<\/a> and Blackbox Exporter, but if you want, you can install both utilities on the same server.<\/p>\n

After installing the Blackbox Exporter, we have to configure it for probing the metrics from the target, the configuration is a YAML formatted file, which contains modules, and modules are responsible for choosing the endpoint type.<\/p>\n

Blackbox itself doesn’t collect the information from the target, so for that we need Prometheus.<\/a><\/p>\n

We have to modify the configuration in Prometheus about the scraping interval, and target information. Also, we have to link the Blackbox Exporter to the Prometheus.<\/p>\n

Based on the scrape interval, Prometheus tries to scrape the metrics from the Blackbox Exporter, once that response reaches the Blackbox, Blackbox will be triggered to collect the information from the target.<\/p>\n

Blackbox Exporter collects the information and stores it in \/probes<\/strong> the directory, from there. Prometheus pulls them and stores them in the Time Series Data Base (TSDB).<\/strong><\/p>\n

Setting Up Prometheus Blackbox Exporter on Virtual Machine(VM)<\/strong><\/h2>\n

I am using two servers in AWS<\/a> for this tutorial.<\/p>\n

Server 1<\/strong>: The instance type is t2.medium<\/strong> and the utilities in this server are Prometheus<\/strong> and Grafana<\/strong>.<\/code><\/p>\n

Server 2<\/strong>: This instance type is t2.micro<\/strong> and the utilities are Prometheus Blackbox Exporter<\/strong>.<\/p>\n

I assume you have the server 1 setup already, so I am skipping that installation part, and let’s begin to install Blackbox Exporter on the server.<\/p>\n

Download Blackbox Exporter binaries from the official repository<\/a>. I am using the latest version v0.24.0<\/strong> for this setup.<\/p>\n

https:\/\/github.com\/prometheus\/blackbox_exporter\/releases\/download\/v0.24.0\/blackbox_exporter-0.24.0.linux-amd64.tar.gz<\/code><\/pre>\n
Unzip the files.<\/p>\n
tar -xvf blackbox_exporter-0.24.0.linux-amd64.tar.gz<\/code><\/pre>\n
This will contain the Blackbox executable file and the configuration file.<\/p>\n
Open the directory<\/p>\n
cd blackbox_exporter-0.24.0.linux-amd64<\/code><\/pre>\n
Create a user for Blackbox with no shell access.<\/p>\n
sudo useradd -rs \/bin\/false blackbox<\/code><\/pre>\n
Create a directory in the \/etc<\/strong> directory to store the configuration file.<\/p>\n
sudo mkdir -p \/etc\/blackbox<\/code><\/pre>\n
Move the configuration blackbox.yml<\/code> file to the directory<\/p>\n
sudo mv blackbox.yml \/etc\/blackbox<\/code><\/pre>\n
Move the executable file blackbox_exporter<\/strong> to the \/usr\/local\/bin<\/strong> directory<\/p>\n
sudo mv blackbox_exporter \/usr\/local\/bin<\/code><\/pre>\n
Change the ownership to the executable file and configuration file to the Blackbox<\/strong> user.<\/p>\n
sudo chown blackbox:blackbox \/usr\/local\/bin\/blackbox_exporter\nsudo chown -R blackbox:blackbox \/etc\/blackbox\/*<\/code><\/pre>\n
Create a service file for the Blackbox Exporter blackbox.service<\/strong><\/p>\n
sudo cat <<EOT > \/lib\/systemd\/system\/blackbox.service\n[Unit]\nDescription=Blackbox Exporter Service\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=simple\nUser=blackbox\nGroup=blackbox\nExecStart=\/usr\/local\/bin\/blackbox_exporter \\\n  --config.file=\/etc\/blackbox\/blackbox.yml \\\n  --web.listen-address=\":9115\"\n\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\nEOT\n<\/code><\/pre>\n
Enable and start the blackbox.service<\/strong><\/p>\n
sudo systemctl enable blackbox.service\nsudo systemctl start blackbox.service<\/code><\/pre>\n
To check the status of the service, use the following command.<\/p>\n
sudo systemctl status blackbox.service<\/code><\/pre>\n
The port number of the Blackbox Exporter is 9115<\/strong>, by default, the external probe metrics<\/strong> will be saved in the \/probe<\/code> directory and the internal metrics<\/strong> will be stored in \/metrics<\/strong> the directory.<\/p>\n
If you want to check it from inside the instance, use the following method.<\/p>\n
curl http:\/\/localhost:9115\/metrics<\/code><\/pre>\n
If you want to access it over the internet, in the browser, use your instance public IP with the port number <Public IP>:9115<\/strong>. Then you will get this output.<\/p>\n
\"prometheus<\/figure>\n
Based on modules, the probe data will be stored here.<\/p>\n
Prometheus Blackbox Exporter Probe Module<\/strong><\/h2>\n
We have to create modules<\/strong> in Blackbox, then we will get the probe metrics<\/strong>. Fundamentally, we have to know which type of endpoint<\/strong> we are going to monitor because the module should be related to the endpoint.<\/p>\n
A simple example module for an HTTP endpoint is<\/p>\n
http_endpoint:\n  prober: http\n  timeout: 10s\n  http:\n    valid_http_versions: [\"HTTP\/1.1\", \"HTTP\/2.0\"]\n    valid_status_codes: [200, 204]\n    no_follow_redirects: false\n    preferred_ip_protocol: \"ip4\"<\/code><\/pre>\n
This module will be configured in the Blackbox Exporter configuration file, which is \/etc\/blackbox\/blackbox.yml.<\/strong><\/p>\n
The collected probe metrics by this particular module are this<\/p>\n
\"probe<\/figure>\n
We can further filter and visualize these metrics using Prometheus<\/a> and Grafana utilities.<\/p>\n
Prometheus Blackbox Exporter Endpoint<\/strong><\/h2>\n
The Blackbox Exporter probes the metrics using responses from the endpoint, protocol defining the endpoint types and the supporting protocols,<\/p>\n
    \n
  1. HTTP<\/li>\n
  2. HTTPS<\/li>\n
  3. ICMP<\/li>\n
  4. TCP<\/li>\n
  5. DNS<\/li>\n
  6. gRPC<\/li>\n<\/ol>\n
    We should choose one of the protocols to create a module, for example, if I am taking the DNS<\/strong> protocol to create a module, I could get the probe metrics like DNS query failure counts, query duration, query count, response size, etc<\/p>\n
    Prometheus Blackbox Exporter Real-World Examples<\/strong><\/h2>\n
    Create a module using HTTP(S)<\/code> protocol<\/h3>\n
    HTTP(S) module can track the SSL\/TLS certificate validation and expiration, so I am creating this module to know that the target web server has a valid SSL\/TLS certificate.<\/p>\n
    Add this configuration to your Blackbox Exporter configuration file \/etc\/blackbox\/blackbox.yml<\/strong>.<\/p>\n
    modules:\n  https_endpoint:\n    prober: http\n    timeout: 15s\n    http:\n      method: GET\n      valid_http_versions:\n        - HTTP\/1.1\n        - HTTP\/2.0\n      fail_if_not_ssl: true\n      no_follow_redirects: false\n      ip_protocol_fallback: false\n      preferred_ip_protocol: ip4\n<\/code><\/pre>\n
    Here, you can see that the prober is HTTP<\/strong>, which indicates this module is meant to be an HTTP endpoint, also I am passing another argument fail_if_not_ssl: true<\/strong> so that if the web server doesn’t have an SSL\/TLS certificate, the status would fail.<\/p>\n
    Restart and check the status of the Blackbox Exporter.<\/p>\n
    sudo systemctl restart blackbox.service\nsudo systemctl status blackbox.service<\/code><\/pre>\n
    Update the Prometheus configuration file<\/h3>\n
    Open the Prometheus<\/a> configuration file \/etc\/prometheus\/prometheus.yml<\/strong>, and append the Blackbox configuration with your existing Prometheus configuration.<\/p>\n
    In the global section, I have given the scrape_interval<\/strong> time as 5s<\/strong>, which means Prometheus<\/a> will pull the metrics from the exporters every 5 seconds. you can modify with your required value.<\/p>\n
    Under the scrape_configs<\/strong> section, we have to create a new job for Blackbox Exporter and have to give the metrics_path<\/code>, we already know that the probe metrics will be stored in the\/probe <\/strong>directory.<\/p>\n
    I am giving the module value https_endpoint, which<\/strong> is my module name, and modifying the replacement value with your Blackbox Exporter servers private (if same network) or public IP with port number.<\/p>\n
    global:\n  scrape_interval: 5s\n  evaluation_interval: 5s\n\nscrape_configs:\n  - job_name: \"blackbox\"\n    metrics_path: \/probe\n    params:\n      module: [https_endpoint]\n    static_configs:\n      - targets:\n        - 35.80.157.25\n        - devopscube.com\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: __param_target\n      - source_labels: [__param_target]\n        target_label: instance\n      - target_label: __address__\n        replacement: 172.31.19.73:9115<\/code><\/pre>\n
    I have given two targets<\/strong> in this configuration file, which are 35.80.157.25<\/code><\/strong> and devopscube.com<\/strong>. The first one has a Nginx web server and doesn’t have an SSL\/TLS certificate, but the second one has a valid certificate.<\/p>\n
    \"ssl<\/figure>\n
    Let’s see if our setup identifies that, before that, we have to restart the Blackbox service and ensure the configurations are properly done.<\/p>\n
    sudo systemctl restart blackbox.service\nsudo systemctl status blackbox.service<\/code><\/pre>\n
    Check the Blackbox Exporter<\/strong> directly. For that open any browser, and paste your Blackbox Exporter public IP and the port number.<\/p>\n
    \"prometheus<\/figure>\n
    Here it shows that the result of the devopscube.com<\/strong> is successful and the 35.80.157.24<\/strong> is failed.<\/p>\n
    If you click the logs, you can view all the probe metrics, which is related to the target.<\/p>\n
    \"prometheus<\/figure>\n
    Now I am picking one of the metrics from that to see the results in Prometheus.<\/p>\n
    To see the presence of the SSL\/TLS certificate, we can use probe_http_ssl<\/strong> metrics.<\/p>\n
    probe_http_ssl<\/code><\/pre>\n
    Prometheus output<\/strong><\/p>\n
    \"prometheus<\/figure>\n
    Grafana output<\/strong><\/p>\n
    \"grafana<\/figure>\n
    Create a module using TCP<\/code> protocol<\/h2>\n
    The main use case of the TCP<\/code> the based module is used to identify whether a user can reach the target over TCP or not also we can check some SSL-related metrics.<\/p>\n
    Add this to the Blackbox configuration file \/etc\/blackbox\/blackbox.yml<\/strong><\/p>\n
    modules:\n  tcp_module:\n    prober: tcp\n    timeout: 10s\n    tcp:\n      preferred_ip_protocol: \"ip4\"\n      tls: true\n      tls_config:\n        insecure_skip_verify: false\n<\/code><\/pre>\n
    tcp_module<\/code> is the module name, you can modify it if you want, and the prober<\/code> value should be tcp<\/strong>.<\/p>\n
    Don’t forget to restart and view the status of the black-box service blackbox.service<\/strong>, this will ensure the configurations are properly done.<\/p>\n
    Update the Prometheus configuration file \/etc\/prometheus\/prometheus.yml.<\/strong> Add another job or replace it with the existing job if you want.<\/p>\n
    scrape_configs:\n  - job_name: \"blackbox-tcp\"\n    metrics_path: \/probe\n    params:\n      module: [tcp_module]\n    static_configs:\n      - targets:\n        - 35.80.157.25:443\n        - devopscube.com:443\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: __param_target\n      - source_labels: [__param_target]\n        target_label: instance\n      - target_label: __address__\n        replacement: 172.31.19.73:9115<\/code><\/pre>\n
    After updating the configuration file, restart and check the status of the Prometheus service prometheus.service<\/strong><\/p>\n
    Providing target with the port number is necessary for the TCP module.<\/p><\/blockquote>\n
    The Blackbox Exporter output is<\/p>\n
    \"prometheus<\/figure>\n
    Here, the first server doesn’t have the Transport Layer Security, so it fails and we can view the probe metrics of the second server.<\/p>\n
    This tcp_module <\/strong>module collects the probe metrics.<\/p>\n
    \"blackbox<\/figure>\n
    I want to know the SSL\/TLS certificate expiry date of the server, for that, I am using probe_ssl_last_chain_expiry_timestamp_seconds<\/strong> metrics.<\/p>\n
    probe_ssl_last_chain_expiry_timestamp_seconds<\/code><\/pre>\n
    Prometheus output<\/strong><\/p>\n
    \"prometheus<\/figure>\n
    Let’s calculate this value 1706436139<\/strong> to find the actual date and time of the expiration.<\/p>\n
    \"timestamp<\/figure>\n
    We can cross-verify this by searching the website on Google and viewing the certificate details.<\/p>\n
    \"ssl<\/figure>\n
    Create a Module Using ICMP the Protocol<\/h3>\n
    Example for the ICMP module in Blackbox configuration file \/etc\/blackbox\/blackbox.yml<\/code><\/strong>.<\/p>\n
    modules: \n  ping:\n    prober: icmp\n    timeout: 5s\n    icmp:\n      preferred_ip_protocol: \"ip4\"<\/code><\/pre>\n
    With this module, we can able to know whether the particular web server or network is reachable or not.<\/p>\n
    Update the Prometheus configuration file with this module \/etc\/prometheus\/prometheus.yml<\/strong><\/p>\n
    scrape_configs:\n  - job_name: \"blackbox-icmp\"\n    metrics_path: \/probe\n    params:\n      module: [ping]\n    static_configs:\n      - targets:\n        - 35.80.157.25\n        - devopscube.com\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: __param_target\n      - source_labels: [__param_target]\n        target_label: instance\n      - target_label: __address__\n        replacement: 172.31.19.73:9115<\/code><\/pre>\n
    Before starting testing, I am blocking the port for ICMP protocol in my EC2 instance so that the result should fail.<\/p>\n
    \"ec2<\/figure>\n
    In the Blackbox Exporter, you can see the failure result<\/p>\n
    \"blackbox<\/figure>\n
    I am enabling the ICMP<\/strong> protocol in the target EC2 instance.<\/p>\n
    \"aws<\/figure>\n
    Now the result is Success<\/strong>, this ensures that the target server is reachable.<\/p>\n
    \"blackbox<\/figure>\n
    If you click the Logs<\/strong>, you can able to see what are the probe metrics this module collects.<\/p>\n
    \"blackbox<\/figure>\n
    Let’s take one of the metrics from this and make a query in Prometheus to see the results, also we can visualize them with Grafana.<\/p>\n
    I am choosing the probe_icmp_duration_seconds <\/strong>metric.<\/p>\n
    Prometheus output<\/strong><\/p>\n
    \"prometheus<\/figure>\n
    Grafana output<\/strong><\/p>\n
    \"grafana<\/figure>\n
    Prometheus Blackbox Exporter Use Cases<\/strong><\/h2>\n
      \n
    1. We can know whether a particular website or server is currently reachable or not and also analyze how much time it takes to reach it.<\/li>\n
    2. Create alerts based on the server’s health, for example, if the server is not reachable or it is taking too much time to respond, we will get alerts.<\/li>\n
    3. SSL\/TLS certificate is one of the essential security components of our servers, we can track whether the certificate is present in the server or is valid.<\/li>\n
    4. We all know, that the DNS resolution should be very fast, only then can we access a website real quick, if it is taking too much time, that will kill the user experience, this exporter can check the DNS-related responses.<\/li>\n<\/ol>\n
      Conclusion<\/strong><\/h2>\n
      This setup gives you an overall idea about, what is Blackbox<\/strong> Exporter <\/strong>and how this utility works with Prometheus.<\/p>\n
      Here, I have created modules, only for testing purposes, but you can create modules with a lot of customization because the requirements will differ from person to person.<\/p>\n
      I am attaching the official documentation<\/a> with this for your further customization.<\/p>\n
      \n
      Ngu\u1ed3n:<\/strong> How To Set Up Blackbox Exporter On VM – Ultimate Guide \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
      Source: https:\/\/devopscube.com\/set-up-blackbox-exporter-on-vm\/<\/p>\n","protected":false},"author":1,"featured_media":754,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-753","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/753","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=753"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/753\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/754"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=753"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=753"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=753"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}