{"id":922,"date":"2021-04-06T16:20:50","date_gmt":"2021-04-06T16:20:50","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=922"},"modified":"2021-04-06T16:20:50","modified_gmt":"2021-04-06T16:20:50","slug":"node-exporter-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=922","title":{"rendered":"How to Setup Prometheus Node Exporter on Kubernetes"},"content":{"rendered":"

If you want to know how the Kubernetes nodes perform or monitor system-level insights<\/a> of kubernetes nodes, you need to set up a Prometheus node exporter on Kubernetes cluster.<\/p>\n

This guide will walk you through the node-exporter setup on a Kubernetes<\/a> cluster and integrate Prometheus scrape config to scrape the node metrics.<\/p>\n

It is also an important topics in the Prometheus Certified Associate<\/a> Certification.<\/p>\n

What is Prometheus Node Exporter?<\/h2>\n

Node exporter is an official Prometheus exporter<\/a> for capturing all the Linux<\/a> system-related metrics.<\/p>\n

It collects all the hardware and Operating System level metrics that are exposed by the kernel.<\/p>\n

You can use the node exporter to collect the system metrics from all your Linux systems. Check this article on node monitoring using node-exporter<\/a>.<\/p>\n

Why do we need Node Exporter on Kubernetes?<\/h2>\n

By default, most of the Kubernetes cluster<\/a>s expose the metric server metrics (Cluster level metrics from the summary API) and Cadvisor (Container<\/a> level metrics). It does not provide detailed node-level metrics.<\/p>\n

To get all the kubernetes node-level system metrics, you need to have a node-exporter running in all the kubernetes nodes. It collects all the Linux system metrics and exposes them via \/metrics<\/code> endpoint on port 9100<\/code><\/p>\n

Similarly, you need to install Kube state metrics<\/a> to get all the metrics related to kubernetes objects.<\/p>\n

Kubernetes Manifests<\/h2>\n

The Kubernetes manifest used in this guide is present in the Github repository<\/a>. Clone the repo to your local system.<\/p>\n

git clone https:\/\/github.com\/bibinwilson\/kubernetes-node-exporter<\/code><\/pre>\n
Setup Node Exporter on Kubernetes<\/h2>\n
\n
Note:<\/strong><\/b> If you don’t have the Prometheus setup, please follow my guide on setting up Prometheus on kubernetes<\/a>.<\/div>\n<\/div>\n
Here is what we are going to do.<\/p>\n
    \n
  1. Deploy node exporter on all the Kubernetes nodes as a daemonset<\/code>. Daemonset makes sure one instance of node-exporter is running in all the nodes. It exposes all the node metrics on port 9100<\/code> on the \/metrics<\/code> endpoint<\/li>\n
  2. Create a service that listens on port 9100<\/code> and points to all the daemonset node exporter pods. We would be monitoring the service endpoints (Node exporter pods)  from Prometheus using the endpoint job config. More explanation on this in the Prometheus config part.<\/li>\n<\/ol>\n
    Lest get started with the setup.<\/p>\n
    Step 1:<\/strong> Create a file name daemonset.yaml<\/code> and copy the following content.<\/p>\n
    \n
    Note: <\/strong><\/b>This Daemonset will be deployed in the monitoring namespace. If you wish to deploy it in a different namespace, change it in the following YAML<\/div>\n<\/div>\n
    apiVersion: apps\/v1\nkind: DaemonSet\nmetadata:\n  labels:\n    app.kubernetes.io\/component: exporter\n    app.kubernetes.io\/name: node-exporter\n  name: node-exporter\n  namespace: monitoring\nspec:\n  selector:\n    matchLabels:\n      app.kubernetes.io\/component: exporter\n      app.kubernetes.io\/name: node-exporter\n  template:\n    metadata:\n      labels:\n        app.kubernetes.io\/component: exporter\n        app.kubernetes.io\/name: node-exporter\n    spec:\n      containers:\n      - args:\n        - --path.sysfs=\/host\/sys\n        - --path.rootfs=\/host\/root\n        - --no-collector.wifi\n        - --no-collector.hwmon\n        - --collector.filesystem.ignored-mount-points=^\/(dev|proc|sys|var\/lib\/docker\/.+|var\/lib\/kubelet\/pods\/.+)($|\/)\n        - --collector.netclass.ignored-devices=^(veth.*)$\n        name: node-exporter\n        image: prom\/node-exporter\n        ports:\n          - containerPort: 9100\n            protocol: TCP\n        resources:\n          limits:\n            cpu: 250m\n            memory: 180Mi\n          requests:\n            cpu: 102m\n            memory: 180Mi\n        volumeMounts:\n        - mountPath: \/host\/sys\n          mountPropagation: HostToContainer\n          name: sys\n          readOnly: true\n        - mountPath: \/host\/root\n          mountPropagation: HostToContainer\n          name: root\n          readOnly: true\n      volumes:\n      - hostPath:\n          path: \/sys\n        name: sys\n      - hostPath:\n          path: \/\n        name: root\n<\/code><\/pre>\n
    Step 2:<\/strong> Deploy the daemonset using the kubectl command.<\/p>\n
    kubectl create -f daemonset.yaml<\/code><\/pre>\n
    Step 3:<\/strong> List the daemonset<\/code> in the monitoring namespace and make sure it is in the available state.<\/p>\n
    kubectl get daemonset -n monitoring<\/code><\/pre>\n
    Step 4: <\/strong>Create a file names service.yaml <\/code>and copy the following contents.<\/p>\n
    ---\nkind: Service\napiVersion: v1\nmetadata:\n  name: node-exporter\n  namespace: monitoring\n  annotations:\n      prometheus.io\/scrape: 'true'\n      prometheus.io\/port:   '9100'\nspec:\n  selector:\n      app.kubernetes.io\/component: exporter\n      app.kubernetes.io\/name: node-exporter\n  ports:\n  - name: node-exporter\n    protocol: TCP\n    port: 9100\n    targetPort: 9100<\/code><\/pre>\n
    Step 5:<\/strong> Create the service.<\/p>\n
    kubectl create -f service.yaml<\/code><\/pre>\n
    Step 6: <\/strong>Now, check the service’s endpoints and see if it is pointing to all the daemonset pods.<\/p>\n
    kubectl get endpoints -n monitoring <\/code><\/pre>\n
    \"Prometheus<\/figure>\n
    As you can see from the above output,  the node-exporter service has three endpoints. Meaning three node-exporter pods running on three nodes as part of Daemonset.<\/p>\n
    Node-exporter Prometheus Config<\/h2>\n
    We have the node-exporter daemonset running on port 9100 and a service pointing to all the node-exporter pods.<\/p>\n
    You need to add a scrape config to the Prometheus config file to discover all the node-exporter pods.<\/p>\n
    Let’s take a look at the Prometheus scrape config required to scrape the node-exporter metrics.<\/p>\n
          - job_name: 'node-exporter'\n        kubernetes_sd_configs:\n          - role: endpoints\n        relabel_configs:\n        - source_labels: [__meta_kubernetes_endpoints_name]\n          regex: 'node-exporter'\n          action: keep<\/code><\/pre>\n
    In this config, we mention the role as endpoints to scrape the endpoints with the name node-exporter<\/code>.<\/p>\n
    See Prometheus config map<\/a> file I have created for the Kubernetes monitoring stack. It includes all the scrape configs for kubernetes components.<\/p>\n
    Once you add the scrape config to Prometheus, you will see the node-exporter targets in Prometheus, as shown below.<\/p>\n
    \"Node<\/figure>\n
    Querying Node-exporter Metrics in Prometheus<\/h2>\n
    Once you verify the node-exporter target state in Prometheus, you can query the Prometheus dashboard’s available node-exporter metrics.<\/p>\n
    All the metrics from node-exporter is prefixed with node_<\/code><\/p>\n
    You can query the metrics with different PromQL expressions. See querying basics<\/a> to learn about PromQL queries.<\/p>\n
    If you type node_<\/code> in the Prometheus dashboard, it will list all the available metrics as shown below.<\/p>\n
    \"Querying<\/figure>\n
    Visualizing Prometheus Node Exporter Metrics as Grafana Dashboards<\/h2>\n
    Visualising the node exporter metrics on Grafana is not difficult as you think.<\/p>\n
    A community Grafana node exporter dashboard template <\/a>has a predefined dashboard with all the supported node exporter metrics.<\/p>\n
    You can modify the template as per your project requirements.<\/p>\n
    If you don’t know how to import a community template, please check my Grafana Prometheus integration<\/a> article, where I have added the steps to import community dashboard templates.<\/p>\n
    So here is how the node-exporter Grafana dashboard looks  for CPU\/memory and disk statistics.<\/p>\n
    \"node<\/figure>\n
    Once you have the dashboard, you will find the following sections. If you expand it, you will find all the metrics panel.<\/p>\n
    \"node<\/figure>\n
    More References<\/h2>\n
      \n
    1. Official Node exporter Github repository<\/a><\/li>\n
    2. Prometheus Linux host metrics guide<\/a><\/li>\n
    3. Prometheus Exporters<\/a><\/li>\n<\/ol>\n
      \n
      Ngu\u1ed3n:<\/strong> How to Setup Prometheus Node Exporter on Kubernetes \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
      Source: https:\/\/devopscube.com\/node-exporter-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":923,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-922","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/922","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=922"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/922\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/923"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=922"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=922"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=922"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}