{"id":454,"date":"2025-01-07T07:56:55","date_gmt":"2025-01-07T07:56:55","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=454"},"modified":"2025-01-07T07:56:55","modified_gmt":"2025-01-07T07:56:55","slug":"setup-prometheus-helm-chart","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=454","title":{"rendered":"How to Setup Prometheus Using Helm Chart? – Detailed Guide"},"content":{"rendered":"

In this guide, we will look at the Prometheus setup on Kubernetes using a helm chart with all the best practices.<\/p>\n


\nIf you want to learn about all the Kubernetes objects<\/a> involved in the Prometheus setup, you can follow the Prometheus on Kubernetes <\/a>guide, where we used plain YAML manifest to deploy Prometheus.<\/span>
\n<\/p>\n

Prerequisites<\/h2>\n

For this setup, ensure you have the following prerequisites.<\/p>\n

    \n
  1. Helm configured on your workstation or the CI server where you want to run the Helm commands. (v3.16.3 or higher)<\/li>\n
  2. A working Kubernetes cluster<\/a> (v1.30 or higher)<\/li>\n<\/ol>\n

    Prometheus Helm Chart Repo<\/h2>\n

    The Prometheus community maintains all the Prometheus related Helm<\/a> charts in the following GitHub repository.<\/p>\n

    https:\/\/github.com\/prometheus-community\/helm-charts\/<\/code><\/pre>\n
    This repo contains Prometheus stack, exporters, Pushgateways, etc. You can install the required charts as per your requirements.<\/p>\n
    To get started, we will deploy the core Prometheus chart that installs the following.<\/p>\n
      \n
    1. Prometheus server<\/li>\n
    2. Alertmanager<\/a><\/li>\n
    3. Kube State Metrics<\/a><\/li>\n
    4. Prometheus Node Exporter<\/a><\/li>\n
    5. Prometheus Pushgateway<\/a><\/li>\n<\/ol>\n
      Except for the Prometheus server, other components are installed from the dependency charts (sub-charts). If you check the Chart.yaml<\/code>, you will find the added chart dependencies below.<\/p>\n
      \"prometheus<\/figure>\n
      You can refer to this Prometheus Architecture blog<\/a> to learn the complete workflow of Prometheus and its components.<\/p>\n
      Install Prometheus Stack Using Helm<\/h2>\n
      Now, let’s get started with the setup.<\/p>\n
      Follow the steps below to set up Prometheus using the community Helm chart.<\/strong><\/p>\n
      Step 1: Add Prometheus Helm Repo<\/h3>\n
      Add the Prometheus chart to your system using the following command.<\/p>\n
      helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts<\/code><\/pre>\n
      You can list all the charts in the repo using the following command. We are going to use the Prometheus<\/code><\/strong> chart.<\/p>\n
      helm search repo prometheus-community<\/code><\/pre>\n
      Before you deploy the Prometheus Helm chart, you can view all the YAML manifests<\/a> by converting the chart to plain YAML files using the following command.<\/p>\n
      helm template prometheus-community prometheus-community\/prometheus --output-dir prometheus-manifests<\/code><\/pre>\n
      Here is the tree view of all the associated charts with YAML and Prometheus YAML.<\/p>\n
      \u279c  prometheus-manifests tree\n.\n\u2514\u2500\u2500 prometheus\n    \u251c\u2500\u2500 charts\n    \u2502   \u251c\u2500\u2500 alertmanager\n    \u2502   \u2502   \u2514\u2500\u2500 templates\n    \u2502   \u2502       \u251c\u2500\u2500 configmap.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 serviceaccount.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 services.yaml\n    \u2502   \u2502       \u2514\u2500\u2500 statefulset.yaml\n    \u2502   \u251c\u2500\u2500 kube-state-metrics\n    \u2502   \u2502   \u2514\u2500\u2500 templates\n    \u2502   \u2502       \u251c\u2500\u2500 clusterrolebinding.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 deployment.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 role.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 service.yaml\n    \u2502   \u2502       \u2514\u2500\u2500 serviceaccount.yaml\n    \u2502   \u251c\u2500\u2500 prometheus-node-exporter\n    \u2502   \u2502   \u2514\u2500\u2500 templates\n    \u2502   \u2502       \u251c\u2500\u2500 daemonset.yaml\n    \u2502   \u2502       \u251c\u2500\u2500 service.yaml\n    \u2502   \u2502       \u2514\u2500\u2500 serviceaccount.yaml\n    \u2502   \u2514\u2500\u2500 prometheus-pushgateway\n    \u2502       \u2514\u2500\u2500 templates\n    \u2502           \u251c\u2500\u2500 deployment.yaml\n    \u2502           \u251c\u2500\u2500 service.yaml\n    \u2502           \u2514\u2500\u2500 serviceaccount.yaml\n    \u2514\u2500\u2500 templates\n        \u251c\u2500\u2500 clusterrole.yaml\n        \u251c\u2500\u2500 clusterrolebinding.yaml\n        \u251c\u2500\u2500 cm.yaml\n        \u251c\u2500\u2500 deploy.yaml\n        \u251c\u2500\u2500 pvc.yaml\n        \u251c\u2500\u2500 service.yaml\n        \u2514\u2500\u2500 serviceaccount.yaml<\/code><\/pre>\n
      From the manifests, you can see that the Prometheus Helm chart deploys the following.<\/p>\n
        \n
      1. Alertmanager (Statefulset)<\/li>\n
      2. Kube State Metrics (Deployment)<\/li>\n
      3. Prometheus Node Exporter (Daemonset)<\/li>\n
      4. Prometheus Pushgateway (Deployment)<\/li>\n
      5. Prometheus Server (Deployment)<\/li>\n<\/ol>\n
        Step 2: Customize Prometheus Helm Chart Configuration Values<\/h3>\n
        While deploying Prometheus, it is very important to know the default values that are part of the values.yaml <\/code><\/strong>file.<\/p>\n
        If you are using the community chart for your project requirements, you should modify the values.yaml<\/code><\/strong> file as per your environment requirements.<\/p>\n
        You can write all the default values to a values.yaml<\/code><\/strong> file using the following command.<\/p>\n
        helm show values prometheus-community\/prometheus > values.yaml<\/code><\/pre>\n
        The following are the images used in this Prometheus Helm chart.<\/p>\n
          \n
        1. quay.io\/prometheus-operator\/prometheus-config-reloader<\/li>\n
        2. quay.io\/prometheus\/prometheus<\/li>\n<\/ol>\n
          The subcharts use the following images.<\/p>\n
            \n
          1. quay.io\/prometheus\/alertmanager<\/li>\n
          2. registry.k8s.io\/kube-state-metrics\/kube-state-metrics<\/li>\n
          3. quay.io\/prometheus\/node-exporter<\/li>\n
          4. quay.io\/prometheus\/pushgateway<\/li>\n<\/ol>\n
            You can customize the values to your needs. For example, the Prometheus Persistent Volume is set to 8Gi by default.<\/p>\n
            Note:<\/strong> If you are running from a corporate network, you might not have access to these public images. You should first push these images to the organization private registry first and then deploy the chart. Also, check if the security guidelines allow you to push community image to private registries.<\/p><\/blockquote>\n
            Step 3: Deploy Prometheus using the Helm Chart<\/h3>\n
            First, create a namespace monitoring<\/code><\/strong>. We will deploy Prometheus in the monitoring namespace.<\/p>\n
            kubectl create namespace monitoring<\/code><\/pre>\n
            Now, let’s deploy Prometheus using the values.yaml file.<\/p>\n
            Here, I am adding two parameters to create a Persistent Volume for Prometheus and AlertManager.<\/p>\n
            We can do the same configurations in the values.yaml<\/code> as well.<\/p>\n
            I am using the EKS cluster for a demo, so the mentioned storage class gp2<\/code> is the default of the EKS cluster<\/a>.<\/p>\n
            helm upgrade -i prometheus prometheus-community\/prometheus \\\n    --namespace monitoring \\\n    --set alertmanager.persistence.storageClass=\"gp2\" \\\n    --set server.persistentVolume.storageClass=\"gp2\"<\/code><\/pre>\n
            You will get the status as deployed on a successful deployment<\/a>, as shown below.<\/p>\n
            \"the<\/figure>\n
            Before we access the Prometheus, we can check that all the components are deployed and running properly.<\/p>\n
            kubectl -n monitoring get all<\/code><\/pre>\n
            \"the<\/figure>\n
            Step 4: Port Forward the Prometheus Pod<\/h3>\n
            The above screenshot clearly shows that each Prometheus stack component has the Service type as Cluster IP, so it can only be accessed from inside the cluster.<\/p>\n
            But we need to access it from our local machine to see the dashboard, so we perform the port forwarding.<\/p>\n
            First, start with the Prometheus port forwarding, identify the Prometheus Pod name, and create that as an environment variable.<\/p>\n
            export POD_NAME=$(kubectl get pods --namespace monitoring -l \"app.kubernetes.io\/name=prometheus,app.kubernetes.io\/instance=prometheus\" -o jsonpath=\"{.items[0].metadata.name}\")<\/code><\/pre>\n
            To perform port forwarding, use the following command.<\/p>\n
            kubectl --namespace monitoring port-forward $POD_NAME 9090<\/code><\/pre>\n
            The port forwarding is properly done; you will see the following output.<\/p>\n
            \"the<\/figure>\n
            Don’t close the terminal; meanwhile, open any of the web browsers from the same machine and paste this URL localhost:9090<\/code><\/p>\n
            \"the<\/figure>\n
            In the Target section, we can see the cluster resources that Prometheus is monitoring by default.<\/p>\n
            \"the<\/figure>\n
            Now, we can try the Alertmanager port forwarding to see the dashboard.<\/p>\n
            export POD_NAME=$(kubectl get pods --namespace monitoring -l \"app.kubernetes.io\/name=alertmanager,app.kubernetes.io\/instance=prometheus\" -o jsonpath=\"{.items[0].metadata.name}\")<\/code><\/pre>\n
            To perform a Port forward to the Alertmanager Pod, use the following command.<\/p>\n
            kubectl --namespace monitoring port-forward $POD_NAME 9093<\/code><\/pre>\n
            Same as Prometheus, use localhost URL from the browser, but this time use the Port 9093.<\/code><\/p>\n
            \"The<\/figure>\n
            The Alertmanger is also working correctly, so the installation was successful.<\/p>\n
            Here, we have explored the port forwarding method to expose the Prometheus application.<\/p>\n
            Note:<\/strong> If you want a static endpoint to access the Prometheus via internal or external DNS, you can either use the type Node Port or Load Balancer<\/a> service. Also you can use ingress<\/a> to expose it via DNS. For TLS, use ingress TLS<\/a> configurations.<\/p><\/blockquote>\n
            Prometheus Advanced Configuration<\/h2>\n
            Given below are the advanced configurations and best practices you can change based on your requirements.<\/p>\n
            You can find the options below in the Helm values file itself.<\/p>\n
            Storage<\/h3>\n
            By default, an 8 GB persistent volume will be attached to the Prometheus pod. If you want Prometheus to collect metrics from more workloads, increase the storage size.<\/p>\n
            server:\n  persistentVolume:\n     size: 20Gi<\/code><\/pre>\n
            Retention by Time or Size<\/h3>\n
            By default, the retention period for metrics will be 15 days, after 15 days, the metric will be deleted.<\/p>\n
            You can modify the retention by date or time.<\/p>\n
            \n
            \ud83d\udca1<\/div>\n
            Don’t mix time based retention and size based retention, use any one<\/div>\n<\/div>\n
            You can modify the retention on the Helm chart, as shown below.<\/p>\n
            server:\n  retention: 30d\n    # OR\n  retentionSize: 160Gi     <\/code><\/pre>\n
            This configuration deletes the metrics after 30 days, or if you choose size-based retention, it will be deleted once it reaches 160 GB.<\/p>\n
            Resource Limit and Request<\/h3>\n
            By default, there will be no resource limit and request, but setting up resource limits and requests is important so that the pod can let the cluster know its resource requirements.<\/p>\n
            Below is an example of a resource request and limit.<\/p>\n
            server:\n  resources:\n    requests:\n      cpu: \"500m\"\n      memory: \"512Mi\"\n    limits:\n      cpu: \"500m\"\n      memory: \"512Mi\"<\/code><\/pre>\n
            If your Prometheus is collecting metrics from more workloads, increase the limits accordingly, or the pod will be restarted or OOMKILL because of insufficient resources.<\/p>\n
            Basic High Availability<\/h3>\n
            For high availability, you can do the following:<\/p>\n