{"id":842,"date":"2024-06-26T12:10:52","date_gmt":"2024-06-26T12:10:52","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=842"},"modified":"2024-06-26T12:10:52","modified_gmt":"2024-06-26T12:10:52","slug":"prometheus-jmx-exporter-on-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=842","title":{"rendered":"How to Setup Prometheus JMX Exporter on Kubernetes"},"content":{"rendered":"

In this blog, we will look at the setup and configuration of the Prometheus JMX Exporter<\/strong> on a Kubernetes using a Java application.<\/p>\n

To better understand the JMX exporter, we will use a Spring Boot Java application and export all the JMX metrics to the Prometheus server.<\/p>\n

By the end of this guide, you will learn to:<\/p>\n

    \n
  1. Setup JMX Exporter with Java application image<\/li>\n
  2. Export JMX metrics to the Prometheus server<\/li>\n
  3. Scrape JMX metrics of the app using Prometheus SD configs.<\/li>\n
  4. Validate and query JMX metrics from the Prometheus dashboard.<\/li>\n<\/ol>\n

    Let\u2019s dive right in.<\/p>\n

    Prerequisites<\/h2>\n

    For setting up and testing this setup, you should have the following.<\/p>\n

      \n
    1. Kubernetes cluster<\/a> with deployment permissions.<\/li>\n
    2. Prometheus Server running on Kubernetes<\/a>.<\/li>\n<\/ol>\n

      JMX Exporter<\/h2>\n

      First, let’s understand how the JMX exporter works.<\/p>\n

      The following image shows the JMX Exporter workflow when deploying it on Kubernetes.<\/p>\n

      \"JMX<\/figure>\n

      The MBean (Managed Bean) server, located within the Java Virtual Machine (JVM) of a Java application, acts as a repository for data related to the application’s health and performance.<\/p>\n

      The MBean component exposes the application metrics such as status, actions, and operation information.<\/p>\n

      Java agents can access the Mbean server and get the health and performance data. Prometheus JMX Exporter is also a Java agent, which is capable of accessing the MBean server to access the data and transform that data into the Prometheus metric format.<\/p>\n

      Prometheus then scrapes the metrics from the JMX Exporter’s default metric storage path, which is \/metrics<\/code>.<\/strong><\/p>\n

      JMX Exporter Setup on Kubernetes<\/strong><\/h2>\n

      Follow the steps given below to setup JMX exporter with java applications on Kubernetes.<\/p>\n

      Step 1: Build Application Docker Image With JMX Agent<\/h3>\n

      JMX Exporter Java agent should be run with the application on the runtime.<\/p>\n

      You can download the JMX exporter agent jar file from here -> JMX Exporter Download<\/a>.<\/p>\n

      So when you build the Docker image for you application, you should add the JMX exporter along with the image.<\/p>\n

      Here is an example of a Dockerfile<\/a> that has both Java application jar and jmx-agent<\/strong> jar file.<\/p>\n

      FROM techiescamp\/jre-17:1.0.0\n\nENV APP_NAME=jmx-java-app\nENV JMX_PORT=5556\n\nWORKDIR \/app\n\nCOPY \/java-agent\/*.jar \/app\/jmx-agent.jar\nCOPY \/java-app\/*.jar \/app\/java.jar\nCOPY config.yaml \/app\/\n\nCMD java -javaagent:jmx-agent.jar=${JMX_PORT}:config.yaml -jar java.jar<\/code><\/pre>\n
      I have already built the docker image and pushed it to Dockerhub. If you are learning ot testing JMX, you can use the following image directly.<\/p>\n
      techiescamp\/jmx-pet-clinic-app:latest<\/code><\/pre>\n
      Step 2: Create ConfigMap for the JMX Exporter<\/h3>\n
      We need to create a ConfigMap<\/strong> named jmx-exporter-configmap.yaml<\/code> object for the JMX configs. In this file, we will provide information to filter the required metrics from the MBean server<\/strong>.<\/p>\n
      This config avoids unnecessary metric collection limiting it to only the metrics we need.<\/p>\n
      Create jmx-exporter-configmap.yaml<\/code><\/p>\n
      apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: jmx-exporter-config\n  namespace: monitoring\ndata:\n  config.yaml: |-\n    lowercaseOutputName: true\n    lowercaseOutputLabelNames: true\n    whitelistObjectNames: [\"java.lang:type=Memory\", \"java.lang:type=GarbageCollector,*\"]<\/code><\/pre>\n
      Here, you can see that I have mentioned that the namespace value is monitoring<\/strong>, this is because I have all my monitoring components in that particular namespace.<\/p>\n
      If you want to use this same configuration, create a namespace with the same name before beginning the configuration.<\/p>\n
      In the data section, we can define what kind of metrics the JMX Exporter has to collect from the MBean server<\/strong>. In this configuration I have given a small example to filter memory-related metrics of the application.<\/p>\n
      To know more about the configuration, please visit the official documentation<\/a>.<\/p>\n
      To deploy this configuration in the Kubernetes, use the following command.<\/p>\n
      kubectl apply -f jmx-exporter-configmap.yaml<\/code><\/pre>\n
      To view the list configmaps in the monitoring namespace.<\/p>\n
      kubectl get configmaps -n monitoring<\/code><\/pre>\n
      Step 3: Create a Deployment Manifest for the Application<\/h3>\n
      When creating the Deployment<\/strong> manifest, we have to expose the application and the metric ports, only then can we access the application and metrics over the Internet.<\/p>\n
      Create a deployment YAML file app-deployment.yaml.<\/code><\/p>\n
      apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: jmx-java-app\n  namespace: monitoring\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      app: jmx-java-app\n  template:\n    metadata:\n      labels:\n        app: jmx-java-app\n    spec:\n      containers:\n        - name: jmx-java-app-container\n          image: techiescamp\/jmx-pet-clinic-app:latest\n          ports:\n            - containerPort: 8080\n            - containerPort: 5556\n          volumeMounts:\n            - name: jmx-exporter-config-volume\n              mountPath: \/app\/config.yaml\n              subPath: config.yaml\n      volumes:\n        - name: jmx-exporter-config-volume\n          configMap:\n            name: jmx-exporter-config<\/code><\/pre>\n
      Here, you can see I am exposing ports 8080 <\/strong>and 5556<\/strong> outside the container.<\/p>\n
      One important thing here is the volumeMounts<\/strong> section. The mountPath is uses the configmap as volumes. It overwrites the container default \/app\/config.yaml<\/code><\/p>\n
      Now deploy the manifest.<\/p>\n
      kubectl apply -f app-deployment.yaml<\/code><\/pre>\n
      To view the list of Deployments in the monitoring Namespace.<\/p>\n
      kubectl get deployments -n monitoring<\/code><\/pre>\n
      To view the list of Pods in the monitoring Namespace.<\/p>\n
      kubectl get pods -n monitoring -o wide<\/code><\/pre>\n
      For testing purposes, I have given two replicas, so we can see two Java application Pods running in the monitoring Namespace.<\/p>\n
      Step 4: Create a Service for Application<\/h3>\n
      Create a Service manifest for the application app-service.yaml<\/code><\/p>\n
      apiVersion: v1\nkind: Service\nmetadata:\n  name: jmx-java-app-nodeport\n  namespace: monitoring\nspec:\n  selector:\n    app: jmx-java-app\n  ports:\n  - name: http\n    protocol: TCP\n    port: 8080\n    targetPort: 8080\n    nodePort: 30750  \n  - name: jmx\n    protocol: TCP\n    port: 5556\n    targetPort: 5556\n    nodePort: 30850  \n  type: NodePort<\/code><\/pre>\n
      I want to access the application and the metrics over the internet so I am using the spec.type: NodePort<\/code>, and defining the nodePort numbers such as 30750 <\/strong>for the application and 30850<\/strong> for the metrics.<\/strong><\/p>\n
      To apply this configuration, use the following command.<\/p>\n
      kubectl apply -f app-service.yaml<\/code><\/pre>\n
      To view the list of Services in the monitoring Namespace.<\/p>\n
      kubectl get svc -n monitoring<\/code><\/pre>\n
      Now, let’s try to access the application and metrics over the internet, for that, you need the Public IP<\/strong> of any one of the instances and the port numbers.<\/strong><\/p>\n
      The output of the application<\/strong><\/p>\n
      \"petclinic<\/figure>\n
      The output of the metrics.<\/strong><\/p>\n
      \"jmx<\/figure>\n
      Step 5: Add Scrape Config in Prometheus<\/h3>\n
      Note<\/strong>: This configuration has to be done on the Prometheus server.<\/p><\/blockquote>\n
      The JMX Exporter is successfully exposing the metrics of the Java application<\/strong>.<\/p>\n
      Now we add a job in the Prometheus server configuration<\/strong> file to scrape these metrics, and then only we can store them in TSDB<\/strong> or make queries.<\/p>\n
      Find the Prometheus ConfigMap, in my setup, the file name is config-map.yaml<\/code>. Open it with a text editor.<\/p>\n
      vim config-map.yaml<\/code><\/pre>\n
      To view the list of configmaps in the monitoring namespace.<\/p>\n
      kubectl get configmaps -n monitoring<\/code><\/pre>\n
      Add the following contents inside the configuration file, under the scrape_configs.<\/strong><\/p>\n
      If you are using a label selector in the deployment manifest, replace all the occurrences of jmx-java-app<\/code><\/strong> with the respective label.<\/p>\n
      scrape_configs:\n    - job_name: 'jmx-java-app'\n        kubernetes_sd_configs:\n        - role: pod\n          namespaces:\n            names:\n              - monitoring\n        relabel_configs:\n        - source_labels: [__meta_kubernetes_pod_label_app]\n          action: keep\n          regex: 'jmx-java-app'\n        - source_labels: [__meta_kubernetes_namespace]\n          target_label: namespace\n        - source_labels: [__meta_kubernetes_pod_name]\n          target_label: pod_name\n        - target_label: __address__\n          replacement: \"jmx-java-app-nodeport:5556\"<\/code><\/pre>\n
      In this promtheus configuration, I am using the kubernetes_sd_configs<\/code> service discovery <\/a>method that automatically identifies the pods that belong to the jmx-java-app<\/code><\/strong> deployment using jmx-java-app<\/code><\/strong> label. So you don’t need to manually provide the target details for scrapping the JMX metrics.<\/p>\n
      To apply this configuration, use the following command.<\/p>\n
      kubectl apply -f config-map.yaml<\/code><\/pre>\n
      It might take a few minutes to replicate the changes in the Prometheus dashboard. If you dont see the changes, then perform a deployment rollout<\/strong> using the following command. Change the prometheus-deployment<\/strong> to your actual Prometheus deployment name.<\/p>\n
      kubectl rollout restart deployment prometheus-deployment -n monitoring<\/code><\/pre>\n
      Step 6: Validate JMX Metrics from Prometheus Dashboard<\/h3>\n
      You can validate the JMX targets from the Prometheus dashbaord.<\/p>\n
      Under the Targets<\/strong> section, we can see the endpoint details, status, and other information as shown below.<\/p>\n
      \"Prometheus<\/figure>\n
      In the Graph section, we can make queries and get results. For example, the following dashboard shows jvm_memory_bytes_used<\/code><\/strong> metrics from the app.<\/p>\n
      \"prometheus<\/figure>\n
      Step 6: Grafana Dashboard<\/h3>\n
      We can visualize<\/strong> the JMX metrics in Grafana dashboards as per our requirements.<\/p>\n
      If you have a Grafana setup<\/a> and added Prometheus as a data source<\/a>, you can create dashboards using the JMX metrics templates<\/a>. You check out visualizing Promtheus metrics on Grafana blog to know more.<\/p>\n
      \"Grafana<\/figure>\n
      Conclusion<\/strong><\/h2>\n
      In this blog we have covered steps to implement JMX Exporter on the Kubernetes<\/strong> cluster. You can make further configuration change that is suitable for your application.<\/p>\n
      Also, in the exporter Configmap, you can add other filter settings for the JMX Exporter to collect other metrics that is required for the monitoring.<\/p>\n
      Also, for setting for JMX exporter on Linux VM based prometheus setup<\/a>, please refer this guide.<\/a><\/p>\n
      If you face any errors or need clarification on the setup, drop a comment below.<\/p>\n
      \n
      Ngu\u1ed3n:<\/strong> How to Setup Prometheus JMX Exporter on Kubernetes \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
      Source: https:\/\/devopscube.com\/prometheus-jmx-exporter-on-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":843,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-842","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/842","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=842"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/842\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/843"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=842"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=842"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=842"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}