{"id":751,"date":"2024-03-28T11:37:00","date_gmt":"2024-03-28T11:37:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=751"},"modified":"2024-03-28T11:37:00","modified_gmt":"2024-03-28T11:37:00","slug":"prometheus-architecture","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=751","title":{"rendered":"Learn Prometheus Architecture: A Complete Guide"},"content":{"rendered":"

In this guide, we will look at Prometheus architecture in detail to understand, configure, and leverage Prometheus architecture effectively.<\/p>\n

Prometheus is a popular open-source monitoring and alerting system written in Golang, capable of collecting and processing metrics<\/strong> from various targets. You can also query, view, analyse the metrics and get alerted based on the thresholds.<\/p>\n

Also, in today’s world, observability<\/a> is becoming paramount for every organisation, and Prometheus is one of the key observability tools in the open source space.<\/p>\n

What is Prometheus Architecture?<\/h2>\n

Here is the high level overview of Prometheus architecture.<\/p>\n

\"Prometheus<\/figure>\n

Prometheus primarily consists of the following.<\/p>\n

    \n
  1. Prometheus Server<\/li>\n
  2. Service Discovery<\/li>\n
  3. Time-Series Database (TSDB)<\/li>\n
  4. Targets<\/li>\n
  5. Exporters<\/li>\n
  6. Push Gateway<\/li>\n
  7. Alert Manager<\/li>\n
  8. Client Libraries<\/li>\n
  9. PromQL<\/li>\n<\/ol>\n

    Lets look at each component in detail.<\/p>\n

    Prometheus Server<\/strong><\/h2>\n

    The Prometheus server is the brain of the metric-based monitoring system<\/strong>. The main job of the server is to collect the metrics from various targets using pull model.<\/p>\n

    Target is nothing but a server, pod, endpoints etc which we will look in to detail in the next topic.<\/p>\n

    The general term for collecting metrics from the targets using Prometheus is called scraping.<\/strong><\/p>\n

    \"Prometheus<\/figure>\n

    Prometheus periodically scrapes the metrics, based on the scrape interval<\/strong> that we mention in the Prometheus configuration file.<\/strong><\/p>\n

    Here is an example configuration.<\/p>\n

    global:\n  scrape_interval: 15s \n  evaluation_interval: 15s \n  scrape_timeout: 10s \n\nrule_files:\n  - \"rules\/*.rules\"\n\nscrape_configs:\n  - job_name: 'prometheus'\n    static_configs:\n      - targets: ['localhost:9090'] \n  - job_name: 'node-exporter'\n    static_configs:\n      - targets: ['node-exporter:9100'] \n\nalerting:\n  alertmanagers:\n    - static_configs:\n        - targets: ['alertmanager:9093']\n<\/code><\/pre>\n
    Time-Series Database (TSDB)<\/strong><\/h2>\n
    The metric data which prometheus receives changes over time (CPU, memory, network IO etc..). It is called time-series data.  So Prometheus uses a Time Series Database<\/strong> (TSDB) to store all its data.<\/p>\n
    By default Prometheus stores all its data in an efficient format (chunks)<\/strong> in the local disk. Overtime, it compacts all the old data<\/strong> to save space. It also has retention policy to get rid of old data.<\/p>\n
    The TSDB has inbuilt mechanisms to manage data kept for long time. You can choose any of the following data retention policies.<\/p>\n
      \n
    1. Time based retention:<\/strong> Data will be kept for the specified days. The default retention is 15 days.<\/li>\n
    2. Size-based retention<\/strong>: You can specify the maximum data TSDB can hold. Once this limit it reached, prometheus will free up the space to accommodate new data.<\/li>\n<\/ol>\n
      Prometheus also offers remote storage options<\/strong>. This is primarily required for storage scalability, long-term storage, backup & disaster recovery etc.<\/p>\n
      Prometheus Targets<\/strong><\/h2>\n
      Target is the source<\/strong> where Prometheus scrape the metrics. A target could be servers, services, Kubernetes pods<\/a>, application endpoints, etc.<\/p>\n
      \"prometheus<\/figure>\n
      By default prometheus looks for metrics under \/metrics<\/code><\/strong> path of the target. The default path can be changed in the target configuration. This means, if you dont specify a custom metric path, Prometheus looks for the metrics under \/metrics<\/code><\/strong>.<\/p>\n
      Target configurations are present under the scrape_configs<\/strong> in the Prometheus<\/code> configuration file. Here is an example configuration.<\/p>\n
      scrape_configs:\n  \n  - job_name: 'node-exporter'\n    static_configs:\n      - targets: ['node-exporter1:9100', 'node-exporter2:9100']\n \n  - job_name: 'my_custom_job'\n    static_configs:\n      - targets: ['my_service_address:port']\n    metrics_path: '\/custom_metrics'\n\n  - job_name: 'blackbox-exporter'\n    static_configs:\n      - targets: ['blackbox-exporter1:9115', 'blackbox-exporter2:9115']\n    metrics_path: \/probe\n\n  - job_name: 'snmp-exporter'\n    static_configs:\n      - targets: ['snmp-exporter1:9116', 'snmp-exporter2:9116']\n    metrics_path: \/snmp<\/code><\/pre>\n
      From the target endpoints, prometheus expects data in certain text format<\/strong>. Each metric has to be on a new line.<\/p>\n
      Usually these metrics are exposed on target nodes using prometheus exporters<\/strong> running on the targets.<\/p>\n
      Prometheus Exporters<\/strong><\/h2>\n
      Exporters are like agents that run on the targets. It converts metrics<\/strong> from specific system to format that prometheus understands.<\/p>\n
      It could be system metrics like CPU, memory etc, or Java JMX metrics, MySQL metrics etc.<\/p>\n
      \"Prometheus<\/figure>\n
      By default these converted metrics are exposed by the exporter on \/metrics<\/code><\/strong> path(HTTPS endpoint) of the target.<\/p>\n
      For example, if you want to monitor a servers CPU and memory<\/strong>, you need to install a node exporter on that server and the node exporter exposes<\/strong> the CPU and memory metrics in the prometheus metrics format on \/metrics<\/code><\/strong>.<\/p>\n
      Once the Prometheus pulls<\/strong> the metrics, it will then combine the metric name, labels, value, and timestamp<\/strong> to give a structure<\/strong> to that data.<\/p>\n
      There are a lot of community Exporters<\/strong> available, but only some of them are officially approved  by Prometheus. In cases where you need more customizations, you need to create your own exporters.<\/p>\n
      Prometheus categorized the Exporters<\/strong> in various sections such as Databases, Hardware, Issue trackers and continuous integration, Messaging systems, Storage, Software exposing Prometheus metrics, Other third-party utilities, etc.<\/p>\n
      You can see the list of Exporters<\/strong> from each category from the official documentation<\/a>.<\/p>\n
      In the Prometheus configuration file, all the exporters’ details will be given under the scrape_configs<\/code>.<\/p>\n
      scrape_configs:\n  - job_name: 'node-exporter'\n    static_configs:\n      - targets: ['node-exporter1:9100', 'node-exporter2:9100']\n\n  - job_name: 'blackbox-exporter'\n    static_configs:\n      - targets: ['blackbox-exporter1:9115', 'blackbox-exporter2:9115']\n    metrics_path: \/probe\n\n  - job_name: 'snmp-exporter'\n    static_configs:\n      - targets: ['snmp-exporter1:9116', 'snmp-exporter2:9116']\n    metrics_path: \/snmp<\/code><\/pre>\n
      Prometheus Service Discovery<\/h2>\n
      Prometheus uses two methods to scrape metrics from the targets.<\/p>\n
        \n
      1. Static configs<\/strong>: When the targets have a static IP or DNS endpoint, we can use those endpoints as targets.<\/li>\n
      2. Sevice Discovery:<\/strong> In most autoscaling systems and distributed systems like Kubernetes, the target will not have a static endpoint. In this case, that target endpoints are discovered using prometheus service discovery and targets are added automatically to the prometheus configuration.<\/li>\n<\/ol>\n
        \"prometheus<\/figure>\n
        Before going further, let me show a small example of the Kubernetes service discovery<\/strong> block of the Prometheus configuration<\/strong> file using kubernetes_sd_configs<\/code>.<\/p>\n
        scrape_configs:\n      - job_name: 'kubernetes-apiservers'\n        kubernetes_sd_configs:\n        - role: endpoints\n        scheme: https\n        tls_config:\n          ca_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt\n        bearer_token_file: \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token\n        relabel_configs:\n        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]\n          action: keep\n          regex: default;kubernetes;https<\/code><\/pre>\n
        Kubernetes is the perfect example for dynamic targets.<\/strong>  Here, you cannot use the static targets method, because targets (pods) in a Kubernetes cluster<\/a> is ephemeral in nature and could be short lived.<\/p>\n
        There is also File-Based Service Discovery<\/strong> file_sd_configs<\/code> in Kubernetes. It is for static targets, but the main difference between the classic Static configuration <\/strong>static_configs<\/code> and the file_sd_configs<\/code> is that, in this, we create separate JSON or YAML files<\/strong> and keep our target information in them.  Prometheus will read the files to identify the targets.<\/p>\n
        Not only these two, but various service discovery methods are available such as consul_sd_configs (where prometheus get target details from consul<\/a>), ec2_sd_configs, etc.<\/p>\n
        To know more about the configuration details, please visit the official documentation<\/a>.<\/p>\n
        Prometheus Pushgateway<\/h2>\n
        Prometheus by default uses pull mechanism to scrap the metrics.<\/p>\n
        However, there are scenarios where metrics need to be pushed<\/strong> to prometheus.<\/p>\n
        Lets take an example of a batch job<\/strong> running on a Kubernetes cronjob<\/a> that runs daily for 5 mins based on certain events. In this scenario, Prometheus will not be able to scrape the service level metrics<\/strong> properly using pull mechanism.<\/p>\n
        So instead for waiting for prometheus to pull the metrics, we need to push the metrics<\/strong> to prometheus. To push metrics, prometheus offers a solution called Pushgateway. <\/strong>It is kind of a intermediate gateway.<\/p>\n
        Pushgateway<\/strong> needs to be run as a standalone component. The batch jobs can push the metrics to the pushgateway using HTTP API. Then Pushgateway<\/strong> exposes those metrics on \/metrics<\/code><\/strong> endpoint. Then prometheus scrapes those metrics from the Pushgateway.<\/p>\n
        \"Prometheus<\/figure>\n
        Pushgateway stores the metrics data temporarily in in-memory storage<\/strong>. It’s more of a temporary cache.<\/p>\n
        Pushgateway <\/strong>configuration also will be configured under the scrape_configs<\/code> section in the Prometheus<\/code> configuration.<\/p>\n
        scrape_configs:\n  - job_name: \"pushgateway\"\n        honor_labels: true\n        static_configs:\n        - targets: [pushgateway.monitoring.svc:9091]<\/code><\/pre>\n
        To send metrics to the Pushgateways, you need to use the prometheus Client Libraries<\/strong> and instrument the application<\/strong> or script to expose the required metrics.<\/p>\n
        Prometheus Client Libraries<\/strong><\/h2>\n
        Prometheus Client Libraries<\/strong> are software libraries that can be used to instrument application code to expose metrics in the way Prometheus understands.<\/p>\n
        In cases where you need custom instrumentation<\/strong> or you want to create your own exporters, you can use the client libraries.<\/p>\n
        A very good use case is batch jobs that need to push metrics to the Pushgateway. The batch job needs to be instrumented with client libraries to expose requirement metrics in prometheus format.<\/p>\n
        Here is an example Python Client Library<\/code> that exposes custom metrics named batch_job_records_processed_total<\/code><\/strong>.<\/p>\n
        from prometheus_client import start_http_server, Counter\nimport time\nimport random\n\nRECORDS_PROCESSED = Counter('batch_job_records_processed_total', 'Total number of records processed by the batch job')\n\ndef process_record():\n    time.sleep(random.uniform(0.01, 0.1))\n    RECORDS_PROCESSED.inc()\n\ndef batch_job():\n   \n    for _ in range(100):\n        process_record()\n\nif __name__ == '__main__':\n \n    start_http_server(8000)\n    print(\"Metrics server started on port 8000\")\n\n    batch_job()\n    print(\"Batch job completed\")\n\n    while True:\n        time.sleep(1)<\/code><\/pre>\n
        Also when using client libraries, prometheus_client<\/strong><\/code> HTTP server exposes the metrics in \/metrics <\/code>endpoint.<\/p>\n
        Prometheus has Client Libraries<\/strong> for almost every programming language, and also if you want to create a Client Library, you can do it.<\/p>\n
        To learn more about the guidelines of the creation and view the list of Client Libraries, you can refer to the official documentation.<\/a><\/p>\n
        Prometheus Alert Manager<\/h2>\n
        Alertmanager<\/a> is the key part of  Prometheus monitoring system. Its primary job is to send alerts based on metric thresholds set in the Prometheus alert configuration<\/strong>.<\/p>\n
        The alert get triggered by Prometheus<\/strong> and sent to Alertmanager. It in turn sends the alerts to the respective notification systems\/receivers<\/strong> (email, slack etc) configured in the alert manager configurations.<\/p>\n
        Also, alert manager takes care of the following.<\/p>\n
          \n
        1. Alert<\/strong> Deduplicating<\/strong>: Process of silencing duplicated alerts.<\/li>\n
        2. Grouping<\/strong>: Process of grouping related alerts togther.<\/li>\n
        3. Silencing<\/strong>: Silence alerts for maintenance or false positives.<\/li>\n
        4. Routing<\/strong>: Routing alerts to appropriate receivers based on severities.<\/li>\n
        5. Inhibition<\/strong>: Process of stopping low severity alert when there is a medium of high severity alert.<\/li>\n<\/ol>\n
          \"Prometheus<\/figure>\n
          Here is an example configuration of an alert rule.<\/p>\n
          groups:\n- name: microservices_alerts\n  rules:\n  - record: http_latency:average_latency_seconds\n    expr: sum(http_request_duration_seconds_sum) \/ sum(http_request_duration_seconds_count)\n  - alert: HighLatencyAlert\n    expr: http_latency:average_latency_seconds > 0.5\n    for: 5m\n    labels:\n      severity: critical\n    annotations:\n      summary: \"High latency detected in microservices\"\n      description: \"The average HTTP latency is high ({{ $value }} seconds) in the microservices cluster.\"\n\n<\/code><\/pre>\n
          This is an example of the routing configuration<\/strong> of the Alertmanager configuration file<\/p>\n
          routes:\n- match:\n    severity: 'critical'\n  receiver: 'pagerduty-notifications'\n\n- match:\n    severity: 'warning'\n  receiver: 'slack-notifications'<\/code><\/pre>\n
          Alert managers supports most of the message and notification systems such as Discord, Email, Slack, etc to reach the alert as a notification to the receiver.<\/p>\n
          PromQL<\/h2>\n
          PromQL is a flexible query language<\/strong> that can be used to query time series metrics from the prometheus.<\/p>\n
          We can directly  used the queries from the Prometheus<\/code> user interface<\/strong> or we can use curl<\/code> command to make a query over the command line interface.<\/p>\n
          Prometheus UI<\/strong><\/p>\n
          \"PromQL<\/figure>\n
          Query over the CLI<\/strong><\/p>\n
          curl \"http:\/\/54.186.154.78:30000\/api\/v1\/query?query=$(echo 'up' | jq -s -R -r @uri)\" | jq .<\/code><\/pre>\n
          Also, when you add prometheus as a data source to Grafana, you can use PromQL to query and create Grafana dashboards as shown below.<\/p>\n
          \"Grafana<\/figure>\n
          Conclusion<\/h2>\n
          This explains the major components of the Prometheus architecture and will give the basic overview of the Prometheus configuration, there are a lot more things you can do with the configuration.<\/p>\n
          Each organization’s requirements would be different and the implementation of Prometheus in different environments also vary such as VMs and Kubernetes. If you understand the fundamentals and the key configurations, you can easily implement it in any platform.<\/p>\n
          Also, it will help you prepare for the Prometheus certification<\/a> as well<\/p>\n
          As a next step, you can try setting up prometheus on Kubernetes<\/a>.<\/p>\n
          \n
          Ngu\u1ed3n:<\/strong> Learn Prometheus Architecture: A Complete Guide \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
          Source: https:\/\/devopscube.com\/prometheus-architecture\/<\/p>\n","protected":false},"author":1,"featured_media":752,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-751","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/751","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=751"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/751\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/752"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=751"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=751"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=751"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}