{"id":633,"date":"2023-11-22T04:46:44","date_gmt":"2023-11-22T04:46:44","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=633"},"modified":"2023-11-22T04:46:44","modified_gmt":"2023-11-22T04:46:44","slug":"setup-prometheus-using-docker","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=633","title":{"rendered":"Setting Up Prometheus Using Docker Compose & Terraform"},"content":{"rendered":"

This blog covers the step-by-step guide to setting up the Prometheus Stack, which contains Prometheus, Grafana, and Alert Manager, using Docker and Docker Compose.<\/p>\n

It is a learning project that is part of the DevOps projects<\/a> initiative.<\/p>\n

To understand the basics of observability, please read the Observability guide.<\/a><\/p>\n

DevOps Tools \/ Service Used<\/h2>\n

In this setup, we have used the following open-source DevOps tools.<\/p>\n

    \n
  1. Docker<\/a><\/li>\n
  2. Prometheus<\/li>\n
  3. Alert Manager<\/li>\n
  4. Grafana<\/li>\n
  5. Prometheus Node Exporter<\/a><\/li>\n
  6. Terraform<\/li>\n<\/ol>\n

    The following are the AWS services used.<\/p>\n

      \n
    1. ec2<\/li>\n<\/ol>\n

      The following are the Linux concepts covered as part of the setup<\/p>\n

        \n
      1. Makefile<\/strong>: Used to modify the server IP address in Prometheus config.<\/li>\n
      2. Linux SWAP: <\/strong>To add swap to the EC2 instance.<\/li>\n
      3. Shell Scripts:<\/strong> To install Docker, Docker compose and add swap as part of ec2 system startup.<\/li>\n<\/ol>\n

        Setup Prerequisites<\/h2>\n

        To deploy the Prometheus stack using Docker Compose, we have the following prerequisites:<\/p>\n

          \n
        1. AWS Account with a key pair.<\/li>\n
        2. AWS CLI configured with the account.<\/li>\n
        3. Terraform is installed on your local machine.<\/li>\n<\/ol>\n

          Prometheus Stack Architecture & Workflow<\/h2>\n

          Here is the high-level overview of our setup architecture and workflow.<\/p>\n

          \"Prometheus<\/figure>\n

          In our setup, we will be using the following components.<\/p>\n

          1. Prometheus<\/h3>\n

          Prometheus is used to scrape the data metrics, pulled from exporters like node exporters. It used to provide metrics to Grafana. It also has a TSDB(Time series database) for storing the metrics. For more info please visit the Prometheus Documentation.<\/a><\/p>\n

          2. Alert Manager<\/h3>\n

          Alert manager is a component of Prometheus that helps us to configure alerts based on rules using a config file. We can notify the alerts using any medium like email, Slack, chats, etc.<\/p>\n

          With a nice dashboard where all alerts can be seen from the prometheus dashboard. For more info, please visit alert manager documentation<\/a><\/p>\n

          3. Node Exporter<\/h3>\n

          Node exporter<\/a> is the Prometheus agent that fetches the node metrics and makes it available for Prometheus to fetch from \/metrics<\/code><\/strong> endpoint. So basically node exporter collects all the server-level metrics such as CPU, memory, etc.<\/p>\n

          While other custom agents are still available and can be used for pushing metrics to prometheus. For more info, please visit Node Exporter Documentation<\/a><\/p>\n

          4. Grafana<\/h3>\n

          Grafana<\/a> is a Data visualization tool that fetches the metrics from Prometheus and displays them as colorful and useful dashboards.<\/p>\n

          It can be integrated with almost all available tools in the market. For more info please visit Grafana Official Documentation<\/a><\/p>\n

          To deploy the Prometheus stack, we will be using the following DevOps Tools<\/a><\/p>\n

          1. Terraform<\/h3>\n

          Terraform is one of the most popular Infrastructure as a Code tools created by HashiCorp. It allows developers to provide the entire infrastructure with code. Please refer to this terraform-archives<\/a> for more related blogs.<\/p>\n

          We will use Terraform to provision the EC2 instance required for the setup.<\/p>\n

          2. Docker<\/h3>\n

          Docker is a tool for packaging, deploying, and running applications in lightweight. If you want to learn about the basics of Docker, refer to the Docker basics<\/a> blog.<\/p>\n

          We will deploy Prometheus components and Grafana on Docker containers.<\/p>\n

          3. Docker Compose<\/h3>\n

          A Docker-based utility to run a multi-container Docker application. It allows you to define and configure the application’s services, networks, and volumes in a simple, human-readable YAML file.<\/p>\n

          Project IaC Code Explained<\/h2>\n

          All the IaC codes and configs used in this setup are hosted on the DevOps Projects Github repository.<\/p>\n

          Clone the DevOps projects repository<\/a> to your workstation to follow the guide.<\/p>\n

          git clone https:\/\/github.com\/techiescamp\/devops-projects<\/code><\/pre>\n
          The project code is present in the 04-prometheus-observability-stack folder. cd into the folder.<\/p>\n
          cd 04-prometheus-observability-stack<\/code><\/pre>\n
          Note: <\/strong>Use visual studio code or relevant IDE to understand the code structure better.<\/p><\/blockquote>\n
          Here is the project structure and config files.<\/p>\n
          .\n\u251c\u2500\u2500 LICENSE\n\u251c\u2500\u2500 Makefile\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 SECURITY.md\n\u251c\u2500\u2500 alertmanager\n\u2502   \u2514\u2500\u2500 alertmanager.yml\n\u251c\u2500\u2500 docker-compose.yml\n\u251c\u2500\u2500 prometheus\n\u2502   \u251c\u2500\u2500 alertrules.yml\n\u2502   \u251c\u2500\u2500 prometheus.yml\n\u2502   \u2514\u2500\u2500 targets.json\n\u2514\u2500\u2500 terraform-aws\n    \u251c\u2500\u2500 README.md\n    \u251c\u2500\u2500 modules\n    \u2502   \u251c\u2500\u2500 ec2\n    \u2502   \u2502   \u251c\u2500\u2500 main.tf\n    \u2502   \u2502   \u251c\u2500\u2500 outputs.tf\n    \u2502   \u2502   \u251c\u2500\u2500 user-data.sh\n    \u2502   \u2502   \u2514\u2500\u2500 variables.tf\n    \u2502   \u2514\u2500\u2500 security-group\n    \u2502       \u251c\u2500\u2500 main.tf\n    \u2502       \u251c\u2500\u2500 outputs.tf\n    \u2502       \u2514\u2500\u2500 variables.tf\n    \u251c\u2500\u2500 prometheus-stack\n    \u2502   \u251c\u2500\u2500 main.tf\n    \u2502   \u251c\u2500\u2500 outputs.tf\n    \u2502   \u2514\u2500\u2500 variables.tf\n    \u2514\u2500\u2500 vars\n        \u2514\u2500\u2500 ec2.tfvars<\/code><\/pre>\n
          Let’s understand the project files.<\/p>\n
          The alertmanager<\/strong> folder contains the alertmanager.yml<\/code><\/strong> file which is the configuration file. If you have details of the email, slack, etc. we can update accordingly.<\/p>\n
          The Prometheus<\/strong> folder contains alertrules.yml<\/code><\/strong> which is responsible for the alerts to be triggered from Prometheus to the alert manager. <\/p>\n
          The prometheus.yml config is also mapped to the alert manager endpoint to fetch, and Service discovery is used with the help of a file `file_sd_configs` to scrape the metrics using the targets.json<\/code><\/strong> file.<\/p>\n
          terraform-aws<\/strong>  directory allows you to manage and isolate resources effectively. Modules contain the reusable Terraform code. These contain the Terraform configuration files (main.tf, outputs.tf, variables.tf) for the respective modules.<\/p>\n
          The EC2 module also includes user-data.sh<\/code><\/strong> script to bootstrap the EC2 instance with Docker and Docker Compose<\/strong>. The security group module will create all the inbound & outbound rules required.<\/p>\n
          Prometheus-stack contains the configuration file main.tf<\/strong> required for running Terraform. Vars contains an ec2.tfvars<\/code><\/strong> file which contains variable values specific to all the files for the terraform project.<\/p>\n
          The Makefile<\/strong> is used to update the provisioned AWS EC2’s public IP address within the configuration files of prometheus.yml<\/code><\/strong> and targets.json<\/code><\/strong> located in the Prometheus directory.<\/p>\n
          The docker-compose.yml<\/code><\/strong> file incorporates various services Prometheus, Grafana, Node exporter & Alert Manager. These services are mapped with a network named ‘monitor<\/strong>‘ and have an ‘always<\/strong>‘ restart flag as well.<\/p>\n
          Docker Images<\/h2>\n
          We are using the following latest official Docker images available from the Docker Hub Registry.<\/p>\n
            \n
          1. prom\/prometheus<\/a><\/li>\n
          2. grafana\/grafana<\/a><\/li>\n
          3. prom\/node-exporter<\/a><\/li>\n
          4. prom\/alertmanager<\/a><\/li>\n<\/ol>\n
            Now that we have learned about the tools and tech and IaC involved in the setup, lets get started with the hands-on installation.<\/p>\n
            Provision Server Using Terraform<\/h2>\n
            Modify the values of ec2.tfvars<\/code><\/strong> file present in the terraform-aws\/vars<\/code><\/strong> folder. You need to replace the values highlighted in bold with values relevant to your AWS account & region.<\/p>\n
            If you are using us-west-2<\/strong>, you can continue with the same AMI ID.<\/p>\n
            # EC2 Instance Variables\nregion         = \"us-west-2\"\nami_id         = \"ami-03fd0aa14bd102718\"\ninstance_type  = \"t4g.micro\"\nkey_name       = \"techiescamp\"\ninstance_count = 1\nvolume-size = 20\n\n# VPC id\nvpc_id  = \"vpc-0a5ca4a92c2e10163\"\nsubnet_ids     = [\"subnet-058a7514ba8adbb07\"]\n\n# Ec2 Tags\nname        = \"prometheus-stack\"\nowner       = \"techiescamp\"\nenvironment = \"dev\"\ncost_center = \"techiescamp-projects\"\napplication = \"monitoring\"<\/code><\/pre>\n
            Now we can provision the AWS EC2 & Security group using Terraform.<\/p>\n
            cd terraform-aws\/prometheus-stack\/\nterraform fmt\nterraform init\nterraform validate<\/code><\/pre>\n
            Execute the plan and apply the changes.<\/p>\n
            terraform plan --var-file=..\/vars\/ec2.tfvars\nterraform apply --var-file=..\/vars\/ec2.tfvars<\/code><\/pre>\n
            Before typing ‘yes<\/strong>‘ make sure the desired resources are being created. After running Terraform, the Output should look like the following:<\/p>\n
            Apply complete! Resources: 2 added, 0 changed, 0 destroyed.\n\nOutputs:\n\ninstance_public_dns = [\n  \"ec2-34-216-95-97.us-west-2.compute.amazonaws.com\",\n]\ninstance_public_ip = [\n  \"34.216.95.97\",\n]\ninstance_state = [\n  \"running\",\n]<\/code><\/pre>\n
            Now we can connect to the AWS EC2 machine just created using the public IP. Replace the key path\/name and IP accordingly.<\/p>\n
            ssh -i ~\/.ssh\/techiescamp.pem ubuntu@34.216.95.97<\/code><\/pre>\n
            We will check the cloud-init logs<\/a> to see if the user data script has run successfully.<\/p>\n
            tail \/var\/log\/cloud-init-output.log<\/code><\/pre>\n
            Below is an example output. It should show the Docker and Docker Compose versions highlighted in the image.<\/p>\n
            \"verify<\/figure>\n
            Let’s verify the docker and docker-compose versions again.<\/p>\n
            sudo docker version\nsudo docker-compose version<\/code><\/pre>\n
            Now that the instance is ready with the required utilities, let’s deploy the Prometheus stack using Docker Compose.<\/p>\n
            Deploy Prometheus Stack Using Docker Compose<\/h2>\n
            First, clone the project code repository to the server.<\/p>\n
            git clone https:\/\/github.com\/techiescamp\/devops-projects\ncd devops-projects\/04-prometheus-observability-stack<\/code><\/pre>\n
            Execute the following make command<\/strong> to update the server IP in the Prometheus config file. <\/p>\n
            We are running the node exporter on the same server to fetch the server metrics. We also update the alert manager endpoint to the server’s public IP address.<\/p>\n
            make all<\/code><\/pre>\n
            You should see an output as shown below.<\/p>\n
            \"makefile<\/figure>\n
            Bring up the stack using Docker Compose. It will deploy Prometheus, Alert Manager, Node exporter, and Grafana<\/p>\n
            sudo docker-compose up -d<\/code><\/pre>\n
            On a successful execution, you should see the following output saying Running 5\/5<\/code><\/strong><\/p>\n
            \"docker-compose<\/figure>\n
            Now, with your servers IP address you can access all the apps on different ports.<\/p>\n
              \n
            1. Prometheus<\/strong>: http:\/\/your-ip-address:9090<\/a><\/li>\n
            2. Alert Manager<\/strong>:  http:\/\/your-ip-address:9093<\/a><\/li>\n
            3. Grafana<\/strong>: http:\/\/your-ip-address:3000<\/a><\/li>\n<\/ol>\n
              Now that the stack deployment is done, the rest of the configuration and testing will be done using the GUI.<\/p>\n
              Validate Prometheus Node Exporter Metrics<\/h2>\n
              If you visit http:\/\/your-ip-address:9090<\/a>, you will be able to access the Prometheus dashboard as shown below.<\/p>\n
              Validate the targets, rules, and configurations as shown below. The target would be the Node exporter URL.<\/p>\n
              \"validating<\/figure>\n
              Now let’s execute a promQL statement to view node_cpu_seconds_total<\/code><\/strong> metrics scraped from the node exporter.<\/p>\n
              avg by (instance,mode) (irate(node_cpu_seconds_total{mode!='idle'}[1m]))<\/code><\/pre>\n
              You should be able to data in graph as shown below.<\/p>\n
              \"executing<\/figure>\n
              Configure Grafana Dashboards<\/h2>\n
              Now, let’s configure Grafana dashboards for the Node Exporter metrics.<\/p>\n
              Grafana can be accessed at: http:\/\/your-ip-address:3000<\/a><\/p>\n
              Use admin<\/strong> as the username and password to log in to Grafana.<\/p>\n
              Now we need to add the Prometheus URL as the data source from Connections\u2192 Add new connection\u2192 Prometheus \u2192 Add new data source.<\/p>\n
              Here is the demo.<\/p>\n
              \"Configuring<\/figure>\n
              Configure Node Exporter Dashboard<\/h3>\n
              Grafana has many pre-built node exporter templates to give us a ready-to-use dashboard for the key node exporter metrics.<\/p>\n
              To import a dashboard, go to Dashboards –> Create Dashboard –> Import Dashboard –> Type 10180 and click load –> Select Prometheus Data source –> Import<\/p>\n
              Here is the demo.<\/p>\n
              \"Configuring<\/figure>\n
              Once the dashboard template is imported,  you should be able to see all the node exporter metrics as shown below.<\/p>\n
              \"Node<\/figure>\n
              Simulate & Test Alert Manager Alerts<\/h2>\n
              You can access the Alertmanager dashboard on http:\/\/your-ip-address:9093<\/a><\/p>\n
              \"Alert<\/figure>\n
              Alert rules are already backed into the Prometheus configuration through alertrules.yaml<\/code><\/strong>. If you go to the alerts option in the Prometheus menu, you will be able to see the configured alerts as shown below.<\/p>\n
              \"prometheus<\/figure>\n
              As you can see, all the alerts are in the inactive stage. To test the alerts, we need to simulate these alerts using a few Linux utilities.<\/p>\n
              You can also check the alert rules using the native promtool<\/a> prometheus CLI. We need to run promtool command from inside the prometheus container as shown below.<\/p>\n
              sudo docker exec -it prometheus promtool check rules \/etc\/prometheus\/alertrules.yml<\/code><\/pre>\n
              Test: High Storage & CPU Alert<\/h3>\n
              dd if=\/dev\/zero of=testfile_16GB bs=1M count=16384; openssl speed -multi $(nproc --all) &<\/code><\/pre>\n
              \"\"<\/figure>\n
              \"\"<\/figure>\n
              Now we can check the Alert Manager UI to confirm the alerts that were fired.<\/p>\n
              \"\"<\/figure>\n
              Now let’s rollback the changes and see the fired alerts has been resolved.<\/p>\n
              rm testfile_16GB && kill $(pgrep openssl)<\/code><\/pre>\n
              \"\"<\/figure>\n
              Cleanup The Setup<\/h2>\n
              To tear down the setup, execute the following Terraform command from your workstation.<\/p>\n
              terraform destroy --var-file=..\/vars\/ec2.tfvars<\/code><\/pre>\n
              Possible Errors<\/h2>\n
              If you don’t have the correct EC2 AMI ID set in the vars file, you will get the following error. To rectify the issue, update the correct AMI ID related to the region you are using.<\/p>\n
              Error: creating EC2 Instance: InvalidAMIID.NotFound: The image id '[ami-0a75bd84854bc95c9]' does not exist\n\u2502       status code: 400, request id: 34d38d4d-c3b8-47e6-9c27-1b1cbccbab83<\/code><\/pre>\n
              Conclusion<\/h2>\n
              As a quick recap we learned to provision the AWS infra using Terraform.<\/p>\n
              Then we brough up the Prometheus Observability stack using Docker compose and configured Grafana to create Node Exporter metrics dashboard.<\/p>\n
              Also we simulated alerts to check the validate the alerting.<\/p>\n
              If you’re facing any kind of issues, let me know in the comments section.<\/p>\n
              If you are interested in Prometheus certification (PCA), check out our Prometheus Certified Associate<\/a> exam Guide.<\/p>\n
              \n
              Ngu\u1ed3n:<\/strong> Setting Up Prometheus Using Docker Compose & Terraform \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
              Source: https:\/\/devopscube.com\/setup-prometheus-using-docker\/<\/p>\n","protected":false},"author":1,"featured_media":634,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-633","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/633","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=633"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/633\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/634"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}