{"id":737,"date":"2023-11-16T14:56:14","date_gmt":"2023-11-16T14:56:14","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=737"},"modified":"2023-11-16T14:56:14","modified_gmt":"2023-11-16T14:56:14","slug":"prometheus-alert-manager","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=737","title":{"rendered":"How to Setup VPC Secondary Network For EKS Cluster"},"content":{"rendered":"

In this comprehensive guide, I have converted detailed steps to set up Prometheus Alert Manager and configure email and Slack alerts.<\/p>\n

Setup Prerequisites<\/h2>\n

You need two Ubuntu servers for this setup.<\/p>\n

server-01<\/strong>, ie the monitoring server contains Prometheus, Alert Manager<\/strong>, and Grafana<\/strong> utilities.<\/p>\n

The second server (server-02<\/strong>), contains the Node Exporter<\/strong> utility.<\/p>\n

Before starting the alert manager setup, ensure you have Prometheus and Grafana configured on the server. You can follow the guides given below.<\/p>\n

    \n
  1. Prometheus Setup<\/a> (Server 01)<\/li>\n
  2. Grafana Setup<\/a> (Server 01)<\/li>\n
  3. Node Exporter Setup<\/a> (Server 02)<\/li>\n<\/ol>\n

    In this guide, we will only look at the Alert Manager setup and its configurations related to email and Slack alerting.<\/p>\n

    Let’s get started with the setup.<\/p>\n

    Step 1: Download Prometheus Alert Manager<\/h2>\n

    Download a suitable Alert Manager binaries<\/a>, which is suitable for your server. here we use the latest version, which is v0.26.0<\/code>.<\/p>\n

    wget https:\/\/github.com\/prometheus\/alertmanager\/releases\/download\/v0.26.0\/alertmanager-0.26.0.linux-amd64.tar.gz<\/code><\/pre>\n
    Create a user and group for the Alert Manager to allow permission only for the specific user.<\/p>\n
    sudo groupadd -f alertmanager\nsudo useradd -g alertmanager --no-create-home --shell \/bin\/false alertmanager<\/code><\/pre>\n
    Creating directories is \/etc<\/code> and \/var\/lib<\/code> to store the configuration and library files and change the ownership of the directory only for the specific user.<\/p>\n
    sudo mkdir -p \/etc\/alertmanager\/templates\nsudo mkdir \/var\/lib\/alertmanager\nudo chown alertmanager:alertmanager \/etc\/alertmanager\nsudo chown alertmanager:alertmanager \/var\/lib\/alertmanager<\/code><\/pre>\n
    Unzip the Alert Manager binaries file and enter it into the directory.<\/p>\n
    tar -xvf alertmanager-0.26.0.linux-amd64.tar.gz\ncd alertmanager-0.26.0.linux-amd64<\/code><\/pre>\n
    Copy the alertmanager<\/code> and amtol<\/code> files in the \/usr\/bin<\/code> directory and change the group and owner to alertmanager<\/code>. As well as copy the configuration file alertmanager.yml<\/code> to the \/etc<\/code> directory and change the owner and group name to alertmanager<\/code>.<\/p>\n
    sudo cp alertmanager \/usr\/bin\/\nsudo cp amtool \/usr\/bin\/\nsudo chown alertmanager:alertmanager \/usr\/bin\/alertmanager\nsudo chown alertmanager:alertmanager \/usr\/bin\/amtool\nsudo cp alertmanager.yml \/etc\/alertmanager\/alertmanager.yml\nsudo chown alertmanager:alertmanager \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n
    Step 2: Setup Alert Manager Systemd Service<\/h2>\n
    Create a service file in \/etc\/systemd\/system<\/code> and the file name is alertmanager.service<\/code>.<\/p>\n
    cat <<EOF | sudo tee \/etc\/systemd\/system\/alertmanager.service\n[Unit]\nDescription=AlertManager\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nUser=alertmanager\nGroup=alertmanager\nType=simple\nExecStart=\/usr\/bin\/alertmanager \\\n    --config.file \/etc\/alertmanager\/alertmanager.yml \\\n    --storage.path \/var\/lib\/alertmanager\/\n\n[Install]\nWantedBy=multi-user.target\nEOF\nsudo chmod 664 \/usr\/lib\/systemd\/system\/alertmanager.service<\/code><\/pre>\n
    After providing the necessary permission to the file reload the background processes and start the Alert Manager service. To prevent the manual restart of the service after reboot, enable the service.<\/p>\n
    sudo systemctl daemon-reload\nsudo systemctl start alertmanager.service\nsudo systemctl enable alertmanager.service<\/code><\/pre>\n
    Check the status of the service and ensure everything is working fine.<\/p>\n
    sudo systemctl status alertmanager<\/code><\/pre>\n
    To access the Prometheus Alert Manager over the internet, use the following command.<\/p>\n
    http:\/\/<alertmanager-ip>:9093<\/code><\/pre>\n
    Instead of <alertmanager><\/code>, provide your instance public IP with the default Alert Manager port number, which is 9093<\/code>.<\/p>\n
    The output user interface you will get is this.<\/p>\n
    \"Alert<\/figure>\n
    Currently, there are no alerts in the dashboard. After finishing the configurations, we conducted some tests to see how the alert works with the Alert Manager.<\/p>\n
    Step 3: Setup SMTP Credentials<\/h2>\n
    You can use any SMTP with the Alert Manager.<\/p>\n
    For this setup, I am using AWS SES as SMTP to configure email alerts with Alert Manager.<\/p>\n
    If you have already configured SES in your AWS account, you can generate the SMTP credentials.<\/p>\n
    During the setup, please note down the SMTP endpoint<\/code> and STARTTLS Port<\/code>. This information is required to configure the Alert Manager email notification.<\/p>\n
    To generate a new SMTP credential, open the SES dashboard and click on Create SMTP credentials<\/code>.<\/p>\n
    \"AWS<\/figure>\n
    It will redirect to IAM user creation. modify the user name if necessary and don’t have to change any other parameters.<\/p>\n
    \"IAM<\/figure>\n
    IAM user name, SMTP user name, and SMTP password will be generated. This information is required to configure the Alert Manager notification setup.<\/p>\n
    \"Getting<\/figure>\n
    Note down and keep the SMTP details. We will use it in the alert manager configuration step.<\/p>\n
    Step 4: Generate Slack Webhook<\/h2>\n
    Create a Slack channel to get the alert manager notifications. You can also use the existing channel and add members who need to get the notifications.<\/p>\n
    \"\"<\/figure>\n
    Navigate to the administration section and select Manage apps<\/code> to reach the Slack app directory.<\/strong><\/p>\n
    \"\"<\/figure>\n
    Search and find Incoming WebHooks<\/strong> from the Slack app directory and select Add to Slack<\/code> to go to the webhook configuration page.<\/p>\n
    \"\"<\/figure>\n
    In the new configuration tab, choose the channel you created before and click on Add Incoming WebHooks integration.<\/strong><\/p>\n
    \"\"<\/figure>\n
    Now, the webhook will be created. store them securely to configure them with Alert Manager.<\/p>\n
    \"\"<\/figure>\n
    Step 5: Configure Alert Manager With SMTP and Slack API<\/h2>\n
    All the configurations for the Alert manager should be part of the alertmanager.yml<\/code><\/strong> file.<\/p>\n
    Open the Alert Manager configuration file.<\/p>\n
    sudo vim \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n
    Inside the configuration file, you should add the Slack and SES information to get the notifications as highlighted below in bold.<\/p>\n
    global:\n  resolve_timeout: 1m\n  slack_api_url: '$slack_api_url'\n\nroute:\n  receiver: 'slack-email-notifications'\n\nreceivers:\n- name: 'slack-email-notifications'\n  email_configs:\n      - to: 'arunlal@abcdefg.com'\n        from: 'email@devopsproject.dev'\n        smarthost: 'email-smtp.us-west-2.amazonaws.com:587'\n        auth_username: '$SMTP_user_name'\n        auth_password: '$SMTP_password'\n\n        send_resolved: false\n\n  slack_configs:\n  - channel: '#prometheus'\n    send_resolved: false\n<\/code><\/pre>\n
    In the global<\/code> section, provide the Slack webhook that you have created already.<\/p>\n
    In the receivers<\/code> section, modify the to<\/code> address and from<\/code> address, and provide the SMTP endpoint and port number in smarthost<\/code> section. auth_username<\/code> as SMTP user name and auth_password<\/code> as SMTP password.<\/p>\n
    Modify the - channel<\/code> with the slack channel name in slack_configs<\/code> section.<\/p>\n
    Once the configuration part is done, restart the Alert Manager service and ensure everything is working fine by checking the status of the service.<\/p>\n
    sudo systemctl restart alertmanager.service\nsudo systemctl status alertmanager.service<\/code><\/pre>\n
    Step 6: Create Prometheus Rules<\/h2>\n
    Prometheus rules are essential to trigger the alerts. Based on the rules, Prometheus will identify the situations and send an alert to the Alert Manager.<\/p>\n
    We can create multiple rules in YAML files as per the alert requirements.<\/p>\n
    Let’s create a couple of alert rules in separate rule YAML files and validate them by simulating thresholds.<\/p>\n
    Rule 1: Create a rule to get an alert when the CPU usage goes more than 60%.<\/strong><\/h3>\n
    cat <<EOF | sudo tee \/etc\/prometheus\/cpu_thresholds_rules.yml\ngroups:\n  - name: CpuThreshold\n    rules:\n      - alert: HighCPUUsage\n        expr: 100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100) > 60\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"High CPU usage on {{ $labels.instance }}\"\n          description: \"CPU usage on {{ $labels.instance }} is greater than 60%.\"\nEOF<\/code><\/pre>\n
    We can add more than one rule in a single file, and modify the values as per your requirements. In this, if the CPU usage goes more than 60% and is sustained for more than a minute, we get an alert.<\/p>\n
    Rule 2: Rule for memory usage alert<\/strong><\/h3>\n
    cat <<EOF | sudo tee \/etc\/prometheus\/memory_thresholds_rules.yml\ngroups:\n  - name: MemoryThreshold\n    rules:\n      - alert: HighRAMUsage\n        expr: 100 * (1 - (node_memory_MemAvailable_bytes \/ node_memory_MemTotal_bytes)) > 60\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"High RAM usage on {{ $labels.instance }}\"\n          description: \"RAM usage on {{ $labels.instance }} is greater than 60%.\"\nEOF<\/code><\/pre>\n
    Here also the same, if the memory usage goes more than 60%, Prometheus sends an alert to Alert Manager.<\/p>\n
    Rule 3: Rule for high storage usage<\/strong> alert<\/h3>\n
    cat <<EOF | sudo tee \/etc\/prometheus\/storage_thresholds_rules.yml\ngroups:\n  - name: StorageThreshold\n    rules:\n      - alert: HighStorageUsage\n        expr: 100 * (1 - (node_filesystem_avail_bytes \/ node_filesystem_size_bytes{mountpoint=\"\/\"})) > 50\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"High storage usage on {{ $labels.instance }}\"\n          description: \"Storage usage on {{ $labels.instance }} is greater than 50%.\"\nEOF<\/code><\/pre>\n
    Here, the primary storage usage goes more than it’s 50%, we get alert notifications through our Slack channel and Email.<\/p>\n
    Rule 4: Rule to get an alert when an instance is down.<\/strong><\/h3>\n
    cat <<EOF | sudo tee \/etc\/prometheus\/instance_shutdown_rules.yml\ngroups:\n- name: alert.rules\n  rules:\n  - alert: InstanceDown\n    expr: up == 0\n    for: 1m\n    labels:\n      severity: \"critical\"\n    annotations:\n      summary: \"Endpoint {{ $labels.instance }} down\"\n      description: \"{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes.\"\nEOF<\/code><\/pre>\n
    Once the rules are created, we should check that the rules are working. for that, we have to make some modifications to the Prometheus configuration file.<\/p>\n
    Step 7: Modify Prometheus Configurations<\/h2>\n
    Navigate to the Prometheus configuration file, which is in the \/etc\/prometheus<\/code> directory.<\/p>\n
    sudo vim \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n
    Add Alert Manager information, and Prometheus rules, and modify the scrape configurations as given below.<\/p>\n
    global:\n  scrape_interval: 15s \n  evaluation_interval: 15s \n\n# Alertmanager configuration\nalerting:\n  alertmanagers:\n    - static_configs:\n        - targets:\n           - localhost:9093\n\nrule_files:\n  - \"cpu_thresholds_rules.yml\"\n  - \"storage_thresholds_rules.yml\"\n  - \"memory_thresholds_rules.yml\"\n  - \"instance_shutdown_rules.yml\"\n\nscrape_configs:\n  - job_name: \"prometheus\"\n    static_configs:\n      - targets: [\"localhost:9090\"]\n\n  - job_name: \"node_exporter\"\n    scrape_interval: 5s\n    static_configs:\n      - targets: ['172.31.31.19:9100']<\/code><\/pre>\n
    In the global<\/code> section, modify the time interval to collect the metrics. the default interval is 15s, so every 15 seconds, Prometheus will collect metrics from Node Exporter.<\/strong>  Node Exporter is a tool, that should be installed on the server you want to monitor. It collects the metrics of the server and stores them in \/metrics<\/code> directory, Prometheus pulls them all to analyze and send notifications based on the rules we mention.<\/p>\n
    In the Alert Manager configuration<\/code> sections, unmask the target if your Prometheus and Alert Manager are on the same server, otherwise, provide the server’s IP address.<\/p>\n
    In the rule_files<\/code> section, provide the path of the Prometheus rules files also ensure the rules files are in the same directory where the Prometheus configuration file is in or give the proper path.<\/p>\n
    scrape_configs<\/code> section will have the server information that you want to monitor, the target<\/code> field contains the IP address of the target server.<\/p>\n
    Here, node-exporter<\/code> is the actual server we want to monitor and in the target<\/code> field we provide the private IP of that server. If your servers are in different network ranges, provide the public IP of that server and 9100<\/code> is the default port of the Node Exporter.<\/strong><\/p>\n
    After modifying the configurations, restart the Prometheus service using the following command.<\/p>\n
    sudo systemctl restart prometheus.service<\/code><\/pre>\n
    Check the status of the service to ensure there are no errors in the above configurations using the following command.<\/p>\n
    sudo systemctl status prometheus.service<\/code><\/pre>\n
    Step 8: Verify Configurations and Rules.<\/h2>\n
    Here, we conduct stress tests to verify the rules and configurations are working fine.<\/p>\n
    Test 1 : Memory Stress Test<\/strong><\/h3>\n
    Install stress-ng<\/code> utility in the server that you want to monitor and make stress tests.<\/p>\n
    sudo apt-get -y install stress-ng<\/code><\/pre>\n
    To stress the Memory of the server, use the following command.<\/p>\n
    stress-ng --vm-bytes $(awk '\/MemFree\/{printf \"%d\\n\", $2 * 1;}' < \/proc\/meminfo)k --vm-keep -m 10<\/code><\/pre>\n
    This will increase the memory usage up to 90% and we get an alert that the usage reaches 60%.<\/p>\n
    To visually analyze the metrics of the server, we use a Grafana dashboard.<\/p>\n
    \"\"<\/figure>\n
    Prometheus identifies the issue when the memory usage reaches 60% and will be ready to send an alert to the Alert Manager.<\/p>\n
    \"\"<\/figure>\n
    Then, Prometheus informed the Alert Manager to send notifications to the intended person through Slack and Email.<\/p>\n
    \"\"<\/figure>\n
    The output of the Slack notification is given below. here we can see the server details, severity level, and the cause of alert.<\/p>\n
    \"\"<\/figure>\n
    In the output of the Email notification also we get the same information. so that we can easily identify the issue at any time.<\/p>\n
    \"\"<\/figure>\n
    Test 2: CPU Stress Test<\/h3>\n
    To stress the CPU of the server, use the following command.<\/p>\n
    stress-ng --matrix 0<\/code><\/pre>\n
    This command will stress the CPU usage as its maximum and you can see the result of the usage from the diagram.<\/p>\n
    \"\"<\/figure>\n
    Test 3: Storage Stress Test<\/h3>\n
    To stress the storage of the server, use the following command.<\/p>\n
    fio --name=test --rw=write --bs=1M --iodepth=32 --filename=\/tmp\/test --size=20G<\/code><\/pre>\n
    The current storage is 30G, this command will increase the storage usage by more than 60% also you can change the value based on your server storage.<\/p>\n
    \"\"<\/figure>\n
    The output of the notifications for the above tests we conduct.<\/p>\n
    \"\"<\/figure>\n
    Conclusion<\/h2>\n
    In this blog, I have covered the real-world usage of the Prometheus Alert Manager with some simple examples.<\/p>\n
    you can further modify the configurations as per your needs. Monitoring is very much important even if it is virtual machines or containers. we can integrate them with Kubernetes as well.<\/p>\n
    Also, If you are preparing for prometheus certification (PCA), check out our Prometheus Certified Associate<\/a> exam Guide for exam tips and resources.<\/p>\n
    \n
    Ngu\u1ed3n:<\/strong> How to Setup VPC Secondary Network For EKS Cluster \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    Source: https:\/\/devopscube.com\/prometheus-alert-manager\/<\/p>\n","protected":false},"author":1,"featured_media":738,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-737","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=737"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/737\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/738"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=737"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=737"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}