{"id":576,"date":"2025-03-07T18:44:53","date_gmt":"2025-03-07T18:44:53","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=576"},"modified":"2025-03-07T18:44:53","modified_gmt":"2025-03-07T18:44:53","slug":"setup-grafana-loki","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=576","title":{"rendered":"How to Setup Grafana Loki on Kubernetes & Query Logs"},"content":{"rendered":"

In this blog, we will look into a step-by-step guide to setup Grafana Loki on Kubernetes using Helm<\/a>.<\/p>\n

We will also look at setting up Alloy which is an agent for Loki to collect logs and setup Grafana for querying logs using Loki.<\/p>\n

Grafana Loki<\/a> is an open-source log aggregation system that supports large-scale log management. It is a widely adopted tool for organizations observability<\/a> stack.<\/p>\n

If you want to learn in detail about Grafana Loki’s architecture, read the detailed Grafana Loki Architecture blog.<\/a><\/p>\n

Pods that will be deployed for Loki are listed below:<\/p>\n

    \n
  1. Loki (sts) – The Loki pod contains all its components, like ingester, distributor, etc.
    This pod is responsible for compressing the logs into chunks and storing them, whether it’s configured for the local filesystem or the external object storage.<\/li>\n
  2. Loki-canary (daemonset): This is a health check pod for Loki. It sends dummy logs to check Loki’s performance, such as latency and success rates.<\/li>\n
  3. Loki-chunks-cache (sts): This caches the log chunks that have been recently queried so that the same chunk can be queried faster next time.<\/li>\n
  4. Loki-gateway (deployment): This is an Nginx Ingress pod that routes requests to the correct Loki components. It will be more helpful if you use it with Loki Micro Services.<\/li>\n
  5. Loki-results (sts): This caches the results from previous queries for faster query if the same query is used next time.<\/li>\n<\/ol>\n

    Loki Workflow<\/h2>\n

    A workflow overview of Loki is shown in the diagram below.<\/p>\n

    \"Loki
    Click to View in HD<\/span><\/figcaption><\/figure>\n

    Let me explain the diagram<\/p>\n

      \n
    1. The diagram shows two types of workflow, one given in numbers and the other in letters. Let’s see about numbers first.<\/li>\n
    2. It starts with Grafana Alloy<\/code>, a Loki log-collecting agent that collects logs from the \/var\/log\/pods<\/code> directory.<\/li>\n
    3. The collected logs are sent to Loki-gateway<\/code>, which routes them to the Loki pod<\/code>.<\/li>\n
    4. In this Loki setup, log ingestion<\/code>, compression<\/code>, and storage<\/code> happen inside the same pod. Once it is done, it stores the compressed log chunks in the PV<\/code>.<\/li>\n
    5. The letter flow starts with a user<\/code> query the logs through the Grafana dashboard<\/code>.<\/li>\n
    6. Grafana<\/code> Sends the query requests to the Loki gateway<\/code> and it routes the request to the Loki pod.<\/li>\n
    7. After getting the query request, the Loki pod sends back the correct logs based on the query.<\/li>\n
    8. The query results are then stored in the result cache<\/code>, and the recently accessed chunk info is cached in the chunk cache<\/code>.<\/li>\n
    9. And there is canary<\/code>, which sends dummy logs to Loki pod to check its health.<\/li>\n<\/ol>\n

      Setup Prerequisites<\/h2>\n

      The following are the prerequisites for this setup.<\/p>\n

        \n
      1. Kubernetes Cluster<\/a><\/li>\n
      2. Kubectl<\/li>\n
      3. Helm installed<\/a> in your system<\/li>\n<\/ol>\n

        Steps to Setup Grafana Loki on Kubernetes<\/h2>\n

        If you are ready with the prerequisites, follow the steps below to set up Loki on Kubernetes using Helm.<\/p>\n

        Step 1: Add Grafana repo<\/h3>\n

        Add a Grafana repo to your system to download the helm chart and set up Loki. Run the following command to add the repo<\/p>\n

        helm repo add grafana https:\/\/grafana.github.io\/helm-charts<\/code><\/pre>\n
        Once the repo is added, run the following command to update the repo to make sure the repo is up-to-date<\/p>\n
        helm repo update<\/code><\/pre>\n
        Now, list every repo with the word Loki using the command<\/p>\n
        helm search repo loki<\/code><\/pre>\n
        You will get the list of repos with the word Loki<\/code> as shown below<\/p>\n
        \"list<\/figure>\n
        From these repos, we are going to use the grafana\/loki<\/code> Helm chart repo to deploy Loki.<\/p>\n
        Step 2: Customize the Loki Helm Values file<\/h3>\n
        Before deploying, we need to update some configurations in the helm chart<\/a>.<\/p>\n
        Use the below command to save the default values of the helm chart in a YAML file.<\/p>\n
        helm show values grafana\/loki > loki.yaml<\/code><\/pre>\n
        You can also use the custom values file that we created below.<\/p>\n
        loki:\n  commonConfig:\n    replication_factor: 1\n  storage:\n    type: 'filesystem'\n    bucketNames:\n      chunks: chunks\n      ruler: ruler\n      admin: admin\n  schemaConfig:\n    configs:\n      - from: \"2024-04-01\"\n        store: tsdb\n        object_store: filesystem\n        schema: v13\n        index:\n          prefix: loki_index_\n          period: 24h\n  storage_config:\n    filesystem:\n      directory: \/tmp\/loki\/chunks\n  pattern_ingester:\n    enabled: true\n  limits_config:\n    allow_structured_metadata: true\n    volume_enabled: true\n  ruler:\n    enable_api: true\n  auth_enabled: false\n\nminio:\n  enabled: false\n      \ndeploymentMode: SingleBinary\n\nsingleBinary:\n  replicas: 1\n  persistence:\n    storageClass: gp2\n    accessModes:\n      - ReadWriteOnce\n    size: 20Gi\n\n  resources:\n    requests:\n      cpu: \"1\"\n      memory: \"2Gi\"\n    limits:\n      cpu: \"2\"\n      memory: \"4Gi\"\n\nsidecar:\n  image:\n    repository: kiwigrid\/k8s-sidecar\n    tag: 1.30.0\n  resources:\n    requests:\n      cpu: 50m\n      memory: 50Mi\n    limits:\n      cpu: 100m\n      memory: 100Mi\n\nbackend:\n  replicas: 0\nread:\n  replicas: 0\nwrite:\n  replicas: 0\n\nchunksCache:\n  allocatedMemory: 500<\/code><\/pre>\n
        \n
        \u26a0\ufe0f<\/div>\n
        In this custom values file, gp2<\/strong><\/b> is the StorageClass name used. Replace it with the StorageClass present in your cluster.<\/div>\n<\/div>\n
        In this, we are using the pods filesystem<\/code> as storage for logs, which is a temporary storage.<\/p>\n
        For the production environment, configure external storage like AWS S3<\/a>, Azure Blob Storage, etc.<\/p>\n
        deploymentMode: SingleBinary\n\nsingleBinary:\n  replicas: 1\n  persistence:\n    storageClass: gp2\n    accessModes:\n      - ReadWriteOnce\n    size: 20Gi<\/code><\/pre>\n
        Note:<\/strong> By specifying the deployment mode as SingleBinary<\/code>, we are deploying Loki as monolithic<\/code>.<\/p>\n
        It is a single binary deployment, which means all its components, such as the ingester, distributor, etc, run as a single pod.<\/p><\/blockquote>\n
        The persistence volume configuration is also set with 20GB<\/code> the storage, you can modify it as per your requirements.<\/p>\n
        Then, we configured the resource limit and requested the Loki and sidecar containers.<\/p>\n
        chunksCache:\n  allocatedMemory: 1024<\/code><\/pre>\n
        We have also set the allocatedMemory for chunksCache pod to 1GB<\/code>, by default it allocates 8GB<\/code> for the pod, and it results in the chunksCache pod in the pending state with the following error.<\/p>\n
        Warning  FailedScheduling  96s (x2 over 97s)  default-scheduler  0\/2 nodes are available: 2 Insufficient memory. preemption: 0\/2 nodes are available: 2 No preemption victims found for incoming pod.<\/code><\/pre>\n
        If you have more resources on your cluster, you can remove the above chunksCache allocatedMemory block from the values file.<\/p>\n
        Since Loki is deployed as a StatefulSet<\/code>, the logs will be saved on the same persistent volume even if the Loki pod is deleted and deployed again.<\/p>\n
        Following are the images used in this Helm chart.<\/p>\n
          \n
        1. grafana\/loki<\/li>\n
        2. nginxinc\/nginx-unprivileged<\/li>\n
        3. grafana\/loki-canary<\/li>\n
        4. grafana\/loki-helm-test<\/li>\n
        5. memcached<\/li>\n
        6. prom\/memcached-exporter<\/li>\n
        7. kiwigrid\/k8s-sidecar<\/li>\n<\/ol>\n
          Note: Setting up Loki as MicroService<\/a> or Scalable<\/a> mode is not so different from setting up Monolithic Loki, you just have to change the deployment mode in values file and increase the replicas of each Loki components.<\/p><\/blockquote>\n
          Step 3: Deploy Loki<\/h3>\n
          Once the values file is set, deploy the Loki Helm chart with the values file using the following command.<\/p>\n
          helm install loki grafana\/loki -n loki --create-namespace -f loki.yaml<\/code><\/pre>\n
          This creates a namespace loki<\/code> and deploys the pods.<\/p>\n
          Once the above command completes running, run the following command to check if the Loki pods are deployed.<\/p>\n
          kubectl get po -n loki<\/code><\/pre>\n
          You will get the following output.<\/p>\n
          \"pods<\/figure>\n
          If every pod is running, move on to the next step.<\/p>\n
          Step4: Customize Grafana Alloy Helm Values file<\/h3>\n
          The next step is to deploy Alloy<\/code>, which is a log collecting agent for Loki.<\/p>\n
          Download the Alloy values file using the following command, if you need to make any changes.<\/p>\n
          helm show values grafana\/alloy > alloy.yaml<\/code><\/pre>\n
          You can also use our custom values file.<\/p>\n
          alloy:\n  configMap:\n    content: |-\n      logging {\n        level = \"debug\"\n        format = \"logfmt\"\n      }\n      discovery.kubernetes \"pods\" {\n        role = \"pod\"\n      }\n      discovery.relabel \"pods\" {\n        targets = discovery.kubernetes.pods.targets\n\n        rule {\n          source_labels = [\"__meta_kubernetes_namespace\"]\n          target_label = \"namespace\"\n          action = \"replace\"\n        }\n\n        rule {\n          source_labels = [\"__meta_kubernetes_pod_label_app_kubernetes_io_name\"]\n          target_label = \"app\"\n          action = \"replace\"\n        }\n\n        rule {\n          source_labels = [\"__meta_kubernetes_pod_container_name\"]\n          target_label = \"container\"\n          action = \"replace\"\n        }\n\n        rule {\n          source_labels = [\"__meta_kubernetes_pod_name\"]\n          target_label = \"pod\"\n          action = \"replace\"\n        }\n      }\n      loki.source.kubernetes \"pods\" {\n        targets    = discovery.relabel.pods.output\n        forward_to = [loki.process.process.receiver]\n      }\n      loki.process \"process\" {\n        forward_to = [loki.write.loki.receiver]\n\n        stage.drop {\n          older_than          = \"1h\"\n          drop_counter_reason = \"too old\"\n        }\n        stage.match { \n          selector = \"{instance=~\\\".*\\\"}\"\n          stage.json {\n            expressions = {\n              level = \"\\\"level\\\"\",\n            }\n          }\n          stage.labels {\n            values = { \n              level = \"level\",\n            }\n          }\n        }\n        stage.label_drop {\n          values = [ \"service_name\" ]\n        }\n      }\n      loki.write \"loki\" {\n        endpoint {\n          url = \"http:\/\/loki-gateway\/loki\/api\/v1\/push\"\n        }\n      }\n  mounts:\n    varlog: true\n    dockercontainers: true\n\n  resources:\n    limits:\n      cpu: 200m\n      memory: 128Mi\n    requests:\n      cpu: 100m\n      memory: 128Mi<\/code><\/pre>\n
          We have added resource request and limit for alloy pods.<\/p>\n
          loki.write \"loki\" {\n  endpoint {\n    url = \"http:\/\/loki-gateway\/loki\/api\/v1\/push\"\n  }\n}<\/code><\/pre>\n
          The above block specifies the loki gateway endpoint, where the logs must be sent.<\/p>\n
          By default the URL will be http:\/\/loki-gateway\/loki\/api\/v1\/push<\/code>.<\/p>\n
          If you disabled Loki gateway, the URL will be changed, which will be Loki\u2019s service URL http:\/\/loki:3100\/loki\/api\/v1\/push<\/code>.<\/p>\n
          This URL will only work if both Loki and Alloy are deployed in the same namespace, if both are in different namespaces, the URL will be http:\/\/loki.loki.svc.cluster.local:3100\/loki\/api\/v1\/push<\/code>.<\/p>\n
          Following are the images used in this Helm chart.<\/p>\n
            \n
          1. grafana\/alloy<\/li>\n
          2. prometheus-operator\/prometheus-config-reloader<\/li>\n<\/ol>\n
            Step 5: Install Grafana Alloy<\/h3>\n
            Once the values file is set, deploy the Alloy<\/code> application with the values file using the following command.<\/p>\n
            helm install grafana-alloy grafana\/alloy -n loki -f alloy.yaml<\/code><\/pre>\n
            Run the following command to check if the alloy pods are deployed.<\/p>\n
            kubectl get po -n loki<\/code><\/pre>\n
            You will get the following output.<\/p>\n
            \"\"<\/figure>\n
            You can see the Alloy pods are deployed as Daemonset<\/code>.<\/p>\n
            Step 6: Customize Grafana Helm Values file<\/h3>\n
            The next step is to deploy Grafana<\/code>, which we will use to query Loki and visualize it.<\/p>\n
            Download the Grafana values file using the following command, if you need to make any changes.<\/p>\n
            You can also use our custom values file.<\/p>\n
            image:\n  repository: grafana\/grafana\n  tag: latest\n  pullPolicy: IfNotPresent\n\nresources:\n  requests:\n    cpu: 100m\n    memory: 128Mi\n  limits:\n    cpu: 500m\n    memory: 512Mi\n\ndatasources:\n  datasources.yaml:\n    apiVersion: 1\n    datasources:\n    - name: Loki\n      type: loki\n      url: http:\/\/loki-gateway.loki.svc.cluster.local\n      access: proxy\n      isDefault: false\n      editable: true\n\nservice:\n  type: NodePort\n  port: 80<\/code><\/pre>\n
            You can see we have added Loki as a datasource<\/code> for Grafana, so we don’t have to configure it manually after logging in to Grafana.<\/p>\n
            Also set the Grafana service  Nodeport<\/code> to access the UI using the node IP and nodeport<\/code>.<\/p>\n
            Exposing service as NodePort is not recommended for production environment, user Load Balancers or Ingress to expose the application.<\/p><\/blockquote>\n
            Following are the images used in this Helm chart.<\/p>\n
              \n
            1. grafana\/grafana<\/li>\n
            2. bats\/bats<\/li>\n
            3. curlimages\/curl<\/li>\n
            4. library\/busybox<\/li>\n
            5. kiwigrid\/k8s-sidecar<\/li>\n<\/ol>\n
              Step 7: Install Grafana<\/h3>\n
              Once the values file is set, deploy Grafana<\/code> with the values file using the following command.<\/p>\n
              helm install grafana grafana\/grafana -n grafana --create-namespace -f g.yaml<\/code><\/pre>\n
              This command will create a namespace grafana<\/code> if not present and deploy the Grafana pod<\/code> in it.<\/p>\n
              Run the following command to check if the Grafana pod is deployed.<\/p>\n
              kubectl get po -n grafana<\/code><\/pre>\n
              You will get the following output.<\/p>\n
              \"checking<\/figure>\n
              Since we have exposed the Grafana service as NodePort, we can access the Grafana dashboard using the node IP and node port assigned to your service.<\/p>\n
              Step 8: Log in to Grafana UI<\/h3>\n
              Access your Grafana on the browser using the node IP and node port<\/code>, you will get the following window.<\/p>\n
              \"Login<\/figure>\n
              Login to Grafana using the username and password. Your default username will be admin<\/code> and to get the password run the following command<\/p>\n
              kubectl get secret --namespace grafana grafana -o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo<\/code><\/pre>\n
              This command will show your password for Grafana. Use the password to log in to Grafana.<\/p>\n
              Once you log in, you can see the home screen of Grafana, press the three lines at the top left corner, you can see the menu, then go to Connections > Data sources<\/code> as shown in the below image.<\/p>\n
              \"Grafana<\/figure>\n
              In the Data sources<\/code> you can see Loki has been configured as default.<\/p>\n
              \"Add<\/figure>\n
              Now, check if you are getting logs.<\/p>\n
              Go to Explore<\/code> by pressing the Explore<\/code> button.<\/p>\n
              To query logs select a Label<\/code> <\/strong>and Value<\/code>, Loki will collect every log in your Kubernetes cluster and label it according to container, pod<\/a>, namespace, and other objects of Kubernetes<\/a>.<\/p>\n
              \"Filtering<\/figure>\n
              After selecting a Label<\/code> and Value<\/code>, press the Live button at the top right corner to query logs.<\/p>\n
              You can also set a time to query logs. For example, if you set the time to 5s<\/code> it queries logs every 5s<\/code>.<\/p>\n
              \"Filtering<\/figure>\n
              After pressing the Live button, scroll down to view the logs from the Label<\/code> and Value<\/code> you select.<\/p>\n
              \"Filtering<\/figure>\n
              Conclusion<\/h2>\n
              I believe this helped you to set up the Loki on Kubernetes using Helm. We also learned how to query logs using Loki from the Grafana dashboard.<\/p>\n
              You can use our Configure S3 for Loki<\/a> blog to add S3 as log storage for Loki.<\/p>\n
              If you want more detailed information about Grafana setup, you can use our guide on setting up Grafana on Kubernetes.<\/a><\/p>\n
              \n
              Ngu\u1ed3n:<\/strong> How to Setup Grafana Loki on Kubernetes &amp; Query Logs \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
              Source: https:\/\/devopscube.com\/setup-grafana-loki\/<\/p>\n","protected":false},"author":1,"featured_media":577,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-576","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/576","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=576"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/576\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/577"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=576"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=576"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=576"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}