{"id":370,"date":"2026-01-02T05:37:00","date_gmt":"2026-01-02T05:37:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=370"},"modified":"2026-01-02T05:37:00","modified_gmt":"2026-01-02T05:37:00","slug":"setup-efk-stack-on-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=370","title":{"rendered":"How to Setup EFK Stack on Kubernetes: Step by Step Guides"},"content":{"rendered":"

In this Kubernetes tutorial<\/a>, you will learn how to set up the EFK stack on a Kubernetes<\/strong> cluster for log streaming, log analysis, and log monitoring.<\/p>\n

Check out part 1 in this Kubernetes logging series, where we have covered Kubernetes logging fundamentals and patterns<\/a> for beginners.<\/p>\n

When running multiple applications and services on a Kubernetes cluster, it makes more sense to stream all of your application and Kubernetes cluster logs to one centralized logging infrastructure<\/strong> for easy log analysis.<\/p>\n

This beginner’s guide walks you through the key technical aspects of Kubernetes logging using the EFK stack.<\/p>\n

What is EFK Stack?<\/h2>\n

EFK stands for Elasticsearch, Fluent Bit, and Kibana<\/strong>. EFK is a popular and the best open-source choice for Kubernetes log aggregation and analysis.<\/p>\n

    \n
  1. Elasticsearch<\/strong> is a distributed, scalable search engine commonly used to search large volumes of log data. It is a NoSQL database based on the Apache Lucene<\/a> search engine (search library from Apache). Its primary work is to store logs and retrieve logs from Fluentbit.<\/li>\n
  2. Fluent Bit<\/strong> is a log shipper. It is a small but fast open-source tool that collects and forwards logs. It collects data from various sources, processes it, and then sends it to the location where the logs are stored and analyzed. It often sends data to services such as Elasticsearch, CloudWatch, Splunk, and Stackdriver.<\/li>\n
  3. Kibana<\/strong> is a UI tool for querying, data visualization, and dashboards. It is a query engine that allows you to explore your log data through a web interface, build visualizations for event logs, and query-specific to filter information for detecting issues. You can virtually build any type of dashboards using Kibana. Kibana Query Language (KQL<\/strong>) is used for querying Elasticsearch data. <\/li>\n<\/ol>\n

    Also, Elasticsearch helps solve the problem of separating huge amounts of unstructured data and is in use by many organizations. Elasticsearch is commonly deployed alongside Kibana.<\/p>\n

    Note:<\/strong> When it comes to Kubernetes, Fluent Bit is the best choice because it is a lightweight, fast tool. However, if you have a complex log-processing<\/strong> scenario, you can consider Fluentd<\/strong>.<\/p><\/blockquote>\n

    Before we move to the setup, let us check the workflow of the EFK stack.<\/p>\n

    EFK Stack Architecture<\/h2>\n

    The following diagram shows the high-level architecture of the EFK stack<\/strong> we will build.<\/p>\n

    \"EFK<\/figure>\n

    EKF components get deployed as follows,<\/p>\n

      \n
    1. Fluentbit:- <\/strong>Deployed as a Daemonset, so an agent pod will run on each node to collect the logs from workloads.<\/li>\n
    2. Elasticsearch<\/strong>:- Deployed as a statefulset as it holds the log data. We also expose the service endpoint for Fluentbit and Kibana to connect to it.<\/li>\n
    3. Kibana<\/strong>:- Deployed as a deployment and connects to the Elasticsearch service endpoint.<\/li>\n<\/ol>\n

      Prerequisites <\/h2>\n

      Before beginning the installation, ensure you have the following <\/p>\n

        \n
      1. A Kubernetes cluster<\/li>\n
      2. Helm [Local Workstation]<\/li>\n
      3. Kubectl [Local Workstation]<\/li>\n<\/ol>\n

        Once you have a Kubernetes cluster, we can start the setup.<\/p>\n

        Set up ECK Operator on Kubernetes<\/h2>\n

        The first step is setting up Elasticsearch<\/strong> and Kibana<\/strong> on the cluster. <\/p>\n

        Here, we use the ECK (Elastic Cloud on Kubernetes) Operator<\/strong> instead of the standard installation method to easily manage resources such as Elasticsearch and Kibana via Custom Resource Definitions.<\/p>\n

        To install the ECK Operator, we even use the Helm chart.<\/p>\n

        Step 1: Add Elastic Helm Repositories <\/h3>\n

        The first step is to add the Elastic Helm repositories to our local machine.<\/p>\n

        helm repo add elastic https:\/\/helm.elastic.co\n\nhelm repo update<\/code><\/pre>\n
        Once we have added and updated all the Elastic charts, we can begin the installation.<\/p>\n
        Step 2: Install ECK Operator<\/h3>\n
        To install the Elasticsearch operator, we need to choose the elastic\/eck-operator<\/code> chart.<\/p>\n
        \"\"<\/figure>\n
        In production, we do not deploy the Helm chart directly. Instead, we will make the necessary customizations based on the project requirements and then deploy.<\/p>\n
        To do that, first download the chart or the original values file that contains the modifiable settings.<\/p>\n
        To download the ECK operator Helm chart, run the following commands.<\/p>\n
        helm pull elastic\/eck-operator --untar<\/code><\/pre>\n
        The following is the directory structure of the ECK operator Helm chart.<\/p>\n
        eck-operator\n\u251c\u2500\u2500 Chart.lock\n\u251c\u2500\u2500 Chart.yaml\n\u251c\u2500\u2500 LICENSE\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 charts\n\u2502   \u2514\u2500\u2500 eck-operator-crds\n\u2502       \u251c\u2500\u2500 Chart.yaml\n\u2502       \u251c\u2500\u2500 README.md\n\u2502       \u251c\u2500\u2500 templates\n\u2502       \u2502   \u251c\u2500\u2500 NOTES.txt\n\u2502       \u2502   \u251c\u2500\u2500 _helpers.tpl\n\u2502       \u2502   \u2514\u2500\u2500 all-crds.yaml\n\u2502       \u2514\u2500\u2500 values.yaml\n\u251c\u2500\u2500 profile-disable-automounting-api.yaml\n\u251c\u2500\u2500 profile-global.yaml\n\u251c\u2500\u2500 profile-istio.yaml\n\u251c\u2500\u2500 profile-restricted.yaml\n\u251c\u2500\u2500 profile-soft-multi-tenancy.yaml\n\u251c\u2500\u2500 templates\n\u2502   \u251c\u2500\u2500 NOTES.txt\n\u2502   \u251c\u2500\u2500 _helpers.tpl\n\u2502   \u251c\u2500\u2500 cluster-roles.yaml\n\u2502   \u251c\u2500\u2500 configmap.yaml\n\u2502   \u251c\u2500\u2500 managed-namespaces.yaml\n\u2502   \u251c\u2500\u2500 managed-ns-network-policy.yaml\n\u2502   \u251c\u2500\u2500 metrics-service.yaml\n\u2502   \u251c\u2500\u2500 operator-namespace.yaml\n\u2502   \u251c\u2500\u2500 operator-network-policy.yaml\n\u2502   \u251c\u2500\u2500 pdb.yaml\n\u2502   \u251c\u2500\u2500 podMonitor.yaml\n\u2502   \u251c\u2500\u2500 role-bindings.yaml\n\u2502   \u251c\u2500\u2500 service-account.yaml\n\u2502   \u251c\u2500\u2500 service-monitor.yaml\n\u2502   \u251c\u2500\u2500 statefulset.yaml\n\u2502   \u251c\u2500\u2500 validate-chart.yaml\n\u2502   \u2514\u2500\u2500 webhook.yaml\n\u2514\u2500\u2500 values.yaml<\/code><\/pre>\n
        Here, you can see the original values.yaml<\/code> file, but we will create a separate one based on the requirements.  <\/p>\n
        Copy and paste the following contents into your terminal to create a custom values file.<\/p>\n
        cat << EOF > dev-es-values.yaml\ninstallCRDs: true\n\nreplicaCount: 1\n\nresources:\n  limits:\n    cpu: 500m\n    memory: 512Mi\n  requests:\n    cpu: 100m\n    memory: 200Mi\n\nwebhook:\n  enabled: true\n  failurePolicy: Ignore\n  manageCerts: true\n\nconfig:\n  logVerbosity: \"0\"\n  metrics:\n    port: 9090 \nEOF<\/code><\/pre>\n
        Once the custom values file is ready, we can install the ECK operator using the following command.<\/p>\n
        helm upgrade --install eck-operator elastic\/eck-operator \\\n  -n es-operator \\\n  --create-namespace \\\n  -f dev-es-values.yaml<\/code><\/pre>\n
        Once installation is complete, we need to ensure the Elasticsearch operator pod is running without issues.<\/p>\n
        Step 3: Validate the Elasticsearch Operator <\/h3>\n
        To ensure the Operator pod runs without issues, run the following command.<\/p>\n
        $ kubectl -n es-operator get po\n\nNAME                 READY   STATUS    RESTARTS      AGE\nelastic-operator-0   1\/1     Running   1 (13m ago)   14m<\/code><\/pre>\n
        Since this is an operator, it has custom resources to manage the Elasticsearch pods.<\/p>\n
        To list all the Custom Resource Definitions of the Elasticsearch, use the following command.<\/p>\n
        $ kubectl get crds | grep elastic\n\nagents.agent.k8s.elastic.co                            2025-12-29T05:17:50Z\napmservers.apm.k8s.elastic.co                          2025-12-29T05:17:50Z\nbeats.beat.k8s.elastic.co                              2025-12-29T05:17:50Z\nelasticmapsservers.maps.k8s.elastic.co                 2025-12-29T05:17:50Z\nelasticsearchautoscalers.autoscaling.k8s.elastic.co    2025-12-29T05:17:50Z\nelasticsearches.elasticsearch.k8s.elastic.co           2025-12-29T05:17:50Z\nenterprisesearches.enterprisesearch.k8s.elastic.co     2025-12-29T05:17:50Z\nkibanas.kibana.k8s.elastic.co                          2025-12-29T05:17:50Z\nlogstashes.logstash.k8s.elastic.co                     2025-12-29T05:17:50Z\nstackconfigpolicies.stackconfigpolicy.k8s.elastic.co   2025-12-29T05:17:50Z<\/code><\/pre>\n
        From this list, we need to use the enterprisesearches.enterprisesearch.k8s.elastic.co<\/code> Custom Resource to create an Elasticsearch deployment.<\/p>\n
        Step 4: Deploy Elasticsearch<\/h3>\n
        Our Elastic operator is ready, so we can create a manifest to create Elasticsearch.<\/p>\n
        Use the following contents to create an Elasticsearch deployment. Since this is a demo, we use an optimized Elasticsearch manifest with a single node and minimal resources.<\/p>\n
        cat <<EOF > elasticsearch.yaml\napiVersion: elasticsearch.k8s.elastic.co\/v1\nkind: Elasticsearch\nmetadata:\n  name: quickstart\n  namespace: es-operator\nspec:\n  version: 8.12.2\n  nodeSets:\n  - name: default\n    count: 1 # Optimized to 1 node for testing (production usually uses 3)\n    config:\n      node.roles: [\"master\", \"data\", \"ingest\"]\n      node.store.allow_mmap: false\n      xpack.ml.enabled: false \n      xpack.security.enabled: true\n      xpack.watcher.enabled: false \n    podTemplate:\n      spec:\n        containers:\n        - name: elasticsearch\n          resources:\n            requests:\n              memory: 1Gi\n              cpu: 200m \n            limits:\n              memory: 1.5Gi \n              cpu: 1000m\n          env:\n          - name: ES_JAVA_OPTS\n            value: \"-Xms512m -Xmx512m\" \n    volumeClaimTemplates:\n    - metadata:\n        name: elasticsearch-data\n      spec:\n        accessModes:\n        - ReadWriteOnce\n        resources:\n          requests:\n            storage: 2Gi\nEOF<\/code><\/pre>\n
        \n
        \ud83d\udca1<\/div>\n
        For production, use minimum 3 nodes for the high availability as well as the appropriate resorces based on your project requirements.<\/div>\n<\/div>\n
        To deploy this, use the following command.<\/p>\n
        kubectl deploy -f elasticsearch.yaml<\/code><\/pre>\n
        Once the Elasticsearch is deployed, we need to check whether the ElasticSearch pods are running without any issues.<\/p>\n
        Step 5: Validate Elasticsearch Deployment<\/h3>\n
        Once the deployment is completed, use the following command to list the Elasticsearch pods<\/p>\n
        $ kubectl get pods -n es-operator -l common.k8s.elastic.co\/type=elasticsearch\n\nNAME                      READY   STATUS    RESTARTS   AGE\nquickstart-es-default-0   1\/1     Running   0          4m2s<\/code><\/pre>\n
        Output show that the Pod is running without any issues.<\/p>\n
        Since ElasticSearch is a custom resource, we can use that to list and see the status as well.<\/p>\n
        $ kubectl -n es-operator get elasticsearch\n\nNAME         HEALTH   NODES   VERSION   PHASE   AGE\nquickstart   green    1       8.12.2    Ready   2m20s<\/code><\/pre>\n
        Elasticsearch deployment is completed, so we need to install the Kibana dashboard to visualize the logs.<\/p>\n
        Step 6: Deploy Kibana<\/h3>\n
        As we deployed the Elasticsearch, we can use one of the Operator’s custom resource called kibanas.kibana.k8s.elastic.co<\/code> to deploy Kibana.<\/p>\n
        Use the following Kibana manifest to deploy the Kibana Pods.<\/p>\n
        cat << EOF > kibana.yaml\napiVersion: kibana.k8s.elastic.co\/v1\nkind: Kibana\nmetadata:\n  name: quickstart\n  namespace: es-operator\nspec:\n  version: 8.12.2\n  count: 1\n  elasticsearchRef:\n    name: quickstart\n  config:\n    server.publicBaseUrl: \"http:\/\/localhost:5601\"\n  podTemplate:\n    spec:\n      containers:\n      - name: kibana\n        resources:\n          requests:\n            memory: 512Mi\n            cpu: 200m\n          limits:\n            memory: 1Gi\n            cpu: 1000m\nEOF<\/code><\/pre>\n
        To deploy Kibana, use the following command.<\/p>\n
        kubectl apply -f kibana.yaml<\/code><\/pre>\n
        Once the Kibana installation is completed, we need to ensure that the Pods are running.<\/p>\n
        Step 7: Validating Kibana Deployment<\/h3>\n
        Once the deployment is completed, use the following command to list the Kibana pods and services.<\/p>\n
        $ kubectl get pods,svc -n es-operator -l common.k8s.elastic.co\/type=kibana\n\nNAME                                 READY   STATUS    RESTARTS   AGE\npod\/quickstart-kb-7f694bf8db-pxlrs   1\/1     Running   0          11m\n\nNAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE\nservice\/quickstart-kb-http   ClusterIP   10.96.138.121   <none>        5601\/TCP   11m<\/code><\/pre>\n
        Output shows that the Kibana pod are running without any issues.<\/p>\n
        Kibana also a custom resource so we can list in that way as well.<\/p>\n
        $ kubectl -n es-operator get kibanas \n\nNAME         HEALTH   NODES   VERSION   AGE\nquickstart   green    1       8.12.2    12m<\/code><\/pre>\n
        Now, both the Elasticsearch and the Kibana is ready. Since, Kibana is a dashboard, we need credentials to login.<\/p>\n
        Step 8: Generate Kibana Credentials <\/h3>\n
        The default user name of Kibana is elastic<\/code> and to generate password, use the following command.<\/p>\n
        $ kubectl get secret quickstart-es-elastic-user -n es-operator -o go-template='{{.data.elastic | base64decode}}'\n\nYf7CCX9C4Gz9lNpnVrk8zh8Q%                      <\/code><\/pre>\n
        Ignore the %<\/code> when copy the password.<\/p>\n
        Step 9: Port Forward Kibana Service<\/h3>\n
        To access the Kibana dashboard from our local machine, we need to perform port forwarding because Kibana will be deployed as a ClusterIP service.<\/p>\n
        kubectl port-forward service\/quickstart-kb-http -n es-operator 5601\n<\/code><\/pre>\n
        \n
        \ud83d\udca1<\/div>\n
        For production, deploy Kibana with a load balancer so that the load will be distributed and can be accessed with a domain name.<\/div>\n<\/div>\n
        Now, we are ready for the Kibana login.<\/p>\n
        Step 10: Access Kibana UI<\/h3>\n
        To access the UI, open any of the web browser and paste the URL https:\/\/localhost:5601<\/code><\/a><\/p>\n
        Once the login page is open, paste the default username and the password to enter into.<\/p>\n
        \"The<\/figure>\n
        Once the login is completed, you will see a following page.<\/p>\n
        \"The<\/figure>\n
        Now, both the Elasticsearch and Kibana deployments ready, so we need to deploy the Fluentbit to collect logs.<\/p>\n
        Setup Fluent Operator on Kubernetes<\/h2>\n
        As like the Elastic operator, we are using an Operator for the FluentBit.<\/p>\n
        Fluentbit is a light weight logging agent that collects logs from the workloads of each nodes.<\/p>\n
        The collected logs will be send to the Elasticsearch for query and visualization. Elasticsearch needs to store the logs so is deployed as a StatefulSets and uses Persistent volumes as storage.<\/p>\n
        To begin the installation, we need to add the Fluent Helm repository.<\/p>\n
        Step 1: Add Fluent Helm repository<\/h3>\n
        To add the Fluent Helm repo on our local machine, use the following command.<\/p>\n
        helm repo add fluent https:\/\/fluent.github.io\/helm-charts\n\nhelm repo update<\/code><\/pre>\n
        To install the Fluent operator, we use the fluent\/fluent-operator<\/code> chart from the list.<\/p>\n
        \"\"<\/figure>\n
        Once all the charts are updated, we can create custom values.<\/p>\n
        Step 2: Create Custom Values File<\/h3>\n
        Before create custom values file, we can pull the chart on the local machine.<\/p>\n
        To pull the chart, use the following command.<\/p>\n
        helm pull fluent\/fluent-operator --untar<\/code><\/pre>\n
        Once the chart been download, you can see the following structure.<\/p>\n
        fluent-operator\n\u251c\u2500\u2500 Chart.lock\n\u251c\u2500\u2500 Chart.yaml\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 charts\n\u2502   \u251c\u2500\u2500 fluent-bit-crds\n\u2502   \u2502   \u251c\u2500\u2500 Chart.yaml\n\u2502   \u2502   \u251c\u2500\u2500 crds\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clusterfilters.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clusterfluentbitconfigs.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clusterinputs.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clustermultilineparsers.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clusteroutputs.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_clusterparsers.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_collectors.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_filters.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_fluentbitconfigs.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_fluentbits.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_multilineparsers.yaml\n\u2502   \u2502   \u2502   \u251c\u2500\u2500 fluentbit.fluent.io_outputs.yaml\n\u2502   \u2502   \u2502   \u2514\u2500\u2500 fluentbit.fluent.io_parsers.yaml\n\u2502   \u2502   \u2514\u2500\u2500 values.yaml\n\u2502   \u2514\u2500\u2500 fluentd-crds\n\u2502       \u251c\u2500\u2500 Chart.yaml\n\u2502       \u251c\u2500\u2500 crds\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_clusterfilters.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_clusterfluentdconfigs.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_clusterinputs.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_clusteroutputs.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_filters.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_fluentdconfigs.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_fluentds.yaml\n\u2502       \u2502   \u251c\u2500\u2500 fluentd.fluent.io_inputs.yaml\n\u2502       \u2502   \u2514\u2500\u2500 fluentd.fluent.io_outputs.yaml\n\u2502       \u2514\u2500\u2500 values.yaml\n\u251c\u2500\u2500 templates\n\u2502   \u251c\u2500\u2500 NOTES.txt\n\u2502   \u251c\u2500\u2500 _helpers.tpl\n\u2502   \u251c\u2500\u2500 fluent-operator-clusterRole.yaml\n\u2502   \u251c\u2500\u2500 fluent-operator-clusterRoleBinding.yaml\n\u2502   \u251c\u2500\u2500 fluent-operator-deployment.yaml\n\u2502   \u251c\u2500\u2500 fluent-operator-service.yaml\n\u2502   \u251c\u2500\u2500 fluent-operator-servicemonitor.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterfilter-containerd.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterfilter-kubernetes.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterfilter-multiline.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterfilter-systemd.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterinput-metrics.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterinput-nodeExporterMetrics.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterinput-systemd.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-clusterinput-tail.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-containerd-config.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-fluentBit.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-fluentbit-edge.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-input-node-exporter-metrics-edge.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-input-prometheus-scrape-metrics-edge.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-lua-config.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-multilineParser-javaMultiline.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-elasticsearch.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-forward.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-kafka.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-loki.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-opensearch.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-opentelemetry.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-prometheus-exporter.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-prometheus-remote-write-edge.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-stackdriver.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-output-stdout.yaml\n\u2502   \u251c\u2500\u2500 fluentbit-servicemonitor.yaml\n\u2502   \u251c\u2500\u2500 fluentbitconfig-fluentBitConfig.yaml\n\u2502   \u251c\u2500\u2500 fluentbitconfig-fluentbitconfig-edge.yaml\n\u2502   \u251c\u2500\u2500 fluentd-clusterfluentdconfig.yaml\n\u2502   \u251c\u2500\u2500 fluentd-filter-kafka.yaml\n\u2502   \u251c\u2500\u2500 fluentd-fluentd.yaml\n\u2502   \u251c\u2500\u2500 fluentd-output-elasticsearch.yaml\n\u2502   \u251c\u2500\u2500 fluentd-output-kafka.yaml\n\u2502   \u251c\u2500\u2500 fluentd-output-opensearch.yaml\n\u2502   \u2514\u2500\u2500 serviceaccount.yaml\n\u2514\u2500\u2500 values.yaml<\/code><\/pre>\n
        You can choose the required settings from the original values file and create a new custom one using the following contents.<\/p>\n
        cat << EOF > dev-fluent-values.yaml\nKubernetes: true\n\noperator:\n  enable: true\n  resources:\n    limits:\n      cpu: 50m\n      memory: 50Mi\n    requests:\n      cpu: 10m\n      memory: 20Mi\n\nfluentbit:\n  enable: true\n  resources:\n    limits:\n      cpu: 100m\n      memory: 100Mi\n    requests:\n      cpu: 10m\n      memory: 25Mi\n\nfluentd:\n  enable: false\nEOF<\/code><\/pre>\n
        Once the custom values file is ready, we can install the Fluent Operator as well as the Pods.<\/p>\n
        Step 3: Install Fluent Operator and Pods<\/h3>\n
        Use the following command to install the Fluent operator and Fluentbit pods.<\/p>\n
        helm upgrade --install fluent-operator fluent\/fluent-operator -f dev-fluent-values.yaml -n fluent --create-namespace<\/code><\/pre>\n
        Once the installation is completed, check whether the Controller and the Fluentbit agents are running.<\/p>\n
        Step 4: Validate the Fluent Operator and Agents<\/h3>\n
        To see the status of the Fluent operator and Fluentbit Pods, use the following command.<\/p>\n
        $ kubectl -n fluent get po,svc\n\nNAME                                   READY   STATUS    RESTARTS      AGE\npod\/fluent-bit-jtkdf                   1\/1     Running   1 (21m ago)   18h\npod\/fluent-bit-n6dtg                   1\/1     Running   1 (21m ago)   18h\npod\/fluent-bit-x2ndg                   1\/1     Running   1 (21m ago)   18h\npod\/fluent-operator-74f784fbb9-nf4gd   1\/1     Running   1 (21m ago)   18h\n\nNAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE\nservice\/fluent-bit        ClusterIP   10.96.151.168   <none>        2020\/TCP   18h\nservice\/fluent-operator   ClusterIP   10.96.71.8      <none>        8080\/TCP   18h<\/code><\/pre>\n
        To see the Custom Resource Definitions of the Fluent, use the following command.<\/p>\n
        $ kubectl get crd | grep fluent\n\nclusterfilters.fluentbit.fluent.io                     2025-12-29T10:32:05Z\nclusterfilters.fluentd.fluent.io                       2025-12-29T10:32:08Z\nclusterfluentbitconfigs.fluentbit.fluent.io            2025-12-29T10:32:05Z\nclusterfluentdconfigs.fluentd.fluent.io                2025-12-29T10:32:08Z\nclusterinputs.fluentbit.fluent.io                      2025-12-29T10:32:05Z\nclusterinputs.fluentd.fluent.io                        2025-12-29T10:32:08Z\nclustermultilineparsers.fluentbit.fluent.io            2025-12-29T10:32:05Z\nclusteroutputs.fluentbit.fluent.io                     2025-12-29T10:32:05Z\nclusteroutputs.fluentd.fluent.io                       2025-12-29T10:32:08Z\nclusterparsers.fluentbit.fluent.io                     2025-12-29T10:32:05Z\ncollectors.fluentbit.fluent.io                         2025-12-29T10:32:06Z\nfilters.fluentbit.fluent.io                            2025-12-29T10:32:06Z\nfilters.fluentd.fluent.io                              2025-12-29T10:32:08Z\nfluentbitconfigs.fluentbit.fluent.io                   2025-12-29T10:32:06Z\nfluentbits.fluentbit.fluent.io                         2025-12-29T10:32:06Z\nfluentdconfigs.fluentd.fluent.io                       2025-12-29T10:32:08Z\nfluentds.fluentd.fluent.io                             2025-12-29T10:32:09Z\ninputs.fluentd.fluent.io                               2025-12-29T10:32:09Z\nmultilineparsers.fluentbit.fluent.io                   2025-12-29T10:32:07Z\noutputs.fluentbit.fluent.io                            2025-12-29T10:32:07Z\noutputs.fluentd.fluent.io                              2025-12-29T10:32:09Z\nparsers.fluentbit.fluent.io                            2025-12-29T10:32:07Z<\/code><\/pre>\n
        Now, the FluentBit pods are ready to fetch the logs from the workloads.<\/p>\n
        Step 5: Create Cluster Output Custom Resource<\/h3>\n
        Now, we need to create cluster output custom resource to see the log output on the Fluentbit pods.<\/p>\n
        cat <<EOF > fluent-stdout-output.yaml\napiVersion: fluentbit.fluent.io\/v1alpha2\nkind: ClusterOutput\nmetadata:\n  name: stdout\n  labels:\n    fluentbit.fluent.io\/enabled: \"true\"\nspec:\n  matchRegex: (?:kube|service)\\.(?:var\\.log\\.containers\\.counter.*)\n  stdout:\n    format: json\nEOF<\/code><\/pre>\n
        Once the manifest is created, use the following command to deploy it.<\/p>\n
        kubectl apply -f fluent-stdout-output.yaml<\/code><\/pre>\n
        Once created the custom resource, we can list that using the following command.<\/p>\n
        $ kubectl get clusteroutputs\n\nNAME     AGE\nstdout   3d4h<\/code><\/pre>\n
        This ensures that the Custom Resource is created. To test the log collection, we will deploy a demo application.<\/p>\n
        Verify Fluentd Setup<\/h2>\n
        To verify the Fluentd installation, let us start a pod that continuously generates logs.<\/p>\n
        cat << EOF | kubectl apply -f -\napiVersion: v1\nkind: Pod\nmetadata:\n  name: counter\nspec:\n  containers:\n  - name: count\n    image: busybox\n    args:\n    - \/bin\/sh\n    - -c\n    - |\n      i=0\n      while true; do\n        echo \"Thanks for visiting devopscube! $i\"\n        i=$((i+1))\n        sleep 1\n      done\nEOF<\/code><\/pre>\n
        Once it is deployed, the pod keeps on generate logs.<\/p>\n
        Step 1: See Logs From Pod<\/h3>\n
        Once we deployed the application, we can check the logs of the same pod to ensure that it generating the logs.<\/p>\n
        $ kubectl logs counter \n\nThanks for visiting devopscube! \nThanks for visiting devopscube! \nThanks for visiting devopscube! \nThanks for visiting devopscube! <\/code><\/pre>\n
        This ensures that the pod is properly generating the logs, so let us check the Fluentbit pods are collecting are not.<\/p>\n
        Step 2: See Logs Through Fluentbit Pods<\/h3>\n
        All the workload Pod’s logs will be stored in the same node on path \/var\/log\/containers<\/code>.<\/p>\n
        Fluentbit pods will be run as a DaemonSet, so each node has a Fluentbit pod. The agent from the same node will collect those logs from the static location.<\/p>\n
        To check the logs of the Fluentbit pods, use the following command.<\/p>\n
        kubectl logs -n fluent -l app.kubernetes.io\/name=fluent-bit --tail=200 | grep \"counter\"\n\n[{\"date\":1767072959.964684,\"log\":\"2025-12-30T05:35:59.964536754Z stdout F Thanks for visiting devopscube! \",\"kubernetes\":{\"pod_name\":\"counter\",\"namespace_name\":\"default\",\"pod_ip\":\"10.244.1.6\",\"container_name\":\"count\",\"docker_id\":\"ec9dcdd990faf638f95a1ca724efb7ecc7da7f15ed34972eabc0c5fa2b015eee\",\"container_image\":\"docker.io\/library\/busybox:latest\"}}]\n[{\"date\":1767072960.966736,\"log\":\"2025-12-30T05:36:00.966605613Z stdout F Thanks for visiting devopscube! \",\"kubernetes\":{\"pod_name\":\"counter\",\"namespace_name\":\"default\",\"pod_ip\":\"10.244.1.6\",\"container_name\":\"count\",\"docker_id\":\"ec9dcdd990faf638f95a1ca724efb7ecc7da7f15ed34972eabc0c5fa2b015eee\",\"container_image\":\"docker.io\/library\/busybox:latest\"}}]<\/code><\/pre>\n
        Here, you can see the same logs generated on the demo application.<\/p>\n
        Step 3: Configure Elasticsearch for Kibana<\/h3>\n
        Copy the Elasticsearch credentials into the Fluent Bit namespace for the Fluentbit to access the Elasticsearch.<\/p>\n
        kubectl get secret quickstart-es-elastic-user -n es-operator -o yaml | sed 's\/namespace: es-operator\/namespace: fluent\/' | kubectl apply -f -<\/code><\/pre>\n
        Create a secret for username as elastic<\/code><\/p>\n
        kubectl create secret generic elastic-username --from-literal=username=elastic -n fluent<\/code><\/pre>\n
        Now, we need to create or modify the Cluster Output custom resource with the credentials.<\/p>\n
        This tells that how the how the FluentBit can authenticate with the Elasticsearch.<\/p>\n
        cat <<EOF > fluent-es-output.yaml\napiVersion: fluentbit.fluent.io\/v1alpha2\nkind: ClusterOutput\nmetadata:\n  name: es\n  labels:\n    fluentbit.fluent.io\/enabled: \"true\"\nspec:\n  matchRegex: (?:kube|service)\\.(?:var\\.log\\.containers\\.counter.*)\n  es:\n    host: quickstart-es-http.es-operator.svc\n    port: 9200\n    httpUser:\n      valueFrom:\n        secretKeyRef:\n          name: elastic-username\n          key: username\n    httpPassword:\n      valueFrom:\n        secretKeyRef:\n          name: quickstart-es-elastic-user\n          key: elastic\n    tls:\n      verify: false\n    index: fluent-bit\n    suppressTypeName: \"On\"\nEOF<\/code><\/pre>\n
        To deploy this, use the following command.<\/p>\n
        kubectl apply -f fluent-es-output.yaml<\/code><\/pre>\n
        Once the resource is deployed, we need to restart the fluent pods.<\/p>\n
        kubectl delete pod -n fluent -l app.kubernetes.io\/name=fluent-operator\nkubectl delete pod -n fluent -l app.kubernetes.io\/name=fluent-bit<\/code><\/pre>\n
        To verify the Elasticsearch index, we need the credentials first.<\/p>\n
        export ES_PASSWORD=$(kubectl get secret quickstart-es-elastic-user -n es-operator -o jsonpath='{.data.elastic}' | base64 --decode)<\/code><\/pre>\n
        To check indices, use the following command.<\/p>\n
        kubectl run curl-test --rm -it --restart=Never --image=curlimages\/curl -- \\\n  -k -u elastic:$ES_PASSWORD https:\/\/quickstart-es-http.es-operator.svc:9200\/_cat\/indices?v<\/code><\/pre>\n
        Now, the index is ready, so we can configure the Kibana dashboard to see the logs.<\/p>\n
        Step 4: Configure Kibana<\/h3>\n
        Create a secret in the fluent namespace of the Elastic secret <\/p>\n
        kubectl get secret quickstart-es-elastic-user -n es-operator -o json | \\\n  jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp)' | \\\n  kubectl apply -n fluent -f -<\/code><\/pre>\n
        To see the logs in Kibana dashboard, you need to create a data view.<\/p>\n
        Navigatew to Analytics -> Discover and click “Create data view” to create a new one.<\/p>\n
        \"\"<\/figure>\n
        Then give a name for the dataview and select the Index patten and save the view.<\/p>\n
        \"\"<\/figure>\n
        Once the dataview is created, you will be able to see the logs for the workload pods.<\/p>\n
        \"\"<\/figure>\n
        Here, you can see the same logs that we get in the Fluentbit pods.<\/p>\n
        \"\"<\/figure>\n
        This is how we can easily track the logs of our Kubernetes workload and is useful to identify the issues using the logs.<\/p>\n
        Kubernetes EFK Best Practices<\/h2>\n
          \n
        1. Elasticsearch uses heap memory extensively<\/strong> for filtering and caching for better query performances, so ample memory should be available for Elasticsearch.\n
          Giving more than half of total memory to Elasticsearch could also leave too less memory for OS functions which could in turn hamper Elasticsearch’s capabilities.<\/p>\n
          So be mindful of this! A 40-50% of total heap space to Elasticsearch is good enough<\/strong>.<\/li>\n
        2. Elasticsearch indices can fill up quickly<\/strong> so it’s important to clean up old indices regularly. Kubernetes cron jobs can help you do this regularly in an automated fashion.<\/li>\n
        3. Having data replicated across multiple nodes can help in disaster recovery <\/strong>and also improve query performance<\/strong>. By default, replication factor in elasticsearch is set to 1.\n
          Consider playing around with this values according to your use case. Having atleast 2 is a good practise.<\/li>\n
        4. Data which is known to be accessed more frequently can be placed in different nodes with more resources allocated. This can be achieved by running a cronjob that moves the indices to different nodes at regular intervals.\n
          Though this is an advance use case – it is good for a beginner to atleast have knowledge that something like this can be done.<\/li>\n
        5. In Elasticsearch, you an archive indices to low cost cloud storage such as aws-s3 and restore when you need data from those indices.\n
          This is a best practise if you need to conserve logs for audit and compliance<\/strong>.<\/li>\n
        6. Having multiple nodes like master, data and client nodes with dedicated functionalities is good for high availability and fault tolerance. <\/li>\n<\/ol>\n
          Fluentd vs Fluent Bit<\/h2>\n
          Most of them are still confused about choosing between Fluent and FluentBit. Here, I am providing a brief comparison to give you a better idea.<\/p>\n
          <\/p>\n\n\n\n\n\n\n\n\n\n
          Aspect<\/th>\nFluent Bit<\/th>\nFluentd<\/th>\n<\/tr>\n<\/thead>\n
          Language<\/strong><\/td>\nC<\/td>\nRuby + C<\/td>\n<\/tr>\n
          Memory Footprint<\/strong><\/td>\n~650 KB<\/td>\n~40 MB<\/td>\n<\/tr>\n
          Plugins<\/strong><\/td>\n100+<\/td>\n700+<\/td>\n<\/tr>\n
          Best For<\/strong><\/td>\nEdge collection, Kubernetes nodes<\/td>\nLog aggregation, complex processing<\/td>\n<\/tr>\n
          Performance<\/strong><\/td>\nHigh throughput, low latency<\/td>\nModerate throughput, feature-rich<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
          <\/p>\n
          Beyond EFK – Further Research<\/h2>\n
          This guide was just a small use case of setting up the Elastic stack on Kubernetes. Elastic stack has tons of other features which help in logging and monitoring solutions.<\/p>\n
          For example, it can ship logs from virtual machines and managed services of various cloud providers. You can even ship logs from data engineering tools like Kafka into the Elastic Stack.<\/p>\n
          The elastic stack has other powerful components worth looking into, such as:<\/p>\n
            \n
          1. Elastic Metrics<\/strong>: Ships metrics from multiple sources across your entire infrastructure and makes it available in elastic search and kibana.<\/li>\n
          2. APM<\/strong>: Expands elastic stack capabilities and lets you analyze where exactly an application is spending time quickly fixing issues in production.<\/li>\n
          3. Uptime<\/strong>: Helps in monitoring and analyzing availability issues across your apps and services before they start appearing in the production.<\/li>\n<\/ol>\n
            Explore and research them!<\/p>\n
            Conclusion<\/h2>\n
            In his Kubernetes EFK setup guide<\/strong>, we learned how to configure the logging infrastructure on Kubernetes.<\/p>\n
            If you want to become a DevOps engineer,<\/a> it is very important to understand all the concepts involved in the Kubernetes logging.<\/p>\n
            In the next part of this series, we are going to explore Kibana dashboards and visualization options. <\/p>\n
            In Kibana, it is a good practice to visualize data through graphs<\/strong> wherever possible as it gives a much more clear picture of your application state. So don’t forget to check out the next part of this series.<\/p>\n
            Till then, keep learning and exploring.<\/p>\n
            \n
            Ngu\u1ed3n:<\/strong> How to Setup EFK Stack on Kubernetes: Step by Step Guides \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
            Source: https:\/\/devopscube.com\/setup-efk-stack-on-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":371,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-370","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/370","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=370"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/370\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/371"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=370"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=370"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=370"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}