{"id":508,"date":"2025-08-18T10:56:00","date_gmt":"2025-08-18T10:56:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=508"},"modified":"2025-08-18T10:56:00","modified_gmt":"2025-08-18T10:56:00","slug":"kubernetes-daemonset","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=508","title":{"rendered":"Kubernetes Daemonset: A Comprehensive Guide"},"content":{"rendered":"

If you’re new to Kubernetes Daemonsets and want to learn more, this complete beginner’s guide is for you.<\/p>\n

You’ll learn about<\/p>\n

    \n
  1. How Daemonset works<\/li>\n
  2. Deploying and managing Daemonsets<\/li>\n
  3. Daemonset use cases<\/li>\n
  4. Daemonset best practices.<\/li>\n<\/ol>\n

    So if you want to learn and implement Daemonset practically, you\u2019ll love this guide.<\/p>\n

    Let\u2019s get started.<\/p>\n

    What is a DaemonSet In Kubernetes?<\/h2>\n

    Kubernetes<\/a> is a distributed system, and there should be some functionality for kubernetes platform administrators to run platform-specific applications on all the nodes. For example, running a logging agent on all the Kubernetes nodes.<\/p>\n

    Here is where Daemonset comes into the picture.<\/p>\n

    Daemonset is a native Kubernetes object<\/a>. As the name suggests, it is designed to run system daemons.<\/p>\n

    The DaemonSet object is designed to ensure that a single pod runs on each worker node<\/strong>. This means you cannot scale daemonset pods in a node. And for some reason, if the daemonset pod gets deleted from the node, the daemonset controller creates it again.<\/p>\n

    Let’s look at an example. <\/p>\n

    If there are 500 worker nodes and you deploy a daemonset, the daemonset controller will run one pod per worker node by default. That is a total of 500 pods. However, using nodeSelector, nodeAffinity, Taints, and Tolerations<\/strong>, you can restrict the daemonset to run on specific nodes.<\/p>\n

    For example, in a cluster of 100 worker nodes, one might have 20 worker nodes labeled GPU enabled to run batch workloads. And you should run a pod on those 20 worker nodes. In this case, you can deploy the pod as a Daemonset using a node selector. We will look at it practically later in this guide.<\/p>\n

    Another example is that you have a specific number of worker nodes dedicated to platform tools (ingress, monitoring, logging, etc.) and want to run Daemonset related to platform tools only on the nodes labeled as platform tools. <\/p>\n

    In this case, you can use the nodeSelector<\/strong> to run the daemonset pods only on the worker nodes dedicated to platform tooling.<\/p>\n

    \"Kubernetes
    Click to view in HD<\/span><\/figcaption><\/figure>\n
    \n
    \ud83d\udca1<\/div>\n
    By default, DeamonSets<\/strong><\/b> run only on worker nodes and if need to run on Controlplane, we need to add tolerations.<\/div>\n<\/div>\n

    Kubernetes DaemonSet Use Cases<\/h2>\n

    The very basic use case of DaemonSet is in the cluster itself. If you look at the Kubernetes architecture<\/a>, the kube-proxy component runs a daemonset.<\/p>\n

    The following are the real-world use cases of Daemonset.<\/p>\n

      \n
    1. Cluster Log Collection:<\/strong> Running a log collector on every node to centralize Kubernetes logging<\/a> data. Eg:\u200a \u200afluentd<\/a>\u200a,\u200alogstash, fluentbit<\/li>\n
    2. Cluster Monitoring:<\/strong> Deploy monitoring agents, such as Prometheus Node Exporter<\/a>, on every node in the cluster to collect and expose node-level metrics. This way prometheus<\/a> gets all the required worker node metrics.<\/li>\n
    3. Security and Compliance:<\/strong> Running CIS Benchmarks on every node using tools like kube-bench<\/a>. Also deploy security agents, such as intrusion detection systems or vulnerability scanners, on specific nodes that require additional security measures. For example, nodes that handle PCI, and PII-compliant data.<\/li>\n
    4. Storage Provisioning: <\/strong>Running a storage plugin on every node to provide a shared storage system to the entire cluster.<\/li>\n
    5. Network Management:<\/strong> Running a network plugin or firewall on every node to ensure consistent network policy enforcement. For example, the Calico CNI plugin <\/a>runs as a Daemonset on all the nodes.<\/li>\n<\/ol>\n

      According to the requirements, we can deploy multiple DaemonSet<\/code> for one kind of daemon, using a variety of flags or memory and CPU requests for various hardware types<\/p>\n

      DaemonSet Example<\/h2>\n

      Like other Kubernetes objects, DaemonSet<\/code> also gets configured by using YAML files. We need to create a manifest file that will contain all of the necessary configuration information for our DaemonSet<\/code>.<\/p>\n

      Let’s assume we want to deploy a Fluentd logging agent as a DaemonSet on all the cluster worker nodes.<\/p>\n

      Below is a sample daemonset.yaml<\/code> file that gets deployed in the logging namespace.<\/p>\n

      You can also get the daemonset YAML examples from the Kubernetes Course Github Repo<\/a><\/p>\n

      apiVersion: apps\/v1\nkind: DaemonSet\nmetadata:\n  name: fluentd\n  namespace: logging\n  labels:\n    app: fluentd-logging\nspec:\n  selector:\n    matchLabels:\n      name: fluentd\n  template:\n    metadata:\n      labels:\n        name: fluentd\n    spec:\n      containers:\n      - name: fluentd-elasticsearch\n        image: quay.io\/fluentd_elasticsearch\/fluentd:v2.5.2\n        resources:\n          limits:\n            memory: 200Mi\n          requests:\n            cpu: 100m\n            memory: 200Mi\n        volumeMounts:\n        - name: varlog\n          mountPath: \/var\/log\n      terminationGracePeriodSeconds: 30\n      volumes:\n      - name: varlog\n        hostPath:\n          path: \/var\/log<\/code><\/pre>\n
      Let’s understand the manifest file.<\/p>\n
        \n
      1. apiVersion:<\/strong><\/code> apps\/v1<\/code> for DaemonSet<\/code><\/li>\n
      2. kind:<\/strong><\/code> DaemonSet<\/code> such as Pod, Deployment<\/a>, and Service<\/li>\n
      3. metadata:<\/strong><\/code> Put the name of the DaemonSet<\/code>, mention namespace, annotations, and labels. In our case DaemonSet's<\/code> name is fluentd.<\/li>\n
      4. spec.selector:<\/strong><\/code> The selector for the pods is managed by the DaemonSet<\/code>. This value must be a label specified in the pod template. This value is immutable.<\/li>\n
      5. spec.template:<\/strong><\/code> This is a required field that specifies a pod template for the DaemonSet<\/code> to use. Along with all the required fields for containers. It has everything of pod schema except apiVersion<\/code> and kind<\/code>.<\/li>\n<\/ol>\n
        template.metadata<\/code> will have the details about the pod and template.spec<\/code> will have the schema of the pod.<\/p>\n
        We don’t provide any replica counts. It is because the replica count of DaemonSet is dynamic in nature, as it depends on the node count of the cluster.<\/p>\n
        Let’s deploy this manifest by using the below commands. First, we will have to create a namespace and deploy the Daemonset in that namespace.<\/p>\n
        kubectl create ns logging\n\nkubectl apply -f daemonset.yaml<\/code><\/pre>\n
        Check the DaemonSet<\/code> status and pods’ status.<\/p>\n
        kubectl get daemonset -n logging<\/code><\/pre>\n
        kubectl get pods -n logging -o wide<\/code><\/pre>\n
        \"Listing<\/figure>\n
        You can see Fluentd pods are running on the two available worker nodes.<\/p>\n
        Here are some other useful commands to describe, edit, and get the DaemonSet<\/code>.<\/p>\n
        kubectl describe daemonset -n logging<\/code><\/pre>\n
        kubectl edit daemonset -n logging<\/code><\/pre>\n
        #shortcut for daemonset is ds\n\nkubectl get ds<\/code><\/pre>\n
        Apply Taint and Tolerations For Daemonset<\/h2>\n
        Taints<\/code> and Tolerations<\/code> are the Kubernetes feature that allows you to ensure that pods are not placed on inappropriate nodes.  We taint the nodes and add tolerations in the pod schema.<\/p>\n
        kubectl taint nodes node1 key1=value1:<Effect><\/code><\/pre>\n
        There are 3 effects:<\/p>\n
          \n
        1. NoSchedule<\/code><\/strong>: Kubernetes scheduler will only allow scheduling pods that have tolerations for the tainted nodes.<\/li>\n
        2. PreferNoSchedule:<\/code><\/strong> Kubernetes scheduler will try to avoid scheduling pods that don\u2019t have tolerations for the tainted nodes.<\/li>\n
        3. NoExecute:<\/code><\/strong> Kubernetes will evict the running pods from the nodes if the pods don\u2019t have tolerations for the tainted nodes.<\/li>\n<\/ol>\n
          Below, I’m tainting one of the nodes with key app and value monitoring and the effect is NoExecute<\/code>. We don’t want DaemonSet<\/code> to run the pod on this particular node.<\/p>\n
          kubectl taint node k8s-worker-2 app=fluentd-logging:NoExecute<\/code><\/pre>\n
          Now adding toleration like this in our daemonset.yaml<\/code><\/p>\n
          spec:\n  tolerations:\n  - key: app\n    value: fluentd-logging\n    operator: Equal\n    effect: NoExecute\n  containers:\n  -----\n  -----<\/code><\/pre>\n
          As you will update the DaemonSet<\/code>, you will see that one pod got deleted which was running on the node k8s-worker-2<\/code>. DaemonSet<\/code> now won’t schedule any pod on this node.<\/p>\n
          \"daemonset<\/figure>\n
          Using Nodeselector For Daemonset Pods<\/h2>\n
          We can use nodeSelector<\/code> to run the pods on some specific nodes. DaemonSet<\/code> controller will create Pods on nodes that match the node selector’s key and value<\/p>\n
          First, you need to add a label to the node.<\/p>\n
          kubectl label node <node-name> key=value<\/code><\/pre>\n
          For example, let’s say you want to label a node as type=platform-tools<\/code>, you can use the following command.<\/p>\n
          kubectl label node k8s-worker-1 type=platform-tools<\/code><\/pre>\n
          Now, to apply a nodeSelector to a Daemonset, under the spec section, add the nodeSelector<\/strong> with the key and value as shown below.<\/p>\n
          spec:\n  nodeSelector:\n    <key>: <value><\/code><\/pre>\n
          The following image shows the Daemonset YAML with the nodeSelector<\/strong> spec highlighter in yellow.<\/p>\n
          \"Daemonset
          Click to view in HD<\/span><\/figcaption><\/figure>\n
          Daemonset Node Affinity<\/h2>\n
          We can also achieve more fine-grained control over how nodes are selected using Node affinity<\/code>. DaemonSet<\/code> controller will create Pods on nodes that match Node affinity<\/code>.<\/p>\n
          Node affinity<\/code> is conceptually similar to nodeSelector<\/code>, allowing you to constrain which nodes your pod can be scheduled on based on node labels. There are two types of node affinity:<\/p>\n
            \n
          1. requiredDuringSchedulingIgnoredDuringExecution<\/code>: The scheduler can’t schedule the Pod unless the rule is met. This functions like nodeSelector<\/code>, but with a more expressive syntax.<\/li>\n
          2. preferredDuringSchedulingIgnoredDuringExecution<\/code>: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.<\/li>\n<\/ol>\n
            We can add an affinity like this to our manifest<\/p>\n
            spec:\n  affinity:\n    nodeAffinity:\n      requiredDuringSchedulingIgnoredDuringExecution:\n        nodeSelectorTerms:\n        - matchFields:\n          - key: key-name\n            operator: In\n            values:\n            - value-name<\/code><\/pre>\n
            The Pods are only allowed to run on nodes that have the key and values mentioned in the matchFields<\/code> section.<\/p>\n
            The following Daemonset YAML uses both Affinity rules highlighted in bold. Required rules for node label and a preferred rule to select a node with instance label instance type t2.large<\/strong><\/p>\n
            apiVersion: apps\/v1\nkind: DaemonSet\nmetadata:\n  name: fluentd\n  namespace: logging\n  labels:\n    app: fluentd-logging\nspec:\n  selector:\n    matchLabels:\n      name: fluentd\n  template:\n    metadata:\n      labels:\n        name: fluentd\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n            nodeSelectorTerms:\n            - matchExpressions:\n              - key: type\n                operator: In\n                values:\n                - platform-tools\n          preferredDuringSchedulingIgnoredDuringExecution:\n          - weight: 1\n            preference:\n              matchExpressions:\n              - key: instance-type\n                operator: In\n                values:\n                - t2.large\n      containers:\n      - name: fluentd-elasticsearch\n        image: quay.io\/fluentd_elasticsearch\/fluentd:v2.5.2\n        resources:\n          limits:\n            memory: 200Mi\n          requests:\n            cpu: 100m\n            memory: 200Mi\n        volumeMounts:\n        - name: varlog\n          mountPath: \/var\/log\n      terminationGracePeriodSeconds: 30\n      volumes:\n      - name: varlog\n        hostPath:\n          path: \/var\/log<\/code><\/pre>\n
            Daemonset Privileged Access<\/h2>\n
            There are use cases where you need privileged access to the host from the Deamonset pod. For example, the calico CNI daemoset requires<\/strong> host-level access for its networking requirements because it needs to modify IPtables.<\/p>\n
            Another example is the Kube-proxy daemonset. It also requires privileged access.<\/p>\n
            You can use securityContext<\/strong> in the Pod Spec to allow or deny Privileged access. A security context defines privilege and access control settings for a Pod or Container. To specify security settings for a pod, you need to include the securityContext<\/code> field in the pod manifest.<\/p>\n
            spec:\n  securityContext:\n    runAsNonRoot: true\n  containers:\n  - name: fluentd-elasticsearch\n    image: quay.io\/fluentd_elasticsearch\/fluentd:v2.5.2\n    securityContext:\n      allowPrivilegeEscalation: false\n    -------<\/code><\/pre>\n
            The first is a pod-level security context defined by the object, and the second is a SecurityContext defined by the individual container.<\/p>\n
              \n
            1. allowPrivilegeEscalation<\/code><\/strong>: AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process.<\/li>\n
            2. privileged<\/code><\/strong>: Run the container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host.<\/li>\n
            3. runAsNonRoot<\/code><\/strong>: Indicates that the container must run as a non-root user.<\/li>\n
            4. runAsUser<\/code><\/strong>: The UID to run the entry point of the container process.<\/li>\n
            5. runAsGroup<\/code><\/strong>: The GID to run the entry point of the container process.<\/li>\n<\/ol>\n
              Rolling Update, Rollback, and Deleting Daemonset<\/h2>\n
              Let’s look at the concepts for updating, deleting, and rolling back Daemonset deployments.<\/p>\n
              Rolling Update<\/h3>\n
              DaemonSet<\/code> has two update strategy types:<\/p>\n
                \n
              1. OnDelete<\/strong><\/code>: Using the OnDelete<\/code> strategy, the DaemonSet<\/code> pod will only be created when we manually delete any pod.<\/li>\n
              2. RollingUpdate<\/strong><\/code>:  This is the default update strategy. Using the RollingUpdate<\/code> strategy, whenever you update a DaemonSet<\/code> template, old pods will be killed, and new pods will be created automatically. At most one pod of the DaemonSet<\/code> will be running.<\/li>\n<\/ol>\n
                spec:\n  updateStrategy:\n    type: RollingUpdate\n    rollingUpdate:\n      maxUnavailable: 1<\/code><\/pre>\n
                Rollback<\/h3>\n
                We can rollback the DaemonSet<\/code> by using the below command:<\/p>\n
                kubectl rollout undo daemonset <daemonset-name><\/code><\/pre>\n
                To check all the revisions of the DaemonSet<\/code>:<\/p>\n
                kubectl rollout history daemonset <daemonset-name><\/code><\/pre>\n
                If you want to rollback to the specific revision, then use:<\/p>\n
                kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision><\/code><\/pre>\n
                Deletion<\/h3>\n
                kubectl delete daemonset <daemonset-name><\/code><\/pre>\n
                Use --cascade=false<\/code> if you want the pods to be running on the node.<\/p>\n
                DaemonSet Pod Priority<\/h2>\n
                Kubernetes Pod Priority<\/a> determines the importance of a pod over another pod.<\/p>\n
                We can set a higher pod PriorityClass<\/code> to the DaemonSet<\/code> in case you are running critical system components as a Deamonset. This ensures that the daemonset pods are not preempted by lower-priority or less critical pods.<\/p>\n
                PriorityClass<\/code> is used to define the priority of the pod. PriorityClass<\/code> objects can have any 32-bit integer value smaller than or equal to 1 billion.<\/em><\/strong> The higher the value, the higher will be the priority.<\/p>\n
                Create a priority class and that to DaemonSet pod spec<\/p>\n
                apiVersion: scheduling.k8s.io\/v1\nkind: PriorityClass\nmetadata:\n  name: high-priority\nvalue: 100000\nglobalDefault: false\ndescription: \"daemonset priority class\"<\/code><\/pre>\n
                Check by running this command<\/p>\n
                kubectl get priorityClass<\/code><\/pre>\n
                We need to add the priorityClass<\/code> in our daemonset.yaml<\/code><\/p>\n
                spec:\n  priorityClassName: high-priority\n  containers:\n  ------\n  ------\n  terminationGracePeriodSeconds: 30\n  volumes:\n  ------<\/code><\/pre>\n
                If you look at Kube-Proxy & Cluser CNI (Calico) Daemonsets, it has the priority class<\/a> set to system-node-critical which<\/strong> has the highest priority. It is a built-in PriorityClass<\/code> in Kubernetes that is applied to pods that should not be evicted in any circumstances.<\/p>\n
                Daemonset Troubleshooting<\/h2>\n
                A DaemonSet<\/code> is called unhealthy when any pod is not running on the node. The generic reason for that is pod status is crashloopbackoff<\/code>, the pod is in the pending<\/code> state or in an error<\/code> state. <\/p>\n
                In such cases, the initial step to check the logs of the Pods using the following command.<\/p>\n
                kubecttl -n <NAMESPACE> logs <POD NAME> -f<\/code><\/pre>\n
                From the output, we can fix this by:<\/p>\n
                  \n
                1. Pod might be running out of resources. We can lower the requested CPU and memory of the DaemonSet<\/code>.<\/li>\n
                2. We can move some pods off of the affected nodes to free up resources. Use taints and tolerations to prevent the pods from running on certain nodes.<\/li>\n
                3. We can scale up the nodes as well.<\/li>\n<\/ol>\n
                  You can perform normal pod troubleshooting<\/a> steps to figure out the issues.<\/p>\n
                  DaemonSet Best Practices<\/h2>\n
                  Below are some practices that we must follow:<\/p>\n
                    \n
                  1. DaemonSet<\/code> pods must have Restart Policy set to Always<\/code> or unspecified<\/code><\/li>\n
                  2. Separate each DaemonSet<\/code> into its own namespace to ensure clear isolation and easier resource management.<\/li>\n
                  3. It is better to use preferredDuringSchedulingIgnoredDuringExecution<\/code> instead of requiredDuringSchedulingIgnoredDuringExecution<\/code> because it is impossible to start new Pods if the number of nodes required for the new Pods is larger than the number of nodes available.<\/li>\n
                  4. DaemonSet<\/code> priority should be 10000. It is not advised for DaemonSet<\/code> Pods to be evicted from cluster nodes<\/li>\n
                  5. We must specify a pod selector that matches the labels of the .spec.template.<\/code><\/li>\n
                  6. We can use minReadySeconds<\/code> in pod schema. It tells Kubernetes how long it should wait until it creates the next pod. This property ensures that all application pods are in the ready state during the update.<\/li>\n
                  7. Use proper labels and selectors to deploy the DaemonSets on specific nodes.<\/li>\n
                  8. DaemonSet resource request\/limit should be kept as minimal as possible, because they run on each node.<\/li>\n
                  9. Use the PodDisruptionBudgets<\/code> to ensure the controlled eviction of Daemonset pods.<\/li>\n<\/ol>\n
                    Conclusion<\/h2>\n
                    In this kubernetes tutorial<\/a>, we learned about Kubernetes DaemonSets<\/code>. DaemonSet<\/code> is a great way to manage and deploy applications in a clustered environment.<\/p>\n
                    We have seen how can we create the DaemonSet<\/code> along with its use case. We can configure them to create pods on certain nodes using tolerations and node selectors and perform rolling updates on them in a controlled manner.<\/p>\n
                    Further, we have talked about unhealthy DaemonSet<\/code> and their best practices. Also, If you are preparing for CKA certification<\/a>, Daemonset is an important concept to prepare for the exam.<\/p>\n
                    If you are learning Kubernetes check out the Kubernetes learning roadmap<\/a> to deepen your Kubernetes knowledge.<\/p>\n
                    \n
                    Ngu\u1ed3n:<\/strong> Kubernetes Daemonset: A Comprehensive Guide \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
                    Source: https:\/\/devopscube.com\/kubernetes-daemonset\/<\/p>\n","protected":false},"author":1,"featured_media":509,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-508","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=508"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/508\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/509"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
        Here in the pod template, we are using quay.io\/fluentd_elasticsearch\/fluentd:v2.5.2<\/code> image that will run on every node in a Kubernetes cluster<\/a>. Each pod would then collect logs and send the data to ElasticSearch. Added resource limit and request for the pod, also volume and volumeMount accordingly.<\/p>\n