{"id":386,"date":"2026-01-16T01:25:00","date_gmt":"2026-01-16T01:25:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=386"},"modified":"2026-01-16T01:25:00","modified_gmt":"2026-01-16T01:25:00","slug":"backup-etcd-restore-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=386","title":{"rendered":"Etcd Backup and Restore on Kubernetes Cluster [Tutorial]"},"content":{"rendered":"

In this kubernetes tutorial, you will learn the etcd backup and restore on Kubernetes cluster with an etcd snapshot.<\/p>\n

In Kubernetes architecture, etcd is an integral part of the cluster. All the cluster objects and their state is stored in etcd. Few things you should know about etcd from a Kubernetes perspective.<\/p>\n

    \n
  1. It is a consistent, distributed, and secure key-value store.<\/li>\n
  2. It uses raft protocol.<\/a><\/li>\n
  3. Supports highly available architecture with stacked etcd.<\/li>\n
  4. It stores kubernetes cluster configurations, all API objects, object states, and service discovery<\/a> details.<\/li>\n<\/ol>\n

    If you want to understand more about etcd and how kubernetes uses it, I recommend reading the comprehensive Kubernetes Architecture<\/a> post.<\/p>\n

    Also, if you consider Kubernetes design best practices<\/a>, Kubernetes etcd backup and restore is one of the important aspects under the backup strategy.<\/p>\n

    Kubernetes etcd Backup Using etcdctl<\/h2>\n

    Here is what you should know about etcd backup.<\/p>\n

      \n
    1. etcd has a built-in snapshot mechanism.<\/li>\n
    2. etcdctl<\/code> is the command line utility that interacts with etcd for snapshots.<\/li>\n<\/ol>\n
      \"etcd
      Click to view in HD<\/span><\/figcaption><\/figure>\n

      Follow the steps given below to take an etcd snapshot.<\/p>\n

      Step 1: Log in to the control plane.<\/h3>\n

      SSH into your control plane to backup ETCD.<\/p>\n

      Step 2:<\/strong> Install Required Tools<\/h3>\n

      We need two tools called etcdctl and etcdutl to backup and restore ETCD.<\/p>\n

      If you don’t have etcdctl and etcdutl installed in your cluster, install it using the following script.<\/p>\n

      ARCH=$(uname -m)\nif [ \"$ARCH\" = \"x86_64\" ]; then\n    ARCH_TYPE=\"amd64\"\nelif [ \"$ARCH\" = \"aarch64\" ]; then\n    ARCH_TYPE=\"arm64\"\nelse\n    echo \"Unsupported architecture: $ARCH\"\n    exit 1\nfi\n\nETCD_VER=v3.6.7\nDOWNLOAD_URL=https:\/\/storage.googleapis.com\/etcd\nDOWNLOAD_FILE=etcd-${ETCD_VER}-linux-${ARCH_TYPE}.tar.gz\n\nmkdir -p \/tmp\/etcd-download\ncurl -L ${DOWNLOAD_URL}\/${ETCD_VER}\/${DOWNLOAD_FILE} -o \/tmp\/${DOWNLOAD_FILE}\n\ntar xzvf \/tmp\/${DOWNLOAD_FILE} -C \/tmp\/etcd-download --strip-components=1\n\nsudo mv \/tmp\/etcd-download\/etcd \/tmp\/etcd-download\/etcdctl \/tmp\/etcd-download\/etcdutl \/usr\/local\/bin\/\nrm -f \/tmp\/${DOWNLOAD_FILE}\nrm -rf \/tmp\/etcd-download<\/code><\/pre>\n
      Step 3: Verify the Installation<\/h3>\n
      Run the following commands to verify the installation.<\/p>\n
      etcdctl version\n\netcdutl version<\/code><\/pre>\n
      Step 4: Get Required Information for Backup<\/h3>\n
      We need to get the following four pieces of information to etcdctl<\/code> to take an etcd snapshot.<\/p>\n
        \n
      1. etcd endpoint (–endpoints)<\/li>\n
      2. ca certificate (–cacert)<\/li>\n
      3. server certificate (–cert)<\/li>\n
      4. server key (–key)<\/li>\n<\/ol>\n
        You can get the above details in two ways.<\/p>\n
        From the etcd static pod manifest file located at \/etc\/kubernetes\/manifests\/etcd.yaml<\/code> the location.<\/p>\n
        \"getting<\/figure>\n
        You can also get the above details by describing the etcd pod running<\/strong> in the kube-system<\/code> namespace.<\/p>\n
        While describing the pod, replace etcd-master-node<\/code><\/strong> with your etcd pod name.<\/p>\n
        kubectl get po -n kube-system\nkubectl describe pod etcd-master-node -n kube-system<\/code><\/pre>\n
        Another way to view the etcd-server parameters faster is using the following command.<\/p>\n
        ps -aux | grep etcd<\/code><\/pre>\n
        Choose a method that you are comfortable with the get the cert details.<\/p>\n
        Step 4: Backup using etcdctl<\/h3>\n
        To backup, take an etcd snapshot backup using the following command.<\/p>\n
        sudo etcdctl \\\n  --endpoints=<etcd endpoint> \\\n  --cacert=<ca-file> \\\n  --cert=<cert-file> \\\n  --key=<key-file> \\\n  snapshot save <backup-file-location><\/code><\/pre>\n
        Before running the backup command, create a backup directory where we will be saving the snapshot.<\/p>\n
        sudo mkdir -p \/opt\/backup\/<\/code><\/pre>\n
        The command looks like the following when you add the actual location and parameters. Execute the command to perform a backup. <\/p>\n
        You can replace \/opt\/backup\/etcd.db<\/code> with the location and name of your choice.<\/p>\n
        sudo etcdctl \\\n  --endpoints=https:\/\/127.0.0.1:2379 \\\n  --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt \\\n  --cert=\/etc\/kubernetes\/pki\/etcd\/server.crt \\\n  --key=\/etc\/kubernetes\/pki\/etcd\/server.key \\\n  snapshot save \/opt\/backup\/etcd.db<\/code><\/pre>\n
        On a successful execution you will get a Snapshot saved at \/opt\/backup\/etcd.db<\/code> message as shown below.<\/p>\n
        root@master-node:\/opt\/backup# sudo etcdctl \\\n  --endpoints=https:\/\/192.168.201.10:2379 \\\n  --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt \\\n  --cert=\/etc\/kubernetes\/pki\/etcd\/server.crt \\\n  --key=\/etc\/kubernetes\/pki\/etcd\/server.key \\\n  snapshot save \/opt\/backup\/etcd.db\n2022-09-29 08:23:43.027261 I | clientv3: opened snapshot stream; downloading\n2022-09-29 08:23:43.088264 I | clientv3: completed snapshot read; closing\nSnapshot saved at \/opt\/backup\/etcd.db<\/code><\/pre>\n
        Also, you verify the snapshot snapshot status using the etcdutl command.<\/p>\n
        etcdutl --write-out=table snapshot status \/opt\/backup\/etcd.db<\/code><\/pre>\n
        Here is an example output.<\/p>\n
        +----------+----------+------------+------------+\n|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |\n+----------+----------+------------+------------+\n| b7147656 |    51465 |       1099 |     5.1 MB |\n+----------+----------+------------+------------+<\/code><\/pre>\n
        Kubernetes etcd Restore Using Snapshot Backup<\/h2>\n
        Now we have the backup in the \/opt\/backup\/etcd.db<\/code> location. We will use the snapshot backup to restore etcd with the help of etcdutl.<\/p>\n
        Before restoring, you should stop the kube-apiserver<\/code> and etcd<\/code> pods. Because, the API server continously reads or writes to etcd, if API server is running during restore, it can cause data corruption.<\/p><\/blockquote>\n
        Step 1: Move Statis Pod Manifests<\/h3>\n
        First, we need to move the manifest files of the API server and ETCD.<\/p>\n
        Since they are static, their pods will be recreated, even if you delete them.<\/p>\n
        Use the following commands to move the files to \/tmp<\/code> location.<\/p>\n
        sudo mv \/etc\/kubernetes\/manifests\/kube-apiserver.yaml \/tmp\/\n\nsudo mv \/etc\/kubernetes\/manifests\/etcd.yaml \/tmp\/<\/code><\/pre>\n
        Once the manifests are moved, the kube-apiserver<\/strong> and ETCD<\/strong> pods will be terminated within a few seconds.<\/p>\n
        You can verify this by running the crictl<\/strong> command. Make sure the API server and ETCD containers are no longer running. Wait until the containers are fully removed before proceeding.<\/p>\n
        crictl ps\n<\/code><\/pre>\n
        Step 2: Restore ETCD using etcdutl<\/h3>\n
        Now we have the backup in the \/opt\/backup\/etcd.db<\/code> location, let’s restore it.<\/p>\n
        Given below is the etcdutl command to restore etcd.<\/strong><\/p>\n
        sudo etcdctl snapshot restore <snapshot-location><\/code><\/pre>\n
        Let’s execute the etcd restore command. In my case \/opt\/backup\/etcd.db<\/code> is the backup file.<\/p>\n
        sudo etcdctl snapshot restore \/opt\/backup\/etcd.db<\/code><\/pre>\n
        If you want to use a specific data directory for the restore, you can add the location using the --data-dir<\/code> flag as shown below.<\/p>\n
        sudo etcdutl --data-dir \/var\/lib\/etcd-from-backup \\\n  snapshot restore \/opt\/backup\/etcd.db<\/code><\/pre>\n
        If the restoration is successful, you will get the output logs with restored snapshot message as shown below.<\/p>\n
        \"etcd<\/figure>\n
        After restoring the etcd snapshot to a new directory, we need to update etcd.yaml<\/code> because it still references the old data path.<\/p>\n
        \ufeffGenerally, you should change only the hostPath<\/code>. This keeps the internal container structure exactly as kubeadm expects it, while pointing it to your new data folder.<\/p>\n
        Edit the \/tmp\/etcd.yaml<\/code> file modify the following sections and change the new directory in the volume section, as given below.<\/p>\n
        volumes:\n  - name: etcd-data\n    hostPath:\n      path: \/var\/lib\/etcd-from-backup\n      type: DirectoryOrCreate<\/code><\/pre>\n
        Then, move the manifest files back to the original static pod directory. This will redeploy the kube-apiserver<\/strong> and ETCD<\/strong> pods.<\/p>\n
        sudo mv \/tmp\/kube-apiserver.yaml \/etc\/kubernetes\/manifests\/\n\nsudo mv \/tmp\/etcd.yaml \/etc\/kubernetes\/manifests\/<\/code><\/pre>\n
        It will take around 10-15 seconds <\/em><\/strong>for the API server and ETCD to start running.<\/p>\n
        Once the Pods are back up, verify the cluster status. It is the ultimate test is whether kubectl<\/code> can see the objects that were in the backup<\/p>\n
        $ kubectl get no\n\nNAME           STATUS   ROLES           AGE     VERSION\ncontrolplane   Ready    control-plane   3h13m   v1.34.0\nnode01         Ready    worker          3h11m   v1.34.0\nnode02         Ready    worker          3h11m   v1.34.0<\/code><\/pre>\n
        You can also check the ETCD status using the following command. Replace the endpoint IP with your ETCD endpoint IP.<\/p>\n
        $ sudo etcdctl \\\n  --endpoints=https:\/\/192.168.201.10:2379 \\\n  --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt \\\n  --cert=\/etc\/kubernetes\/pki\/etcd\/server.crt \\\n  --key=\/etc\/kubernetes\/pki\/etcd\/server.key \\\n  endpoint health\n\nhttps:\/\/192.168.201. 10:2379 is healthy: successfully committed proposal: took = 13.107842ms<\/code><\/pre>\n
        etcd Backup FAQs<\/h2>\n
        How to take etcd backup in Kubernetes?<\/h3>\n
        To take the etcd backup, you need to the etcdctl command line utility. You need to use the etcdctl snapshot command with the etcd certificates to perform a backup operation.<\/p>\n
        Conclusion<\/h2>\n
        In this blog, we learned Kubernetes etcd backup and restore<\/strong> using etcdctl<\/code> and etcdutl<\/code> command line utility.<\/p>\n
        etcd backup, and restore<\/strong> are essential tasks in Kubernetes cluster administration. Also, it is an important topic in the CKA certification exam.<\/p>\n
        If you are preparing for the CKA exam, do check out the CKA exam study guide<\/a> and for the exam discount voucher, check the Kubernetes certification coupon<\/a>.<\/p>\n
        \n
        Ngu\u1ed3n:<\/strong> Etcd Backup and Restore on Kubernetes Cluster [Tutorial] \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
        Source: https:\/\/devopscube.com\/backup-etcd-restore-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":387,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-386","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/386","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=386"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/386\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/387"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}