{"id":594,"date":"2022-04-26T08:45:21","date_gmt":"2022-04-26T08:45:21","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=594"},"modified":"2022-04-26T08:45:21","modified_gmt":"2022-04-26T08:45:21","slug":"troubleshoot-kubernetes-pods","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=594","title":{"rendered":"How To Troubleshoot Kubernetes Pods: Beginners Guide"},"content":{"rendered":"

In this blog, you will learn to troubleshoot <\/strong>kubernetes pods<\/strong><\/a> and debug issues associate with the containers inside the pods.<\/p>\n

If you are trying to become a devops engineer<\/a> with Kubernetes skills, it is essential to understand pod troubleshooting<\/strong>.<\/p>\n

In most cases, you can get the pod error details by describing the pod event. With the error message, you can figure out the cause of the pod failure and rectify it.<\/p>\n

How to Troubleshoot Pod Errors?<\/h2>\n

The first step in troubleshooting a pod is getting the status of the pod.<\/p>\n

kubectl get pods<\/code><\/pre>\n
The following output shows the error states under the status.<\/p>\n
\u279c  kubectl get pods\nNAME                            READY   STATUS                       RESTARTS   AGE\nconfig-service                  0\/1     CreateContainerConfigError   0          20s\nimage-service-fdf74c785-9znfd   0\/1     InvalidImageName             0          30s\nsecret-pod                      0\/1     ContainerCreating            0          15s<\/code><\/pre>\n
Now that you know the error type, the next step is to describe the individual pod and browse through the events to pinpoint the reason that is causing the pod error.<\/p>\n
For example,<\/p>\n
kubectl describe pod config-service<\/code><\/pre>\n
Where config-service<\/code> is the pod name. Now let’s look into detail on how to troubleshoot and debug different types of pod errors.<\/p>\n
\"\"<\/figure>\n
Types of Pod Errors<\/h2>\n
Before diving into pod debugging, it’s essential to understand different types of Pod errors.<\/p>\n
Container & Image Errors<\/h3>\n
All these error states are part of the kubernetes container package<\/a> & Kubernetes image package<\/a><\/p>\n
Following is the list of official Kubernetes pod errors<\/strong> with error descriptions.<\/p>\n
<\/p>\n\n\n\n\n<\/tr>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Pod Error Type<\/th>\nError Description<\/th>\n<\/tr>\n<\/thead>\n
ErrImagePull<\/code><\/td>\nIf kubernetes is not able to pull the image mentioned in the manifest. <\/td>\n<\/tr>\n
ErrImagePullBackOff<\/code><\/td>\nContainer image pull failed, kubelet is backing off image pull<\/td>\n<\/tr>\n
ErrInvalidImageName<\/code><\/td>\nIndicates a wrong image name.<\/td>\n<\/tr>\n
ErrImageInspect<\/code><\/td>\nUnable to inspect the image.<\/td>\n<\/tr>\n
ErrImageNeverPull<\/code><\/td>\nSpecified Image is absent on the node and PullPolicy is set to NeverPullImage<\/td>\n<\/tr>\n
ErrRegistryUnavailable<\/code><\/td>\nHTTP error when trying to connect to the registry<\/td>\n<\/tr>\n
ErrContainerNotFound<\/code><\/td>\nThe specified container is either not present or not managed by the kubelet, within the declared pod.<\/td>\n<\/tr>\n
ErrRunInitContainer<\/code><\/td>\nContainer initialization failed.<\/td>\n<\/tr>\n
ErrRunContainer<\/code><\/td>\nPod\u2019s containers don’t start successfully due to misconfiguration.<\/td>\n<\/tr>\n
ErrKillContainer<\/code><\/td>\nNone of the pod\u2019s containers were killed successfully.<\/td>\n<\/tr>\n
ErrCrashLoopBackOff<\/code><\/td>\nA container has terminated. The kubelet will not attempt to restart it.<\/td>\n<\/tr>\n
ErrVerifyNonRoot<\/code><\/td>\nA container or image attempted to run with root privileges.<\/td>\n<\/tr>\n
ErrCreatePodSandbox<\/code><\/td>\nPod sandbox creation did not succeed.<\/td>\n<\/tr>\n
ErrConfigPodSandbox<\/code><\/td>\nPod sandbox configuration was not obtained.<\/td>\n<\/tr>\n
ErrKillPodSandbox<\/code><\/td>\nA pod sandbox did not stop successfully.<\/td>\n<\/tr>\n
ErrSetupNetwork<\/code><\/td>\nNetwork initialization failed.<\/td>\n<\/tr>\n
ErrTeardownNetwork<\/code><\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/p>\n
Now let’s look at some of the most common pod errors <\/strong>and how to debug them.<\/p>\n
Troubleshoot ErrImagePullBackOff<\/h3>\n
\u279c  pods kubectl get pods\nNAME                                READY   STATUS             RESTARTS   AGE\nnginx-deployment-599d6bdb7d-lh7d9   0\/1     ImagePullBackOff   0          7m17s<\/code><\/pre>\n
If you see ErrImagePullBackOff<\/code> in pod status, it is most likely for the following reasons.<\/p>\n
    \n
  1. The specified image is not present in the registry.<\/li>\n
  2. A typo in the image name or tag.<\/li>\n
  3. Image pull access was denied from the given registry due to credential issues.<\/li>\n
  4. The contianer image that you are trying to pull doesnt support the worker node architecture.<\/li>\n<\/ol>\n
    If you check the pod events, you will see the ErrImagePull<\/code> error followed by ErrImagePullBackOff<\/code>. This means the kubelet stops trying to pull the image again and again.<\/p>\n
    kubectl describe pod <pod-name><\/code><\/pre>\n
    Error reference<\/p>\n
    Warning  Failed     24m (x4 over 25m)    kubelet            Error: ErrImagePull\n  Normal   BackOff    23m (x6 over 25m)    kubelet            Back-off pulling image \"ngasdinx:latest\"\n  Warning  Failed     29s (x110 over 25m)  kubelet            Error: ImagePullBackOff<\/code><\/pre>\n
    Troubleshoot Error: InvalidImageName<\/h3>\n
    \u279c  pods kubectl get pod\nNAME                                READY   STATUS             RESTARTS   AGE\nnginx-deployment-6f597fc4cd-j86mm   0\/1     InvalidImageName   0          7m26s<\/code><\/pre>\n
    If you specify a wrong image URL in the manifest, you will get the InvalidImageName<\/code> error.<\/p>\n
    For example, if you have a private container registry and you mention the image name with https<\/code>, it will throw the InvalidImageName<\/code> error. You need to specify the image name without https<\/code><\/p>\n
    \"troubleshoot<\/figure>\n
    If you have trailing slashes in the image name, you will get both InspectFailed<\/code> & InvalidImageName<\/code> errors. You can check it by describing the pod.<\/p>\n
    \"troubleshoot<\/figure>\n
    Error reference<\/p>\n
    Warning  InspectFailed  4s (x6 over 42s)  kubelet            Failed to apply default image tag \"registry.hub.docker.com\/library\/\/nginx:latest\": couldn't parse image reference \"registry.hub.docker.com\/library\/\/nginx:latest\": invalid reference format\n  Warning  Failed         4s (x6 over 42s)  kubelet            Error: InvalidImageName<\/code><\/pre>\n
    Troubleshoot RunContainerError<\/h3>\n
    secret-pod     0\/1     RunContainerError            0 (9s ago)   12s<\/code><\/pre>\n
    Pod Configmap & Secret Errors [CreateContainerConfigError]<\/h3>\n
    CreateContainerConfigError<\/code> is one of the common errors related to Configmaps<\/code> and Secrets<\/code> in pods.<\/p>\n
    This normally occurs due to two reasons.<\/p>\n
      \n
    1. You have the wrong configmap or secret keys referenced as environment variables<\/li>\n
    2. The referenced configmap is not available<\/li>\n<\/ol>\n
      If you describe the pod you will see the following error.<\/p>\n
      Warning  Failed     3s (x2 over 10s)  kubelet     Error: configmap \"nginx-config\" not found<\/code><\/pre>\n
      If you have a typo in the key name, you will see the following error in the pod events.<\/p>\n
      Warning  Failed     2s  kubelet  Error: couldn't find key service-names in ConfigMap default\/nginx-config<\/code><\/pre>\n
      To rectify this issue,<\/p>\n
        \n
      1. Ensure the config map is created.<\/li>\n
      2. Ensure you have the correct configmap name & key name<\/strong> added to the env declaration.<\/li>\n<\/ol>\n
        Let’s look at the correct example. Here is a configmap where service-name<\/code> is the key that is needed as an env variable inside the pod.<\/p>\n
        apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: nginx-config\n  namespace: default\ndata:\n  service-name: front-end-service<\/code><\/pre>\n
        Here is the correct pod definition using the key (service-name<\/code>) & configmap name (nginx-config<\/code>)<\/p>\n
        apiVersion: v1\nkind: Pod\nmetadata:\n  name: config-service\nspec:\n  containers:\n    - name: nginx\n      image: nginx\n      env:\n        - name: SERVICE\n          valueFrom:\n            configMapKeyRef:\n              name: nginx-config\n              key: service-name<\/code><\/pre>\n
        Pod Pending Error<\/h3>\n
        To troubleshoot pod pending error, you need to be aware of Pod LifeCycle Phases<\/a>. Pending in the first phase of the pod it means, the pod has been created but none of the main containers are running.<\/p>\n
        To understand the root cause, you can describe the pod and check the events.<\/p>\n
        For example,<\/p>\n
        kubectl describe pod <pod-name> -n <namespace><\/code><\/pre>\n
        Here is an example, output of a pending pod that shows FailedScheduling due to no node availability. It could happen due to<\/p>\n
        Events:\n  Type     Reason            Age                  From               Message\n  ----     ------            ----                 ----               -------\n  Warning  FailedScheduling  38s (x24 over 116m)  default-scheduler  no nodes available to schedule pods<\/code><\/pre>\n
        It could happen due to<\/p>\n
          \n
        1. Node Less CPU and Memory Resources<\/li>\n
        2. No suitable node due to Affinity\/Anti-affinity rules.<\/li>\n
        3. Nodes could reject the pod due to Taints and Tolerations<\/li>\n
        4. The node might not be ready to schedule the pods.<\/li>\n
        5. The pod could not find the volume to be attached.<\/li>\n<\/ol>\n
          \n
          Ngu\u1ed3n:<\/strong> How To Troubleshoot Kubernetes Pods: Beginners Guide \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
          Source: https:\/\/devopscube.com\/troubleshoot-kubernetes-pods\/<\/p>\n","protected":false},"author":1,"featured_media":595,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-594","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/594","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=594"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/594\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/595"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=594"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=594"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=594"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}