{"id":1094,"date":"2023-10-06T01:46:00","date_gmt":"2023-10-06T01:46:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=1094"},"modified":"2023-10-06T01:46:00","modified_gmt":"2023-10-06T01:46:00","slug":"kubernetes-cluster-configurations","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=1094","title":{"rendered":"Essential Guide to Kubernetes Configuration: Key Cluster Settings"},"content":{"rendered":"

In this blog, we will look at the important Kubernetes configurations every devops engineer<\/a> or Kubernetes administrator should know.<\/p>\n

Toward the end of the guide, I added the configurations involved in a real-world Kubernetes cluster setup.<\/p>\n

Note<\/strong>: All the config locations refered in this guide are based on the default Kubernetes installation using kubeadm<\/a>.<\/p><\/blockquote>\n

Kubernetes Cluster Configurations<\/h2>\n

Whether you are preparing for Kubernetes certifications<\/a> or plan to work on kubernetes projects, it is very important to know about the key configuration of the Kubernetes cluster.<\/p>\n

Also, when it comes to CKA certification<\/a>, you will get scenarios to rectify issues in the cluster. So understanding the cluster configurations will make it easier for you to troubleshoot and find issues in the cluster the right way.<\/p>\n

Let’s start with the configurations related to control plane components.<\/p>\n

Static Pod Manifests<\/h3>\n

As we discussed in the Kubernetes architecture<\/a>, all the control plane components are started by the kubelet from the static pod manifests present in the \/etc\/kubernetes\/manifests<\/code> directory. Kubelet manages the lifecycle of all the pods<\/a> created from the static pods manifests.<\/p>\n

The following components are deployed from the static pod manifests.<\/p>\n

    \n
  1. etcd<\/li>\n
  2. API server<\/li>\n
  3. Kube controller manager<\/li>\n
  4. Kube scheduler.<\/li>\n<\/ol>\n
    manifests\n  \u251c\u2500\u2500 etcd.yaml\n  \u251c\u2500\u2500 kube-apiserver.yaml\n  \u251c\u2500\u2500 kube-controller-manager.yaml\n  \u2514\u2500\u2500 kube-scheduler.yaml<\/code><\/pre>\n
    You can get all the configuration locations of these components from these pod manifests.<\/p>\n
    API Server Configurations<\/h3>\n
    If you look at the kube-apiserver.yaml<\/code><\/strong>, under the container<\/a> spec you can see all the parameters that point to TLS certs and other required parameters for the API server to work and communicate with other cluster components.<\/p>\n
    apiVersion: v1\nkind: Pod\nmetadata:\n  annotations:\n    kubeadm.kubernetes.io\/kube-apiserver.advertise-address.endpoint: 172.31.42.106:6443\n  creationTimestamp: null\n  labels:\n    component: kube-apiserver\n    tier: control-plane\n  name: kube-apiserver\n  namespace: kube-system\nspec:\n  containers:\n  - command:\n    - kube-apiserver\n    - --advertise-address=172.31.42.106\n    - --allow-privileged=true\n    - --authorization-mode=Node,RBAC\n    - --client-ca-file=\/etc\/kubernetes\/pki\/ca.crt\n    - --enable-admission-plugins=NodeRestriction\n    - --enable-bootstrap-token-auth=true\n    - --etcd-cafile=\/etc\/kubernetes\/pki\/etcd\/ca.crt\n    - --etcd-certfile=\/etc\/kubernetes\/pki\/apiserver-etcd-client.crt\n    - --etcd-keyfile=\/etc\/kubernetes\/pki\/apiserver-etcd-client.key\n    - --etcd-servers=https:\/\/127.0.0.1:2379\n    - --kubelet-client-certificate=\/etc\/kubernetes\/pki\/apiserver-kubelet-client.crt\n    - --kubelet-client-key=\/etc\/kubernetes\/pki\/apiserver-kubelet-client.key\n    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname\n    - --proxy-client-cert-file=\/etc\/kubernetes\/pki\/front-proxy-client.crt\n    - --proxy-client-key-file=\/etc\/kubernetes\/pki\/front-proxy-client.key\n    - --requestheader-allowed-names=front-proxy-client\n    - --requestheader-client-ca-file=\/etc\/kubernetes\/pki\/front-proxy-ca.crt\n    - --requestheader-extra-headers-prefix=X-Remote-Extra-\n    - --requestheader-group-headers=X-Remote-Group\n    - --requestheader-username-headers=X-Remote-User\n    - --secure-port=6443\n    - --service-account-issuer=https:\/\/kubernetes.default.svc.cluster.local\n    - --service-account-key-file=\/etc\/kubernetes\/pki\/sa.pub\n    - --service-account-signing-key-file=\/etc\/kubernetes\/pki\/sa.key\n    - --service-cluster-ip-range=10.96.0.0\/12\n    - --tls-cert-file=\/etc\/kubernetes\/pki\/apiserver.crt\n    - --tls-private-key-file=\/etc\/kubernetes\/pki\/apiserver.key\n    image: registry.k8s.io\/kube-apiserver:v1.26.3<\/code><\/pre>\n
    So if you want to troubleshoot or verify the cluster components configurations, first you should look at the static pod manifest configurations.<\/p>\n
    ETCD Configurations<\/h3>\n
    If you want to interact with the etcd component, you can use the details from the static pod YAML.<\/p>\n
    For example, if you want to backup etcd<\/a> you need to know the etcd service endpoint and related certificates to authenticate against etcd and create a backup.<\/p>\n
    If you open the etcd.yaml<\/code><\/strong> manifest you can view all the etcd-related configurations as shown below.<\/p>\n
    <\/figure>\n
    TLS Certificates<\/h3>\n
    In Kubernetes, all the components talk to each other over mTLS. Under the PKI folder, you will find all the TLS certificates and keys. Kubernetes control plane components use these certificates to authenticate and communicate with each other.<\/p>\n
    Also, there is an etcd subdirectory that contains the etcd-specific certificates and private keys. These are used to secure communication between etcd nodes and between the API server and etcd nodes.<\/p>\n
    The following image shows the file structure of the PKI folder.<\/p>\n
    \"kubernetes<\/figure>\n
    The static pod manifests refer to the required TLS certificates and keys from this folder.<\/p>\n
    When you work on a self-hosted cluster using tools like kubeadm, these certificates are automatically generated by the tool. In managed kubernetes clusters, the cloud provider takes care of all the TLS requirements as it is their responsibility to manage control plane components.<\/p>\n
    However, if you are setting up a self-hosted cluster for production use, these certificates have to be requested from the organization’s network or security team. They will generate these certificates signed by the organization’s internal Certificate authority and provide them to you.<\/p>\n
    Kubeconfig Files<\/h3>\n
    Any components that need to authenticate to the API server need the kubeconfig file<\/a>.<\/p>\n
    All the cluster Kubeconfig files are present in the \/etc\/kubernetes<\/strong><\/code> folder (.conf files). You will find the following files.<\/p>\n
      \n
    1. admin.conf<\/li>\n
    2. controller-manager.conf<\/li>\n
    3. kubelet.conf<\/li>\n
    4. scheduler.conf<\/li>\n<\/ol>\n
      It contains the API server endpoint, cluster CA certificate, cluster client certificate, and other information.<\/p>\n
      The admin.conf,<\/code><\/strong> file, which is the admin kubeconfig file used by end users to access the API server to manage the clusters. You can use this file to connect the cluster from a remote workstation.<\/p>\n
      The Kubeconfig for the Controller manager, scheduler, and Kubelet is used for API server authentication and authorization.<\/p>\n
      For example, if you check the Controller Manager static pod manifest file, you can see the controller-manager.conf<\/code><\/strong> added as the authentication and authorization parameter.<\/p>\n
      <\/figure>\n
      Kubelet Configurations<\/h2>\n
      Kubelet service runs as a systems service on all the cluster nodes.<\/p>\n
      You can view the kubelet systemd service under \/etc\/systemd\/system\/kubelet.service.d<\/code><\/strong><\/p>\n
      Here are the system file contents.<\/p>\n
      [Service]\nEnvironment=\"KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --kubeconfig=\/etc\/kubernetes\/kubelet.conf\"\nEnvironment=\"KUBELET_CONFIG_ARGS=--config=\/var\/lib\/kubelet\/config.yaml\"\nEnvironmentFile=-\/var\/lib\/kubelet\/kubeadm-flags.env\nEnvironmentFile=-\/etc\/default\/kubelet\nExecStart=\nExecStart=\/usr\/bin\/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS<\/code><\/pre>\n
      I have highlighted two important kubelet configurations in bold.<\/p>\n
        \n
      1. kubelet kubeconfig file: \/etc\/kubernetes\/kubelet.conf<\/strong><\/li>\n
      2. kubelet config file: \/var\/lib\/kubelet\/config.yaml<\/strong><\/li>\n
      3. EnvironmentFile=-\/var\/lib\/kubelet\/kubeadm-flags.env<\/strong><\/li>\n<\/ol>\n
        The kubeconfig file will be used for API server authentication and authorization.<\/p>\n
        The \/var\/lib\/kubelet\/config.yaml<\/strong> contains all the kubelet-related configurations. The static pod manifest location is added as part of the staticPodPath<\/strong> parameter.<\/p>\n
        staticPodPath: \/etc\/kubernetes\/manifests<\/code><\/pre>\n
        \/var\/lib\/kubelet\/kubeadm-flags.env<\/strong> file contains the container runtime environment Linux socket and the infra container (pause container) image.<\/p>\n
        For example, here is the kubelet config that is using the CRI-O container runtime, as indicated by the Unix socket and the pause container image.<\/p>\n
        KUBELET_KUBEADM_ARGS=\"--container-runtime-endpoint=unix:\/\/\/var\/run\/crio\/crio.sock --pod-infra-container-image=registry.k8s.io\/pa<\/code><\/pre>\n
        A pause container is a minimal container that is the first to be started within a Kubernetes Pod. Then the role of the pause container is to hold the networking namespace and other shared resources for all the other containers in the same Pod.<\/p>\n
        If you look at the kubelet configuration in the managed k8s cluster, it looks a little different than the kubeadm setup.<\/p>\n
        For example, here is a kubelet service file for an AWS EKS cluster<\/a>.<\/p>\n
        \"EKS<\/figure>\n
        Here you can see the container runtime is containerd and its Unix socket flag is directly added to the service file<\/p>\n
        The kubelet kubeconfig file is in a different directory as compared to kubeadm configurations.<\/p>\n
        CoreDNS Configurations<\/h2>\n
        CoreDNS<\/a> addon components deal with the cluster DNS configurations.<\/p>\n
        All the CoreDNS configurations are part of a configmap named CoreDNS in the kubesystem namespace.<\/p>\n
        If you list the Configmaps<\/code><\/strong> in the kube-system namespace, you can see the CoreDNS configmap.<\/p>\n
        kubectl get configmap --namespace=kube-system<\/code><\/pre>\n
        <\/figure>\n
        use the following command to view the CoreDNS<\/strong> configmap contents.<\/p>\n
        kubectl edit configmap coredns --namespace=kube-system<\/code><\/pre>\n
        You will see the following contents.<\/p>\n
        apiVersion: v1\ndata:\n  Corefile: |\n    .:53 {\n        errors\n        health {\n           lameduck 5s\n        }\n        ready\n        kubernetes cluster.local in-addr.arpa ip6.arpa {\n           pods insecure\n           fallthrough in-addr.arpa ip6.arpa\n           ttl 30\n        }\n        prometheus :9153\n        forward . \/etc\/resolv.conf {\n           max_concurrent 1000\n        }\n        cache 30\n        loop\n        reload\n        loadbalance\n    }<\/code><\/pre>\n
        When it comes to DNS connectivity, applications may need to connect to:<\/p>\n
          \n
        1. Internal services using Kubernetes service endpoints.<\/li>\n
        2. Publicly available services using public DNS endpoints.<\/li>\n
        3. In hybrid cloud environments, services are hosted in on-premise environments using private DNS endpoints.<\/li>\n<\/ol>\n
          If you have a use case where you need to have custom DNS servers, for example, the applications in the cluster need to connect to private DNS endpoints in the on-premise data center, you can add the custom DNS server to the core DNS configmap configurations.<\/p>\n
          For example, let’s say the custom DNS server IP is 10.45.45.34<\/strong> and your DNS suffix is dns-onprem.com<\/strong>, we have to add a block as shown below. So that all the DNS requests related to that domain endpoint will be forwarded to 10.45.45.34<\/strong> DNS server.<\/p>\n
          dns-onprem.com:53 {\n    errors\n    cache 30\n    forward . 10.45.45.34\n}<\/code><\/pre>\n
          Here is the full configmap configuration with the custom block highlighted in bold.<\/p>\n
          apiVersion: v1\ndata:\n  Corefile: |\n    .:53 {\n        errors\n        health {\n           lameduck 5s\n        }\n        ready\n        kubernetes cluster.local in-addr.arpa ip6.arpa {\n           pods insecure\n           fallthrough in-addr.arpa ip6.arpa\n           ttl 30\n        }\n        prometheus :9153\n        forward . \/etc\/resolv.conf {\n           max_concurrent 1000\n        }\n        cache 30\n        loop\n        reload\n        loadbalance\n    }\n    dns-onprem.com:53 {\n    errors\n    cache 30\n    forward . 10.45.45.34\n    }<\/code><\/pre>\n
          Audit Logging Configuration<\/h2>\n
          When it comes to production clusters, audit logging is a must-have feature.<\/p>\n
          Audit logging is enabled in the kube-api-server.yaml<\/code><\/strong> static pod manifest.<\/p>\n
          The command argument should have the following two parameters. The file path is arbitrary and it depends on the cluster administrator.<\/p>\n
           --audit-policy-file=\/etc\/kubernetes\/audit\/audit-policy.yaml\n --audit-log-path=\/var\/log\/kubernetes\/audit.log<\/code><\/pre>\n
          audit-policy.yaml <\/code>contains all the audit policies and audit.log<\/code> file contains the audit logs generated by Kubernetes.<\/p>\n
          Kubernetes Configurations Video<\/h2>\n
          I have made a video explaining the Kubernetes configuration. You can watch the video on Youtube. Here is the embedded video.<\/p>\n