Prerequisites<\/h2>\n
This setup is performed in Ubuntu 22.04 Instance<\/strong> and the instance type is t2.medium<\/strong>.<\/p>\n You need a minimum of two nodes for this setup. (One controller and one worker node).<\/p>\n Also, ensure you have opened up all the required ports for Kubernetes on both nodes.<\/p>\n Let’s get started with the setup.<\/p>\n Both controller and worker nodes need IAM roles with required permissions for the cloud controller manager to interact with the AWS APIs.<\/p>\n Create an IAM role<\/a> with the following permissions and attach it to the controller node<\/p>\n Create an IAM role with the following permissions and attach it to the worker nodes.<\/p>\n Ensure each node hostname has its own private DNS address. If not, change the hostname using the following command. This is a requirement.<\/p>\n Run the following command on each node (controller and worker).<\/p>\n Verify the hostname by executing the hostname command.<\/p>\n Container Runtime (CRI-O<\/a><\/strong>), kubelet, Kubeadm, and kubectl are the important utilities that should be present in each node (controller and worker).<\/p>\n Install utilities using a shell script To run the script, provide execute permission.<\/p>\n Login as root and execute the script.<\/p>\n In this step, we will initialize the control plane with configurations required for the cloud controller manager.<\/p>\n Create configuration file In Initialize the configuration to bootstrap the kubeadm configuration.<\/p>\n At the end of the initialization, a token will be generated by the controller to join the workers. This is required to create the controller configuration<\/strong> file.<\/p>\n It would like the following highlighted token. Note down the token and cert hash.<\/p>\n Create a configuration file in each worker node Change the Modify the node registration name to your worker node\u2019s private DNS address.<\/p>\n To join the workers to the controller, use the following command.<\/p>\n Modify the intended values of the configuration file and run the Tagging is essential to configure the Cloud Controller Manager because it ensures which AWS resources are used by which cluster.<\/p>\n For example, if a cluster uses AWS Network Load Balancer<\/a> and now the cluster is being destroyed, the specified NLB also be destroyed and not affect other resources. To tag the resource, the cluster-ID is required<\/p>\n To find the Cluster ID, use the following command.<\/p>\n The output would be like this.<\/p>\n In the If the AWS resources are managed by one cluster, we can provide the tag key Tags should be added to resourecs the controller and worker node consumes, such as VPC, Subnet, EC2 instance, Security Group, etc.<\/p><\/blockquote>\n Here is an example.<\/p>\n Clone the AWS Cloud Controller repository to the controller plane node where you have the Kubectl access.<\/p>\n Navigate to the base directory. It has all the kubernetes manifests<\/a> for the cloud controller manager and the Kustomize<\/a> file.<\/p>\n Create the daemonset<\/a> using the following command. -k is for Kustomize.<\/p>\n To verify the daemonset is running properly, use the following command.<\/p>\n To ensure the CCM pod<\/a> is running, use the following command.<\/p>\n To test if the Cloud controller manager is working, we will deploy a sample Nginx deployment and expose it over service type LoadBlancer. The cloud controller manager should be able to create a Network load balancer that routes traffic to the nginx deployment.<\/p>\n Create a deployment file for the Nginx web server using the following YAML. Execute it directly.<\/p>\n Create a service file to provision a load balancer To ensure the deployment is running properly, use the following command.<\/p>\n To ensure the service is running properly, use the following command.<\/p>\n You should see the load balancer endpoint in the output as highlighted below.<\/p>\n The registration process takes a few minutes. If you check the load balancer after a while you should see the worker nodes registered as healthy targets. The service NodePort would be used as a health check port.<\/p>\n Once the nodes are registered to the NLB, you should see the Nginx homepage if you visit the Load balancer URL as shown below.<\/p>\n The following image shows the CCM load balancer traffic workflow to pods.<\/p>\n To find the logs of the CCM pod, first, you need to find the pod name and use the following command<\/p>\n Following are the errors I have faced in this setup<\/p>\n Solution:<\/strong><\/p>\n Ensure all resources which are used by CCM (controller and worker) should be tagged with the correct cluster-ID.<\/p>\n Example:<\/strong><\/p>\n solution:<\/strong><\/p>\n Before starting the cluster configuration, ensure all the servers\u2019 hostname is their private DNS address also keep in mind that the configuration files also use private DNS addresses.<\/p>\n Example:<\/strong><\/p>\n Solution:<\/strong><\/p>\n Ensure you have a copy of Example:<\/strong><\/p>\n Solution:<\/strong><\/p>\n Usually, this error happens when you remove the worker node and try to join again. To resolve this issue, remove the Then join the worker node again using the following command.<\/p>\n I have done a high-level setup and tested provisioning the Network load balancer.<\/p>\n Next, I will be testing storage provisioning using the cloud controller manager.<\/p>\n I have added the steps after several trials and errors due to less information in the documentation. If you face any error during the setup, do drop an comment and I will look into it. To explore more, you can refer to the official documentation<\/a>.<\/p>\n Also, if you are getting started with the Kubeadm cluster for CKA certification<\/a>, take a look at the detailed kubeadm setup guide<\/a>.<\/p>\nStep 1: Attach IAM Roles<\/h2>\n
IAM Policy for the controller node<\/h3>\n
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"autoscaling:DescribeAutoScalingGroups\",\n \"autoscaling:DescribeLaunchConfigurations\",\n \"autoscaling:DescribeTags\",\n \"ec2:DescribeInstances\",\n \"ec2:DescribeRegions\",\n \"ec2:DescribeRouteTables\",\n \"ec2:DescribeSecurityGroups\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeVolumes\",\n \"ec2:DescribeAvailabilityZones\",\n \"ec2:CreateSecurityGroup\",\n \"ec2:CreateTags\",\n \"ec2:CreateVolume\",\n \"ec2:ModifyInstanceAttribute\",\n \"ec2:ModifyVolume\",\n \"ec2:AttachVolume\",\n \"ec2:AuthorizeSecurityGroupIngress\",\n \"ec2:CreateRoute\",\n \"ec2:DeleteRoute\",\n \"ec2:DeleteSecurityGroup\",\n \"ec2:DeleteVolume\",\n \"ec2:DetachVolume\",\n \"ec2:RevokeSecurityGroupIngress\",\n \"ec2:DescribeVpcs\",\n \"elasticloadbalancing:AddTags\",\n \"elasticloadbalancing:AttachLoadBalancerToSubnets\",\n \"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer\",\n \"elasticloadbalancing:CreateLoadBalancer\",\n \"elasticloadbalancing:CreateLoadBalancerPolicy\",\n \"elasticloadbalancing:CreateLoadBalancerListeners\",\n \"elasticloadbalancing:ConfigureHealthCheck\",\n \"elasticloadbalancing:DeleteLoadBalancer\",\n \"elasticloadbalancing:DeleteLoadBalancerListeners\",\n \"elasticloadbalancing:DescribeLoadBalancers\",\n \"elasticloadbalancing:DescribeLoadBalancerAttributes\",\n \"elasticloadbalancing:DetachLoadBalancerFromSubnets\",\n \"elasticloadbalancing:DeregisterInstancesFromLoadBalancer\",\n \"elasticloadbalancing:ModifyLoadBalancerAttributes\",\n \"elasticloadbalancing:RegisterInstancesWithLoadBalancer\",\n \"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer\",\n \"elasticloadbalancing:AddTags\",\n \"elasticloadbalancing:CreateListener\",\n \"elasticloadbalancing:CreateTargetGroup\",\n \"elasticloadbalancing:DeleteListener\",\n \"elasticloadbalancing:DeleteTargetGroup\",\n \"elasticloadbalancing:DescribeListeners\",\n \"elasticloadbalancing:DescribeLoadBalancerPolicies\",\n \"elasticloadbalancing:DescribeTargetGroups\",\n \"elasticloadbalancing:DescribeTargetHealth\",\n \"elasticloadbalancing:ModifyListener\",\n \"elasticloadbalancing:ModifyTargetGroup\",\n \"elasticloadbalancing:RegisterTargets\",\n \"elasticloadbalancing:DeregisterTargets\",\n \"elasticloadbalancing:SetLoadBalancerPoliciesOfListener\",\n \"iam:CreateServiceLinkedRole\",\n \"kms:DescribeKey\"\n ],\n \"Resource\": [\n \"*\"\n ]\n }\n ]\n}\n<\/code><\/pre>\nIAM Policy for the Worker Nodes<\/h3>\n
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:DescribeInstances\",\n \"ec2:DescribeRegions\",\n \"ecr:GetAuthorizationToken\",\n \"ecr:BatchCheckLayerAvailability\",\n \"ecr:GetDownloadUrlForLayer\",\n \"ecr:GetRepositoryPolicy\",\n \"ecr:DescribeRepositories\",\n \"ecr:ListImages\",\n \"ecr:BatchGetImage\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\n<\/code><\/pre>\nStep 2: Set Hostname (On All Nodes)<\/h2>\n
sudo hostnamectl set-hostname $(curl -s http:\/\/169.254.169.254\/latest\/meta-data\/local-hostname)<\/code><\/pre>\nubuntu@node01:~$ hostname\nip-172-31-16-213.us-west-2.compute.internal<\/code><\/pre>\nStep 3: Install Kubeadm System Utilities<\/h2>\n
utilities.sh<\/code><\/strong>. It installs kubeadm version 1.30<\/code><\/strong><\/p>\n#!\/bin\/bash\n#\n# Common setup for all servers (Control Plane and Nodes)\n\nset -euxo pipefail\n\n# Kubernetes Variable Declaration\nKUBERNETES_VERSION=\"v1.30\"\nCRIO_VERSION=\"v1.30\"\nKUBERNETES_INSTALL_VERSION=\"1.30.0-1.1\"\n\n# Disable swap\nsudo swapoff -a\n\n# Keeps the swap off during reboot\n(crontab -l 2>\/dev\/null; echo \"@reboot \/sbin\/swapoff -a\") | crontab - || true\nsudo apt-get update -y\n\n# Create the .conf file to load the modules at bootup\ncat <<EOF | sudo tee \/etc\/modules-load.d\/k8s.conf\noverlay\nbr_netfilter\nEOF\n\nsudo modprobe overlay\nsudo modprobe br_netfilter\n\n# Sysctl params required by setup, params persist across reboots\ncat <<EOF | sudo tee \/etc\/sysctl.d\/k8s.conf\nnet.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\nEOF\n\n# Apply sysctl params without reboot\nsudo sysctl --system\n\nsudo apt-get update -y\nsudo apt-get install -y apt-transport-https ca-certificates curl gpg\n\n# Install CRI-O Runtime\nsudo apt-get update -y\nsudo apt-get install -y software-properties-common curl apt-transport-https ca-certificates\n\ncurl -fsSL https:\/\/pkgs.k8s.io\/addons:\/cri-o:\/stable:\/$CRIO_VERSION\/deb\/Release.key |\n gpg --dearmor -o \/etc\/apt\/keyrings\/cri-o-apt-keyring.gpg\n\necho \"deb [signed-by=\/etc\/apt\/keyrings\/cri-o-apt-keyring.gpg] https:\/\/pkgs.k8s.io\/addons:\/cri-o:\/stable:\/$CRIO_VERSION\/deb\/ \/\" |\n tee \/etc\/apt\/sources.list.d\/cri-o.list\n\nsudo apt-get update -y\nsudo apt-get install -y cri-o\n\nsudo systemctl daemon-reload\nsudo systemctl enable crio --now\nsudo systemctl start crio.service\n\necho \"CRI runtime installed successfully\"\n\n# Install kubelet, kubectl, and kubeadm\ncurl -fsSL https:\/\/pkgs.k8s.io\/core:\/stable:\/$KUBERNETES_VERSION\/deb\/Release.key |\n gpg --dearmor -o \/etc\/apt\/keyrings\/kubernetes-apt-keyring.gpg\n\necho \"deb [signed-by=\/etc\/apt\/keyrings\/kubernetes-apt-keyring.gpg] https:\/\/pkgs.k8s.io\/core:\/stable:\/$KUBERNETES_VERSION\/deb\/ \/\" |\n tee \/etc\/apt\/sources.list.d\/kubernetes.list\n\nsudo apt-get update -y\nsudo apt-get install -y kubelet=\"$KUBERNETES_INSTALL_VERSION\" kubectl=\"$KUBERNETES_INSTALL_VERSION\" kubeadm=\"$KUBERNETES_INSTALL_VERSION\"\n\n# Prevent automatic updates for kubelet, kubeadm, and kubectl\nsudo apt-mark hold kubelet kubeadm kubectl\n\nsudo apt-get update -y\n\n# Install jq, a command-line JSON processor\nsudo apt-get install -y jq\n\n# Retrieve the local IP address of the eth0 interface and set it for kubelet\nlocal_ip=\"$(ip --json addr show eth1 | jq -r '.[0].addr_info[] | select(.family == \"inet\") | .local')\"\n\n# Write the local IP address to the kubelet default configuration file\ncat > \/etc\/default\/kubelet << EOF\nKUBELET_EXTRA_ARGS=--node-ip=$local_ip\nEOF<\/code><\/pre>\nchmod +x utilities.sh<\/code><\/pre>\n.\/utilities.sh<\/code><\/pre>\nStep 4: Initialize Kubeadm Configuration (Only in Controller)<\/h2>\n
kubeadm.config<\/code><\/p>\napiVersion: kubeadm.k8s.io\/v1beta3\nkind: ClusterConfiguration\napiServer:\n certSANs:\n - 127.0.0.1\n - 52.38.15.235\n extraArgs:\n bind-address: \"0.0.0.0\"\n cloud-provider: external\nclusterName: kubernetes\nscheduler:\n extraArgs:\n bind-address: \"0.0.0.0\"\ncontrollerManager:\n extraArgs:\n bind-address: \"0.0.0.0\"\n cloud-provider: external\nnetworking:\n podSubnet: \"10.244.0.0\/16\"\n serviceSubnet: \"10.96.0.0\/12\"\n---\napiVersion: kubeadm.k8s.io\/v1beta3\nkind: InitConfiguration\nnodeRegistration:\n name: ip-172-31-21-29.us-west-2.compute.internal\n kubeletExtraArgs:\n cloud-provider: external\n<\/code><\/pre>\ncertSANs<\/code> instead of 52.38.15.235<\/strong><\/code> choose your controller’s public IP and change the node registration name<\/strong> to your controller’s private DNS (hostname we set in step 2). Modify the pod and service subnet if required.<\/p>\nkubeadm init --config=kubeadm.config<\/code><\/pre>\nkubeadm join 172.31.21.29:6443 --token wthapw.xgfjvonbiidea4nr --discovery-token-ca-cert-hash sha256:8b2127f960d88432fb35fd7488a501ad189e1a4ab319158d0e01a1db7fec96d7\n<\/code><\/pre>\nStep 5: Join the Worker Nodes (Only in Worker Nodes)<\/h2>\n
kubeadm-join-config.yaml<\/code>.<\/p>\n---\napiVersion: kubeadm.k8s.io\/v1beta3\nkind: JoinConfiguration\ndiscovery:\n bootstrapToken:\n token: 30am0s.hleb5xs1dyz4ridc \n apiServerEndpoint: \"172.31.21.29:6443\"\n caCertHashes:\n - \"sha256:8b2127f960d88432fb35fd7488a501ad189e1a4ab319158d0e01a1db7fec96d7\" \nnodeRegistration:\n name: ip-172-31-18-193.us-west-2.compute.internal\n kubeletExtraArgs:\n cloud-provider: external\n<\/code><\/pre>\ntoken<\/code> apiServerEndpoint<\/code>, and caCertHashes<\/code> values to the intended values that your control plane generated when it initialized. name<\/code><\/strong> should be the full hostname of the respective node.<\/p>\nkubeadm join --config kubeadm-join-config.yaml<\/code><\/pre>\nkubeadm join<\/code> command on each worker node to join with the controller node.<\/p>\nStep 6: Tag AWS Resources<\/h2>\n
kubectl config view<\/code><\/pre>\napiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: DATA+OMITTED\n server: https:\/\/172.31.21.29:6443\n name: kubernetes\ncontexts:\n- context:\n cluster: kubernetes\n user: kubernetes-admin\n name: kubernetes-admin@kubernetes\ncurrent-context: kubernetes-admin@kubernetes\nkind: Config\npreferences: {}\nusers:\n- name: kubernetes-admin\n user:\n client-certificate-data: DATA+OMITTED\n client-key-data: DATA+OMITTED\n<\/code><\/pre>\nclusters<\/code> section, the value of name<\/code> is taken as the Cluster ID.<\/p>\n kubernetes.io\/cluster\/kubernetes<\/code> and the tag value as owned<\/code> and if the resources are managed by multiple clusters, the tag key would be the same as kubernetes.io\/cluster\/kubernetes<\/code> and the tag value should be mentioned as shared<\/code>.<\/p>\n![](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-282.png\")
<\/figure>\nStep 7: Configure the Cloud Controller Manager<\/h2>\n
git clone https:\/\/github.com\/kubernetes\/cloud-provider-aws.git<\/code><\/pre>\ncd cloud-provider-aws\/examples\/existing-cluster\/base<\/code><\/pre>\nkubectl create -k .<\/code><\/pre>\nkubectl get daemonset -n kube-system<\/code><\/pre>\nkubectl get pods -n kube-system<\/code><\/pre>\nStep 8: Provision Network Load Balancer<\/h2>\n
cat <<EOF | kubectl apply -f -\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\nspec:\n replicas: 3 \n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx-container\n image: nginx:latest\n ports:\n - containerPort: 80\nEOF<\/code><\/pre>\nnginx-service-nlb.yaml<\/code>. Here the annotation service.beta.kubernetes.io\/aws-load-balancer-type: nlb<\/code> <\/strong> is very important. It tells the cloud controller manager to create a Network Load Balancer. By default, it deploys a classic load balancer.<\/p>\ncat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: Service\nmetadata:\n name: nginx-service\n annotations:\n service.beta.kubernetes.io\/aws-load-balancer-type: nlb\nspec:\n selector:\n app: nginx\n ports:\n - protocol: TCP\n port: 80\n targetPort: 80\n type: LoadBalancer\nEOF<\/code><\/pre>\nkubectl get pods<\/code><\/pre>\nkubectl get svc<\/code><\/pre>\n$ kubectl get svc nginx-service\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nnginx-service LoadBalancer 10.101.171.34 aa64c7e28c3384b5598493b6fbb04d4c-f53de39b06106733.elb.us-west-2.amazonaws.com 80:30249\/TCP 39s<\/code><\/pre>\n![\"AWS](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-2-37.png\")
<\/figure>\n![\"Accessing](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-3-42.png\")
<\/figure>\n![](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/cloud-controller-aws-1.gif\")
<\/figure>\nPossible Errors & Troubleshooting<\/h2>\n
kubectl logs -n kube-system aws-cloud-controller-manager-zwctc<\/code><\/pre>\nError 1:<\/strong> Cloud provider could not be initialized<\/h3>\n
E1028 08:39:59.394822 1 tags.go:95] Tag \"KubernetesCluster\" nor \"kubernetes.io\/cluster\/...\" not found; Kubernetes may behave unexpectedly.\nF1028 08:39:59.394858 1 main.go:106] Cloud provider could not be initialized: could not init cloud provider \"aws\": AWS cloud failed to find ClusterID\n<\/code><\/pre>\n![](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/untitled-10-1.png\")
<\/figure>\nError 2:<\/strong> node has no providerID<\/h3>\n
E1028 13:25:07.544115 1 node_controller.go:277] Error getting instance metadata for node addresses: error fetching node by provider ID: Invalid format for AWS instance (), and error by node name: could not look up instance ID for node \"ip-172-31-30-122\": node has no providerID\n<\/code><\/pre>\nroot@ip-172-31-21-29:~# hostname\nip-172-31-21-29.us-west-2.compute.internal\n<\/code><\/pre>\nError 3:<\/strong> couldn’t get current server API group list<\/h3>\n
E1030 06:04:54.829375 9140 memcache.go:265] couldn't get current server API group list: Get \"<http:\/\/localhost:8080\/api?timeout=32s>\": dial tcp 127.0.0.1:8080: connect: connection refused\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?\n<\/code><\/pre>\nadmin.conf<\/code> file in your home directory, if not make a copy using cp<\/code> command<\/p>\ncp \/etc\/kubernetes\/admin.conf ~\/.kube\/config<\/code><\/pre>\nroot@ip-172-31-21-29:~# ls -l \/etc\/kubernetes\/ ~\/.kube\/\n\/etc\/kubernetes\/:\ntotal 36\n-rw------- 1 root root 5648 Oct 30 05:17 admin.conf\n-rw------- 1 root root 5676 Oct 30 05:48 controller-manager.conf\n-rw------- 1 root root 2112 Oct 30 05:49 kubelet.conf\ndrwxr-xr-x 3 root root 4096 Oct 30 06:17 manifests\ndrwxr-xr-x 3 root root 4096 Oct 30 05:17 pki\n-rw------- 1 root root 5624 Oct 30 05:17 scheduler.conf\n\n\/root\/.kube\/:\ntotal 12\ndrwxr-x--- 4 root root 4096 Oct 30 06:07 cache\n-rw------- 1 root root 5648 Oct 30 06:07 config\n<\/code><\/pre>\nError 4:<\/strong> error execution phase preflight<\/h3>\n
[preflight] Running pre-flight checks\nerror execution phase preflight: [preflight] Some fatal errors occurred:\n [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: \/etc\/kubernetes\/kubelet.conf already exists\n [ERROR Port-10250]: Port 10250 is in use\n [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: \/etc\/kubernetes\/pki\/ca.crt already exists\n[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`\nTo see the stack trace of this error execute with --v=5 or higher\n<\/code><\/pre>\nkubelet.conf<\/code> and ca.crt<\/code>.<\/p>\nsudo rm -f \/etc\/kubernetes\/kubelet.conf\nsudo rm -f \/etc\/kubernetes\/pki\/ca.crt<\/code><\/pre>\nkubeadm join --config kubeadm-join-config.yaml --ignore-preflight-errors=Port-10250<\/code><\/pre>\nConclusion<\/h2>\n
\n