{"id":741,"date":"2023-12-17T11:25:10","date_gmt":"2023-12-17T11:25:10","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=741"},"modified":"2023-12-17T11:25:10","modified_gmt":"2023-12-17T11:25:10","slug":"secondary-network-eks-cluster","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=741","title":{"rendered":"How to Setup VPC Secondary Network For EKS Cluster"},"content":{"rendered":"

In EKS IPV4 clusters, IP exhaustion problem can be solved by setting up a VPC secondary network. In this blog we will look at step by step guide for setting up EKS cluster using secondary IP ranges.<\/p>\n

Need for VPC Secondary Network<\/h2>\n

Before we jump to the hands on section, lets first understand the need for VPC<\/a> secondary network configuration for IPV4 clusters.<\/p>\n

When you launch a EKS cluster, by default it uses the VPC subnet IP addresses<\/strong> for pods and services. Meaning, if the VPC CIDR is 10.0.0.0\/20<\/code><\/strong> (4,096<\/strong> IPs), the node IPs, pod IPs and service IPs are assigned from CIDR range.<\/p>\n

It is not an issue for small clusters with less workloads. However, for large organizations using IPV4 with hybrid cloud networks this could be an issue.<\/p>\n

In such organizations when you request for large IP ranges for Kubernetes clusters<\/a>., network teams often express concerns. Because they need to allocate IP address for multiple clusters in multiple environments and for multiple teams.<\/p>\n

You can mitigate the IPv4 exhaustion issue by using AWS VPC secondary IP ranges such as shared address space<\/a> 100.64.0.0\/16<\/code><\/strong> or 172.x series.<\/p>\n

If you look at the following image, the Primary ENI of worker nodes<\/strong> use the primary subnet CIDR and pods uses the ENI from the secondary CIDR range.<\/p>\n

\"AWS<\/figure>\n

Image src: aws.github.io<\/p>\n

A good thing is, you can re-use the secondary ranges<\/strong> in other clusters and the primary ENI does the source network translation and the traffic going outside the cluster uses the primary ENI as source address. So you will not face any routing issues.<\/p>\n

Note<\/strong>: You can use CNI plugins like calico to mitigate this issue. But for external CNI plugins you will not get any support from AWS.<\/p><\/blockquote>\n

Secondary Network setup is not required for the following.<\/p>\n

    \n
  1. If you are implementing IPv6 based clusters.<\/li>\n
  2. Private NAT<\/a> Gateway implementation where VPCs can have overlapping IP addresses.<\/li>\n<\/ol>\n

    EKS Secondary Network Setup<\/h2>\n

    Let’s have a look at what is the secondary network in the Kubernetes cluster.<\/p>\n

    Here we use a single dedicated VPC with two different CIDR ranges, which I am considering 192.168.0.0\/16<\/code> as a primary range for nodes and 100.64.0.0\/16<\/code> as the secondary range for the pods.<\/p>\n

    Step 1: Create a VPC and Associate a Secondary CIDR<\/h3>\n

    We are using AWS CLI<\/a> commands to create the VPC<\/a>. So some utilities required are given below.<\/p>\n

      \n
    1. AWS CLI<\/li>\n
    2. jq<\/li>\n<\/ol>\n

      Create a VPC with a CIDR range 192.168.0.0\/16<\/code>. We use this range to create the nodes.<\/p>\n

      export VPC_ID=$(aws ec2 create-vpc --cidr-block 192.168.0.0\/16 --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=EKS-VPC}]' --query 'Vpc.VpcId' --output text)\n<\/code><\/pre>\n
      Add secondary CIDR 100.64.0.0\/16<\/code> to the VPC. This range is for creating pods.<\/p>\n
      aws ec2 associate-vpc-cidr-block --vpc-id $VPC_ID --cidr-block 100.64.0.0\/16<\/code><\/pre>\n
      Create an Internet Gateway for the VPC to get internet access.<\/p>\n
      export IGW_ID=$(aws ec2 create-internet-gateway --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=EKS-IGW}]' --query 'InternetGateway.InternetGatewayId' --output text)<\/code><\/pre>\n
      Attach Internet Gateway to the intended VPC<\/p>\n
      aws ec2 attach-internet-gateway --vpc-id ${VPC_ID} --internet-gateway-id ${IGW_ID}<\/code><\/pre>\n
      Create a route table for our subnets.<\/p>\n
      export RTB_ID=$(aws ec2 create-route-table --vpc-id $VPC_ID --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=EKS-RouteTable}]' --query 'RouteTable.RouteTableId' --output text)<\/code><\/pre>\n
      Add route to the internet by attaching IGW in the route table.<\/p>\n
      aws ec2 create-route --route-table-id ${RTB_ID} --destination-cidr-block 0.0.0.0\/0 --gateway-id ${RTB_ID}<\/code><\/pre>\n
      Set up environment variables for availability zones.<\/p>\n
      AZ_1=\"us-west-2a\"\nAZ_2=\"us-west-2b\"\nAZ_3=\"us-west-2c\"<\/code><\/pre>\n
      Create subnets for nodes from range 192.168.0.0\/16<\/code>, each in a different availability zone.<\/p>\n
      SUBNET_ID_NODE_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 192.168.0.0\/19 --availability-zone $AZ_1 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Node-Subnet-1}]' | jq -r '.Subnet.SubnetId')\n\nSUBNET_ID_NODE_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 192.168.32.0\/19 --availability-zone $AZ_2 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Node-Subnet-2}]' | jq -r '.Subnet.SubnetId')\n\nSUBNET_ID_NODE_3=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 192.168.64.0\/19 --availability-zone $AZ_3 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Node-Subnet-3}]' | jq -r '.Subnet.SubnetId')<\/code><\/pre>\n
      Create three subnets for pods from CIDR 100.64.0.0\/16<\/code>.<\/p>\n
      SUBNET_ID_POD_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 100.64.0.0\/19 --availability-zone $AZ_1 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Pod-Subnet-1}]' | jq -r '.Subnet.SubnetId')\n\n\nSUBNET_ID_POD_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 100.64.32.0\/19 --availability-zone $AZ_2 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Pod-Subnet-2}]' | jq -r '.Subnet.SubnetId')\n\n\nSUBNET_ID_POD_3=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 100.64.64.0\/19 --availability-zone $AZ_3 --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=Pod-Subnet-3}]' | jq -r '.Subnet.SubnetId')<\/code><\/pre>\n
      Associate each subnet with the routing table.<\/p>\n
      \naws ec2 associate-route-table --subnet-id $SUBNET_ID_NODE_1 --route-table-id $RTB_ID\naws ec2 associate-route-table --subnet-id $SUBNET_ID_NODE_2 --route-table-id $RTB_ID\naws ec2 associate-route-table --subnet-id $SUBNET_ID_NODE_3 --route-table-id $RTB_ID\n\naws ec2 associate-route-table --subnet-id $SUBNET_ID_POD_1 --route-table-id $RTB_ID\naws ec2 associate-route-table --subnet-id $SUBNET_ID_POD_2 --route-table-id $RTB_ID\naws ec2 associate-route-table --subnet-id $SUBNET_ID_POD_3 --route-table-id $RTB_ID\n<\/code><\/pre>\n
      Enable auto-assign public IPv4 address<\/code> to all subnets. this is required to automatically assign a public IP address to all nodes.<\/p>\n
      aws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_NODE_1 --map-public-ip-on-launch\naws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_NODE_2 --map-public-ip-on-launch\naws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_NODE_3 --map-public-ip-on-launch\n\naws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_POD_1 --map-public-ip-on-launch\naws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_POD_2 --map-public-ip-on-launch\naws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID_POD_3 --map-public-ip-on-launch<\/code><\/pre>\n
      If you are dont enable the Enable auto-assign public IPv4 address<\/code> option, the nodes won’t be attached with the EKS cluster.<\/p><\/blockquote>\n
      Step 2: Create an EKS Cluster<\/h3>\n
      We use eksctl<\/code> command line utility to create a cluster in EKS<\/a>, make sure you have already installed this in your local system.<\/p>\n
      Create a cluster config file to build an EKS cluster<\/p>\n
      vim eks-cluster.yml<\/code><\/pre>\n
      Modify the details as per your requirements.<\/p>\n
      Here the EKS version we use is v1.28<\/code>, the latest one, and subnets, created from 192.168.0.0\/16<\/code> CIDR for nodes.<\/p>\n
      In the nodeGroups<\/code> section, choose your instance type, desired capacity, and your public key name. For this setup, we use t3.medium<\/code> instance type with two nodes.<\/p>\n
      apiVersion: eksctl.io\/v1alpha5\nkind: ClusterConfig\n\nmetadata:\n  name: custom-cluster\n  region: us-west-2\n  version: \"1.28\"\n\nvpc:\n  subnets:\n    public:\n      us-west-2a: $SUBNET_ID_NODE_1\n      us-west-2b: $SUBNET_ID_NODE_2\n      us-west-2c: $SUBNET_ID_NODE_3\n\nnodeGroups:\n  - name: ng-spot\n    instanceType: t3.medium\n    labels: { role: builders }\n    minSize: 2\n    maxSize: 4\n    volumeSize: 30\n    desiredCapacity: 2\n    ssh:\n      allow: true\n      publicKeyName: techiescamp\n    tags:\n      Name: ng-spot\n<\/code><\/pre>\n
      Step 3: Configure AWS VPC CNI Plugin<\/h3>\n
      The VPC CNI Plugin is an add-on that will be available in each node of the clusters. basically, it is a deamonset<\/strong> named aws-node<\/code>, and will be present in each node. This plugin attaches the Elastic Network Interface (ENI) to the nodes and assigns private IPs for pods from the VPC.<\/p>\n
      We have to make some modifications in the CNI configuration for pods to use the secondary CIDR range.<\/p>\n
      Prerequisites:<\/strong><\/p>\n