{"id":619,"date":"2025-04-07T05:22:02","date_gmt":"2025-04-07T05:22:02","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=619"},"modified":"2025-04-07T05:22:02","modified_gmt":"2025-04-07T05:22:02","slug":"private-eks-resources","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=619","title":{"rendered":"Access Private EKS Resources Using OpenVPN(complete Guide)"},"content":{"rendered":"

In this blog, you will learn to securely access private EKS resources with the help of an OpenVPN server.<\/p>\n

Setup Architecture and Workflow<\/h2>\n

When it comes to secure EKS implementation, the clusters needs to be in a fully private subnet. <\/p>\n

When you deploy applications in a Private EKS cluster, you need a bastion host or a VPN setup to seamlessly access the applications in the cluster.<\/p>\n

For example, lets say you deploy Prometheus or Loki stack in the cluster, to access the dashboards, you need to have access to the Internal Load Balancer. <\/p>\n

Here is where OpenVPN server comes in to picture. <\/p>\n

From our local workstations, the VPN server will create a secure, encrypted tunnel<\/strong> over the internet to communicate with the private networks in AWS cloud infrastrucure. <\/p>\n

We will set up an OpenVPN self-hosted server on an EC2 instance and configure it to communicate with private subnets where the EKS cluster resides. This way, the resources on that private subnets cab be accessed from our local machine.<\/p>\n

The following architecture diagram explains the private communication between the client (local workstation) and the AWS resources. <\/p>\n

\"\"
l<\/span><\/figcaption><\/figure>\n

The above diagram explains, <\/p>\n

    \n
  1. The OpenVPN community edition will be installed in the EC2 instance <\/a>(Amazon Linux 2) with the public subnet, because the server needs internet access. <\/li>\n
  2. After the installation of the VPN Server, we will generate a client configuration file for the connection implementation. <\/li>\n
  3. Once the user (Client) gets the configuration file, it will be imported into the OpenVPN Client app to establish the connection to the VPN Server, so the connection will be established.<\/li>\n
  4. Through the secure connection implemented, we can connect to the private EKS resources, which are implemented on private subnets.<\/li>\n<\/ol>\n

    Overall Setup <\/h2>\n

    We will deploy the Prometheus<\/a> stack on the private EKS cluster for testing purposes so that we will know how we are able to access its dashboard from our local machine. <\/p>\n

    But you can install anything, such as a private Nginx server pod, which is also ok for testing. <\/p>\n

    After the deployment, we will test the connection using the Node Port service as well as the internal load balancer with the help of the AWS Load Balancer Controller<\/a> and Ingress object.<\/p>\n

    Setup Prerequisites<\/h2>\n
      \n
    1. A VPC with Public Subnets (Where OpenVPN gets deployed)<\/li>\n
    2. A bastion host\/jump server deployed in the EKS VPC to access to EKS cluster.<\/li>\n
    3. A Running Private EKS cluster (>= v1.30)<\/li>\n
    4. Helm installed and configured on the local system to deploy apps to EKS.<\/li>\n<\/ol>\n

      Set OpenVPN Server<\/h2>\n

      We have covered the OpenVPN server setup in detail in a separate blog.<\/p>\n

      It covers the following topics,<\/p>\n

        \n
      1. Setting up OpenVPN on Amazon Linux<\/li>\n
      2. OpenVPN server configuration<\/li>\n
      3. OpenVPN Client Cert Configuration<\/li>\n
      4. Validating VPN Tunnel Configurations.<\/li>\n<\/ol>\n

        Refer:<\/strong> OpenVPN Server In EC2<\/a><\/p>\n

        Once you have the VPN server up and running in the public subnet, proceed to the Following.<\/p>\n

        Deploy a Bastion Server in EKS VPC<\/h2>\n

        To directly access the private EKS cluster, we need a jump server on the same VPN with internet access. <\/p>\n

        So, we create an Ubuntu server on the same VPC of the EKS cluster, but the difference is that we use the public subnet instead.<\/p>\n

        Once the instance is created, you need to add the instance profile (IAM Role) with the required permission to access the cluster, or you can add the IAM user’s credentials, such as the access key and secret access key for the cluster access.<\/p>\n

        Ensure that all the necessary tools, such as AWS CLI, Helm, Kubectl, and Eksctl, are bastion servers available in the cluster management. <\/p>\n

        Once every thing is set, use the following command to, update the kubeconfig to the bastion server.<\/p>\n

        aws eks update-kubeconfig --name <CLUSTER_NAME> --region <REGION><\/code><\/pre>\n
        Note: Add a rule in Private EKS cluster’s security group to enable port 443<\/code> to the bastion server for API access.<\/p><\/blockquote>\n
        Note: Add the instance profile or the IAM user, which is using on the bastion server for access should be updated on the EKS clusters access section.<\/p><\/blockquote>\n
        Now, we can deploy the demo apps on the private EKS cluster for the testing.<\/p>\n
        Deploy Prometheus Stack in Private EKS Cluster<\/h2>\n
        \n
        \ud83d\udccc<\/div>\n
        For demo purposes, to access multiple endpoints, we are using the Prometheus stack with Prometheus, Grafana, and Alertmanager endpoints. You can choose to deploy different apps for testing.<\/div>\n<\/div>\n
        To achieve this, we need to expose the applications with Node Port<\/strong> service, so I am creating a new values.yaml<\/code> file to deploy the Prometheus stack with Node Port service.<\/p>\n
        Create a values file for the Prometheus stack Helm deployment<\/p>\n
        cat <<EOF > node-port-values.yaml\ngrafana:\n  service:\n    type: NodePort\n    nodePort: 30000\n\nprometheus:\n  service:\n    type: NodePort\n    nodePort: 30001\n\nalertmanager:\n  service:\n    type: NodePort\n    nodePort: 30002\nEOF<\/code><\/pre>\n
        To add the Prometheus Operator repo, use the following command.<\/p>\n
        helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts<\/code><\/pre>\n
        To install the Prometheus Operator stack with a custom values file, use the following command.<\/p>\n
        helm install prometheus prometheus-community\/kube-prometheus-stack -n monitoring --create-namespace -f node-port-values.yaml<\/code><\/pre>\n
        Once the installation is completed, you can see the status of the deployments and the services.<\/p>\n
        kubectl -n monitoring get po,svc<\/code><\/pre>\n
        \"\"<\/figure>\n
        \n
        \ud83d\udca1<\/div>\n
        Note down the NodePort<\/code> number. <\/p>\n
        We need to enable these Ports on the Private EKS cluster security group to be accessed from OpenVPN server.<\/p><\/div>\n<\/div>\n
        Navigate to the EKS dashboard and open the Networking<\/code> section and click the cluster security group of the cluster.<\/p>\n
        \"\"<\/figure>\n
        Need to add three inbound rules for Grafana<\/a> (30000), Prometheus (30001), and Alertmanager (30002) NodePorts<\/code><\/strong>.<\/p>\n
        As highlighted in the following image, we need to add the port, and in the source, you need to add the security group ID of the OpenVPN server<\/strong>. This will allow the OpenVPN server to access these ports on the EKS cluster.<\/p>\n
        \"\"<\/figure>\n
        Save the security groups rules. Next step is to validate the connectivity from VPN.<\/p>\n
        Validate VPN to Cluster Connectivity<\/h2>\n
        First , ensure the OpenVPN connection is active.<\/p>\n
        \"\"<\/figure>\n
        To test the connection, we need any of the EKS cluster node’s private IP and the NodePort<\/code> number of the service.<\/p>\n
        To get the node’s IP, use the following kubectl command.<\/p>\n
        kubectl get no -o wide<\/code><\/pre>\n
        \"\"<\/figure>\n
        Now, lets try accessing the Prometheus dashboard.<\/p>\n
        Open any of the web browsers from the local machine and paste the <NODE_IP>:30001<\/code><\/strong><\/p>\n
        As shown below, you should be able to access the Prometheus dashboard using the Private IP of the EKS node.<\/p>\n
        \"\"<\/figure>\n
        For Alertmanager Dashboard, access <NODE_IP>:30002<\/code><\/strong><\/p>\n
        \"\"<\/figure>\n
        To access Grafana Dashboard, access <NODE_IP>:30003<\/code><\/strong><\/p>\n
        \"\"<\/figure>\n
        Using the IP method is ok for the testing purposes. <\/p>\n
        As you might know the EKS nodes are ephemeral. Meaning when a node gets deleted and recreated, the IP will changed. So you cannot rely on worker node IPs.<\/p>\n
        To avoid this, we need to expose the services with an Internal Load Balancer.<\/strong> Then with the load balancer IP or DNS name, we can access the applications with the static endpoint without worrying about IP changes.<\/p>\n
        Expose Applications via an Internal AWS Load Balancer<\/h2>\n
        Assuming you will have more than one applications to access, you will need a ingress based setup. Or else you will have to deploy one loadbalancer for each service which gets expensive.<\/p>\n
        For ingress to work, we need the AWS load balancer controller.<\/strong> <\/p>\n
        We have covered the installation and setup of the AWS Load Balancer Controller, in a different blog. <\/p>\n
        Refer:<\/strong> AWS Load Balancer Controller on EKS<\/a><\/p>\n
        \n
        \ud83d\udca1<\/div>\n
        Note:<\/strong><\/b> When deploying the AWS Load Balancer Controller for a private EKS cluster, you need to tag the private subnets with the following tag to enable internal load balancer provisioning:
        kubernetes.io\/role\/internal-elb: 1<\/code><\/div>\n<\/div>\n
        Update Prometheus Service to Cluster IP<\/h2>\n
        For this, we don’t have to deploy the Prometheus stack with Node Port<\/strong> service, instead we can use the Cluster IP<\/strong> service, which is also the default deployment method.<\/p>\n
        To install the Prometheus stack with default configuration, use the following command.<\/p>\n
        helm install prometheus prometheus-community\/kube-prometheus-stack -n monitoring --create-namespace<\/code><\/pre>\n
        Now, we can check the services so that we can make sure the Prometheus components are deployed in ClusterIP services.<\/p>\n
        kubectl -n monitoring get po,svc<\/code><\/pre>\n
        \"\"<\/figure>\n
        Once the AWS Load Balancer deployment is completed, an internal load balancer will automatically be provisioned when we deploy an ingress object.<\/p>\n
        Note down the Prometheus, Grafana, and Alertmanager service names and port numbers to create the ingress object.<\/p>\n
        Now, let’s create an ingress object for the Prometheus stack.<\/p>\n
        cat <<EOF > ingress.yaml\napiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: monitoring-ingress\n  namespace: monitoring\n  annotations:\n    alb.ingress.kubernetes.io\/scheme: internal  \n    alb.ingress.kubernetes.io\/target-type: ip  \nspec:\n  ingressClassName: alb\n  rules:\n    - host: grafana.techiescamp.com  \n      http:\n        paths:\n          - path: \/\n            pathType: Prefix\n            backend:\n              service:\n                name: prometheus-grafana\n                port:\n                  number: 80\n    - host: prometheus.techeiscamp.com  \n      http:\n        paths:\n          - path: \/\n            pathType: Prefix\n            backend:\n              service:\n                name: prometheus-kube-prometheus-prometheus\n                port:\n                  number: 9090\n    - host: alertmanager.techiescamp.com  \n      http:\n        paths:\n          - path: \/\n            pathType: Prefix\n            backend:\n              service:\n                name: prometheus-kube-prometheus-alertmanager\n                port:\n                  number: 9093\nEOF<\/code><\/pre>\n
        Here, I have used host names for each of the services, so we can easily remember the names.<\/p>\n
        Note: If you already have a domain in Route53 or GoDaddy, use that one for the application.<\/p><\/blockquote>\n
        To deploy the Ingress object, use the following command.<\/p>\n
        kubectl apply -f ingress.yaml<\/code><\/pre>\n
        To check the Ingress object,<\/p>\n
        kubectl -n monitoring get ingress<\/code><\/pre>\n
        To describe the Ingress object.<\/p>\n
        kubectl -n monitoring describe ingress monitoring-ingress<\/code><\/pre>\n
        \"\"<\/figure>\n
        We can see that the Ingress is successfully created, which means the Internal Load Balancer was also automatically created.<\/p>\n
        Let’s open the AWS console and ensure the Load Balancer is properly provisioned.<\/p>\n
        \"\"<\/figure>\n
        Note down the Internal Load Balancer DNS name for the upcoming configuration.<\/p>\n
        When we deploy the ingress and Load Balancer creation, a new rule will be added to the EKS cluster security group to expose the Ports of the ingress services.<\/p>\n
        \"\"<\/figure>\n
        We need to expose these same ports in the OpenVPN server’s security group.<\/p>\n
        \"\"<\/figure>\n
        Local DNS Resolution for ILB<\/h2>\n
        We can map the DNS records of the AWS Internal Load Balancer to the DNS servers if we have, or for testing purposes we can use the Local DNS resolution.<\/p>\n
        For this, first we need to identify the IP addresses of the Internal Load Balancer.<\/p>\n
        nslookup <INTERNAL_LOADBALANCER_DNS_NAME><\/code><\/pre>\n
        \"\"<\/figure>\n
        Copy the IP address and paste it to the \/etc\/hosts<\/code> of your local machine and map the host names that we mentioned in the Ingress object.<\/p>\n
        sudo vim \/etc\/hosts<\/code><\/pre>\n
        \"\"<\/figure>\n
        This will help to resolve the DNS query from the local machine.<\/p>\n
        Note: If you are using to map with DNS servers, make sure, you have private hosted zone.<\/p><\/blockquote>\n
        To access the dashboard, open any of the web browser and paste the hostname.<\/p>\n
        Prometheus Dashboard<\/h3>\n
        \"\"<\/figure>\n
        Grafana Dashboard<\/h3>\n
        \"\"<\/figure>\n
        Alertmanager Dashboard<\/h3>\n
        \"\"<\/figure>\n
        Conclusion<\/h2>\n
        We have implemented the OpenVPN server to access the private EKS resources. This has created an encrypted tunnel over the Internet for secure communication. <\/p>\n
        This self-hosted VPN Server method is cost-effective for testing and small projects, especially since we are using the open-source version of OpenVPN, which has some limitations.<\/p>\n
        So if you want to implement for a big project or want the complete benefits, you can go with the enterprise edition or can use the AWS client VPN endpoint solution.<\/p>\n
        \n
        Ngu\u1ed3n:<\/strong> Access Private EKS Resources Using OpenVPN(complete Guide) \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
        Source: https:\/\/devopscube.com\/private-eks-resources\/<\/p>\n","protected":false},"author":1,"featured_media":620,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-619","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=619"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/619\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/620"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}