{"id":1011,"date":"2025-01-07T08:04:52","date_gmt":"2025-01-07T08:04:52","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=1011"},"modified":"2025-01-07T08:04:52","modified_gmt":"2025-01-07T08:04:52","slug":"secrets-store-csi-dirver-eks","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=1011","title":{"rendered":"Manage Secrets on AWS EKS using Secrets Store CSI Driver"},"content":{"rendered":"

In this blog, you will learn to integrate AWS Secrets Manager secrets into the EKS cluster using Secrets Store CSI Driver.<\/p>\n

Keeping the Secrets secure on Kubernetes<\/a> is an important task, whether it’s an API key, TLS certificate, or authentication credentials. This is where Secrets Store CSI Driver comes in.<\/p>\n

What is Secrets Store CSI Driver?<\/h2>\n

Secrets Store CSI Driver<\/a> is a Kubernetes driver deployed as a DaemonSet<\/a>. It integrates secrets stored in external secrets management tools and mounts secrets on pods<\/a> as a volume.<\/p>\n

It uses the Container Storage Interface (CSI)<\/strong> to interact with secret providers like Azure Key Vault, HashiCorp Vault<\/a>, AWS Secrets Manager, and others.<\/p>\n

It is primarily used when you want to securely use sensitive data like API keys, passwords, or certificates in your Kubernetes applications without exposing them in Kubernetes Secrets or ConfigMaps.<\/p>\n

Secrets Store CSI Driver Workflow<\/h2>\n

Below is the diagrammatic workflow of the Secret Stores CSI Driver, which gets secrets from the AWS Secrets Manager and mounts them in a pod.<\/p>\n

\"Secrets
Click to View in HD<\/span><\/figcaption><\/figure>\n

Here is how it works.<\/p>\n

    \n
  1. The pod initiates the process by defining the SecretProviderClass<\/code> object and uses a service account<\/a> with the necessary permissions to authenticate Secrets Manager.<\/li>\n
  2. The SecretProviderClass<\/code> object<\/a> contains the details of the secret stored in secrets manager..<\/li>\n
  3. The CSI driver uses the secret details on SecretProviderClass<\/code> to fetch the secret from the external secret store and mounts it inside the pod as a file.<\/li>\n<\/ol>\n

    Setup Prerequisites<\/h2>\n

    The prerequisites for the setup are given below.<\/p>\n

      \n
    1. kubectl<\/li>\n
    2. eksctl<\/li>\n
    3. AWS CLI<\/a> with access to IAM and EKS<\/li>\n
    4. EKS cluster<\/a> with OIDC assigned to it<\/li>\n
    5. Helm<\/a><\/li>\n<\/ol>\n

      Secrets Store CSI Driver Setup on EKS<\/h2>\n

      If you are ready with the prerequisites, follow the steps below to set up a Secrets Store CSI Driver on EKS.<\/p>\n

      Step 1: Create an IAM Policy<\/h3>\n

      We need an IAM policy to access the secrets in secrets manager through the pods service account.<\/p>\n

      Create a policy with permission to get secrets from the External Secrets, this policy will only contain read permission.<\/p>\n

      Use the policy command to create a JSON file with permission for the policy.<\/p>\n

      cat << EOF > policy.json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"AllowAccessToSecretsManager\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"secretsmanager:GetSecretValue\",\n                \"secretsmanager:DescribeSecret\"\n            ],\n            \"Resource\": \"arn:aws:secretsmanager:$Region:$Account-ID:secret:$Secret-Name\"\n        }\n    ]\n}\nEOF<\/code><\/pre>\n
      Update the region, account ID, and secret name before running the above command.<\/p>\n
      This policy gives read-only permission to the specific secret you specify on the policy.<\/p>\n
      Note:<\/strong> If you want to give permission for all the secrets on a region, use ‘*’ instead of a specific secret name.<\/p>\n
      For example: “arn:aws:secretsmanager:$Region:$Account-ID<\/strong>:secret:*<\/strong>“<\/p>\n
      Also, giving permissions to all secrets is not a recommended practice, especially in production environments.<\/p><\/blockquote>\n
      Run the following command to create the policy with the policy.json<\/code><\/strong> file.<\/p>\n
      aws iam create-policy \\\n    --policy-name SecretStoreCSIDriverPolicy \\\n    --policy-document file:\/\/policy.json<\/code><\/pre>\n
      Once the policy is created, run the following command to get the policy’s ARN and save it as a variable.<\/p>\n
      export POLICY_ARN=$(aws iam list-policies --query \"Policies[?PolicyName=='SecretStoreCSIDriverPolicy'].Arn\" --output text)<\/code><\/pre>\n
      Now, run the following command to check if the ARN is saved as a variable.<\/p>\n
      echo $POLICY_ARN<\/code><\/pre>\n
      This command will show the policy\u2019s ARN.<\/p>\n
      Step 2: Install CSI Driver and AWS Provider<\/h3>\n
      The next step is to install the Secrets Store CSI Driver and AWS provider for the CSI Driver using Helm.<\/p>\n
      Run the following command to add the helm repo of Secrets Store CSI Driver on your system.<\/p>\n
      helm repo add secrets-store-csi-driver https:\/\/kubernetes-sigs.github.io\/secrets-store-csi-driver\/charts<\/code><\/pre>\n
      Before installing the CSI Driver, download the values.yaml<\/code> file and enable secret auto-rotation because, by default, the secret auto-rotation is disabled.<\/p>\n
      Download the values.yaml<\/code> using the following command.<\/p>\n
      helm show values secrets-store-csi-driver\/secrets-store-csi-driver > value.yaml<\/code><\/pre>\n
      Now enable the following, as shown in the image below.<\/p>\n
      \"modify<\/figure>\n
      As shown in the above image enable syncSecret<\/code><\/strong> to save secrets as Kubernetes secrets when a pod defines it. It is an optional parameter. By default the CSI driver mounts the secrets as volumes in pods.<\/p>\n
      With syncSecret<\/code><\/strong> the Kubernetes secret will only be available when the pod is deployed. Once the pod is deleted, the secret will also be deleted from the cluster.<\/p>\n
      The enableSecretRotation<\/code><\/strong> option syncs the secret on external secrets with the mounted secret whenever the secret is updated on the external secrets manager.<\/p>\n
      The rotationPollInterval<\/code><\/strong> option is the time interval for syncing the secret with the external secret and secret in the cluster, we have set the time to 1 minute which means it will sync the secret every 1 minute.<\/p>\n
      By default, the poll interval is 2 minutes, if you don’t specify a poll interval, it will sync every 2 minutes, and this option will only available if you enable enableSecretRotation<\/code><\/strong>.<\/p>\n
      Then, run the following command to install the CSI Driver using Helm.<\/p>\n
      helm install -n kube-system csi-secrets-store secrets-store-csi-driver\/secrets-store-csi-driver --values values.yaml <\/code><\/pre>\n
      Now, to install the AWS provider for the CSI Driver, add the helm repo for the AWS provider on your system using the following command.<\/p>\n
      helm repo add aws-secrets-manager https:\/\/aws.github.io\/secrets-store-csi-driver-provider-aws<\/code><\/pre>\n
      Run the following command to install the AWS provider using Helm.<\/p>\n
      helm install -n kube-system secrets-provider-aws aws-secrets-manager\/secrets-store-csi-driver-provider-aws<\/code><\/pre>\n
      Or you can also install the AWS provider using YAML and run the following command to install the AWS provider using YAML.<\/p>\n
      kubectl apply -f https:\/\/raw.githubusercontent.com\/aws\/secrets-store-csi-driver-provider-aws\/main\/deployment\/aws-provider-installer.yaml<\/code><\/pre>\n
      Step 3: Create a Service Account<\/h3>\n
      Now, create a Service Account<\/a> using a namespace where you want to use the secret.<\/p>\n
      Before creating the service account, make sure your EKS cluster has OIDC<\/strong> associated with it, if not, run the following command to enable OIDC<\/strong> for your cluster.<\/p>\n
      eksctl utils associate-iam-oidc-provider --cluster $Cluster-Name --approve<\/code><\/pre>\n
      I am going to create a service account on the namespace web; <\/strong>this is where I am going to use the secret.<\/p>\n
      If you want to create a namespace, run the following command.<\/p>\n
      kubectl create ns web<\/code><\/pre>\n
      Replace the namespace name if needed.<\/p>\n
      Now, run the following command to create a service account with the policy you created on step 1<\/strong>.<\/p>\n
      eksctl create iamserviceaccount \\\n--cluster=$Cluster-name \\\n--namespace=web \\\n--name=csi-sa \\\n--attach-policy-arn $POLICY_ARN \\\n--approve<\/code><\/pre>\n
      Update the cluster name before running the above command. Also, change the namespace if you are using another namespace.<\/p>\n
      You don’t have to worry about creating an IAM role<\/a> and attaching the policy to the role, OIDC will take care of it automatically.<\/p>\n
      When you run the command, OIDC will create a role for the policy and link it to the service account.<\/p>\n
      Step 4: Create a SecretProviderClass<\/h3>\n
      The next step is to create a SecretProviderClass<\/code>. Think of this as a connection between a pod and the CSI driver.<\/p>\n
      SecretProviderClass has the details of the secret that it needs to fetch and where it should be mounted.<\/p>\n
      Whenever a pod defines the SecretProviderClass to use the secret, the CSI driver fetches the secret from the external secret store<\/strong> and mounts it inside the pod as a file.<\/p>\n
      The secret that is stored on my AWS Secrets Manager is given below.<\/p>\n
      \"secret<\/figure>\n
      The above image shows how the secret details are specified in the SecretProviderClass<\/code> manifest file.<\/p>\n
      Create a file secret-class.yaml<\/code><\/strong> and copy the below content.<\/p>\n
      apiVersion: secrets-store.csi.x-k8s.io\/v1\nkind: SecretProviderClass\nmetadata:\n  name: external-secrets\n  namespace: web\nspec:\n  provider: aws\n  parameters:\n    objects: |\n      - objectName: \"testing-secrets-manager\"\n        objectType: \"secretsmanager\"\n        jmesPath:\n          - path: \"secret\"\n            objectAlias: \"secrets-manager-secret\"\n  secretObjects:\n    - secretName: external-secrets\n      type: Opaque\n      data:\n        - objectName: \"secrets-manager-secret\"\n          key: \"secret\"<\/code><\/pre>\n
      Change the namespace if you are using another namespace.<\/p>\n
      In the above file:<\/p>\n
      The provider is selected as AWS because I am using AWS cloud, if you are using any other cloud specify that as provider.<\/p>\n
      The objectName<\/strong> is the name of the secret stored in the Secret Manager that you want to use.<\/p>\n
      The objectType<\/strong> is specified as secretsmanager because I am fetching secrets from AWS Secrets Manager.<\/p>\n
      jmesPath<\/strong> is a query language which is used to filter and transfer JSON data. In our case it fetched the specified secret from the secrets manager and mount it in the pod.<\/p>\n
      Under the jmespath<\/strong>, path<\/strong> is the name of the secret key on secrets manager which you want to fetch and objectAlias<\/strong> is the name of the file where the fetched secret value will be mounted inside the pod.<\/p>\n
      And you can see the secretObjects<\/strong> block, it will only work if you enable syncSecret<\/strong> in step 2.<\/p>\n
      It is the block that creates a Kubernetes secret of the external secret.<\/p>\n
      Run the following command to create the SecretProviderClass.<\/p>\n
      kubectl apply -f secret-class.yaml<\/code><\/pre>\n
      Then run the following command to verify if it’s created.<\/p>\n
      kubectl get secretproviderclass -n web<\/code><\/pre>\n
      You will get an output as shown below.<\/p>\n
      \"verify<\/figure>\n
      Step 5: Test Mounting the Secret on a Pod<\/h3>\n
      Now that the setup has been completed, test the setup by mounting the secret on a pod as volume to check if it’s fetching and mounting the secret on the pod.<\/p>\n
      Create a file deploy.yaml <\/code>and copy the below content.<\/p>\n
      apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: csi-secret-app\n  namespace: web\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: web-app\n  template:\n    metadata:\n      labels:\n        app: web-app\n    spec:\n      serviceAccountName: csi-sa\n      containers:\n        - name: app-container\n          image: nginx:latest\n          volumeMounts:\n            - name: secrets-store\n              mountPath: \"\/mnt\/secrets\"\n              readOnly: true\n      volumes:\n        - name: secrets-store\n          csi:\n            driver: secrets-store.csi.k8s.io\n            readOnly: true\n            volumeAttributes:\n              secretProviderClass: external-secrets<\/code><\/pre>\n
      This file creates a deployment with 1 replica on the web namespace and mounts the secret as volume.<\/p>\n
      If you notice, the volumes<\/code> definition in the pod manifest tells the CSI driver to use the secretProviderClass external-secrets<\/code> to get the information about the secret.<\/p>\n
      When the pod is created, Kubelet makes a NodePublishVolume call to the CSI driver, which tells the CSI driver to fetch the secret using the secret information in the specified secretProviderClass<\/code> and use the service account csi-sa<\/code> for authentication.<\/p>\n
      If you are wondering what NodePublishVolume<\/code> call is, it is a function of CSI that makes sure that the volume is mounted in the requested node for the requested pod to access.<\/p>\n
      Run the following command to apply the manifest file.<\/p>\n
      kubectl apply -f deploy.yaml<\/code><\/pre>\n
      Once the pod is up and running, verify if the secret is mounted in the pod using the below exec command.<\/p>\n
      kubectl exec -n web csi-secret-app-68454658fd-gzd55 -- cat \/mnt\/secrets\/secrets-manager-secret<\/code><\/pre>\n
      Replace csi-secret-app-68454658fd-gzd55<\/strong> with your pod names.<\/p>\n
      If everything is okay, you can see your secret below.<\/p>\n
      \"check<\/figure>\n
      You can see the secret is saved inside the mounted file secrets-manager-secret,<\/strong> which you specified on the SecretProviderClass<\/strong>.<\/p>\n
      testing external secrets manager<\/strong> is the secret value I stored inside my SecretsManager.<\/p>\n
      If you enable the syncSecret<\/code> step and specify secretObjects on the SecretProviderClass <\/strong>manifest file, you can also see a secret created on the same namespace, so run the following command to list it.<\/p>\n
      kubectl get secret -n web<\/code><\/pre>\n
      Step 6: Cleanup<\/h3>\n
      If you no longer need to use the setup and want to clean it up, run the following commands.<\/p>\n
      Delete the SecretProviderClass and Service Account using the following commands.<\/p>\n
      kubectl delete SecretProviderClass external-secrets -n web\n\nkubectl delete sa csi-sa<\/code><\/pre>\n
      To delete the helm deployment of CSI Driver and the AWS provider, run the following command.<\/p>\n
      helm delete -n kube-system secrets-provider-aws\n\nhelm delete -n kube-system csi-secrets-store<\/code><\/pre>\n
      Finally, delete the namespace using the following command.<\/p>\n
      kubectl delete ns web<\/code><\/pre>\n
      Real-World Use Cases<\/h2>\n
      Given below are a few real-world use cases of Secrets Store CSI drivers.<\/p>\n
        \n
      1. Microservices applications will have separate databases for each service, they use secrets store CSI driver to get the secret from a central secret manager and mount it in the deployments.<\/li>\n
      2. Secrets Store CSI drivers also have the feature to store secrets as Kubernetes secrets, which will be useful for managing TLS\/SSL certificates for Ingress controllers.<\/li>\n
      3. The automatic Secret Rotation feature makes sure secrets are updated without manual intervention.<\/li>\n
      4. It plays a major role in the CI\/CD pipeline by mounting the secret on pipeline pods, which can be authentication credentials for Docker registered, Cloud platform, etc.<\/li>\n<\/ol>\n
        Best Practices<\/h2>\n
        The best practices for Secret Store CSI driver are given below.<\/p>\n
          \n
        1. Install the driver on the kube-system<\/strong> namespace so that only the cluster admin can access it.<\/li>\n
        2. Use RBAC to restrict users from modifying or accessing the SecretProviderClassPodStatus CRD<\/strong> resources, make sure only the cluster admin can access it.
          Because it has information about where the secrets are mounted.<\/li>\n
        3. Always apply the least privilege principle for the service account, only give permission that they need.
          For example, only provide permission to read the specific secret you need, not all the secrets in the External Secrets.<\/li>\n
        4. Always keep the driver version up to date.<\/li>\n
        5. Enable encryption for secrets. For example, in AWS Secrets Manager<\/strong>, use the KMS <\/strong>feature to encrypt the secrets.<\/li>\n<\/ol>\n
          Conclusion<\/h2>\n
          In this blog, we have seen about setting up Secrets Store CSI Driver on the AWS EKS cluster and using it to get the secrets from AWS Secrets Manager and mount it on a pod.<\/p>\n
          We have also seen how to check if the secrets are mounted correctly and how to use commands to clean up the setup once they are no longer needed.<\/p>\n
          If you want to learn about similar tools, take a look at the following blogs:<\/p>\n
          Kubernetes External secrets operator<\/a> is used to fetch secrets from the AWS Secrets Manager.<\/p>\n
          \n
          Ngu\u1ed3n:<\/strong> Manage Secrets on AWS EKS using Secrets Store CSI Driver \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
          Source: https:\/\/devopscube.com\/secrets-store-csi-dirver-eks\/<\/p>\n","protected":false},"author":1,"featured_media":1012,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1011","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/1011","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1011"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/1011\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/1012"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1011"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1011"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1011"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}