Offload Secret Management to AWS Secrets Manager from Amazon EKS

Table of Contents

Introduction

Secrets in Kubernetes, are Base 64 encoded. As such, its trivial for anyone with access to the secret objects, to decode them & fetch the secret value.

In Amazon EKS, the managed Kubernetes control plane runs on EC2 instances with EBS volumes that are encrypted at rest. So when your secrets are created in etcd & saved to disk, they’re encrypted at rest. However, within the cluster, they’re still essentially plaintext.

A better way to keep secrets “secret”, would be to have them encrypted, preferably with a key of your choosing, until they’re needed in a workload pod. To achieve this, you can store your secrets in either AWS Secrets Manager or SSM Parameter Store (as SecureString), instead of within the EKS cluster.

This article describes how to consume secrets from Secrets Manager & Parameter Store in EKS pods.

How It Works

The Kubernetes community has created a CSI driver to facilitate the use of secrets from external secret stores like AWS Secrets Manager & HashiCorp Vault, in Kubernetes pods.

The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, & certs stored in enterprise-grade external secret stores into Kubernetes pods as a volume.

— Secrets Store CSI Driver

Every secret store has a “provider” for the CSI driver, which is how you use a particular secret store with the driver. To consume secrets from AWS Secrets Manager & parameters from AWS Systems Manager Parameter Store, as mounted volumes in EKS pods, you can use the AWS Secrets and Configuration Provider (ASCP) for Kubernetes Secrets Store CSI Driver.

Install ASCP Helm Chart

To get started, install the ASCP Helm chart in your cluster:

helm repo add eks https://aws.github.io/eks-charts

helm install aws-secrets-provider \
eks/csi-secrets-store-provider-aws \
--namespace kube-system

Or if you prefer Flux GitOps:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: eks
  namespace: flux-system
spec:
  interval: 30m
  url: https://aws.github.io/eks-charts
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aws-secrets-provider
  namespace: kube-system
spec:
  releaseName: aws-secrets-provider
  chart:
    spec:
      chart: csi-secrets-store-provider-aws
      version: 0.0.3
      sourceRef:
        kind: HelmRepository
        name: eks
        namespace: flux-system
  interval: 30m

Create Secrets & Parameters & SecretProviderClass

Create the secrets & parameters in AWS & a SecretProviderClass in the namespace where your workload will be deployed:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-secret-provider
  namespace: my-namespace
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "my-secret"
        objectType: "secretsmanager"
      - objectName: "my-parameter"
        objectType: "ssmparameter"

Grant Permissions to Access Secrets

Create an AWS IAM policy to allow access to the secret & parameter:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": [
                "arn:aws:secretsmanager:*:*:secret:my-secret-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:parameter/my-parameter"
            ]
        }
    ]
}

Then create a Kubernetes service account with this policy, so the workloads using this service account, can access the secret & parameter:

eksctl create iamserviceaccount \
--name my-service-account --cluster my-cluster \
--attach-policy-arn arn:aws:iam::my-account:policy/my-policy \
--namespace my-namespace --approve --override-existing-serviceaccounts

Deploy Workload

Finally, create a Pod to consume the secret & parameter:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: my-secret-provider
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: secrets
      mountPath: /mnt/secrets
      readOnly: true

To view the secrets, exec into the Pod and:

# cd /mnt/secrets/

# ls
my-secret
my-parameter

# cat my-secret
my-secret-value

# cat my-parameter
my-parameter-value

Conclusion

So there you go: secrets & parameters right in your pods. You can use your own KMS CMKs to encrypt the secrets & parameters & as long as the IAM policy has the right permissions, you can fetch them into your workload pods.

Additionally, if you use the secret rotation feature of AWS Secrets Manager, you can configure the ASCP to reconcile secrets with your pods periodically. See the resources below for more details. Note that this feature is currently in alpha.

Resources

Check out the following resources to learn more about secrets in Kubernetes:

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Leave a Reply

Your email address will not be published. Required fields are marked *