Migrate Amazon EKS Apps from NFS CSI to NetApp Trident CSI Driver

In one of our projects here at QloudX, we run hundreds of microservices in an Amazon EKS cluster, with persistent backend storage provided by an Amazon FSx for NetApp ONTAP file system. The official Container Storage Interface (CSI) driver for ONTAP is Trident. However, Trident did not support Bottlerocket until recently & all our EKS nodes run Bottlerocket OS. So we used NFS CSI to mount FSx volumes as NFS volumes in EKS pods. This article describes how we migrated from NFS CSI to Trident when Trident added support for Bottlerocket.

Install Trident CSI

Trident can co-exist with NFS CSI in an EKS cluster. Start by installing it via Helm. Using Flux for GitOps, create the Helm repo & Helm release:

kind: HelmRepository
apiVersion: source.toolkit.fluxcd.io/v1
metadata:
  name: trident
  namespace: trident
spec:
  interval: 1h
  url: https://netapp.github.io/trident-helm-chart
kind: HelmRelease
apiVersion: helm.toolkit.fluxcd.io/v2
metadata:
  name: trident
  namespace: trident

spec:
  interval: 1h
  chart:
    spec:
      sourceRef:
        name: trident
        kind: HelmRepository
      chart: trident-operator
      version: 100.2510.0 # Trident 25.10.0 from October 2025

  values:
    cloudProvider: AWS
    cloudIdentity: "'eks.amazonaws.com/role-arn: arn:aws:iam::<AWS-account-ID>:role/<role-name>'"
    tridentImage: <AWS-account-ID>.dkr.ecr.<AWS-region>.amazonaws.com/netapp/trident:25.10.0

The Trident container image above, is hosted in Amazon ECR. The IAM role grants Trident pods access to FSx using EKS’s “IAM roles for service accounts” (IRSA) feature.

Trident Backend Config

To connect Trident to FSx, create a Trident backend config:

kind: TridentBackendConfig
apiVersion: trident.netapp.io/v1
metadata:
  name: fsx
  namespace: trident

spec:
  svm: fsx
  version: 1
  backendName: fsx
  storageDriverName: ontap-nas
  aws:
    fsxFilesystemID: fs-<random-ID>
  credentials:
    type: awsarn
    name: arn:aws:secretsmanager:<AWS-region>:<AWS-account-ID>:secret:<secret-name>

The admin credentials for Trident to log into the FSx file system’s Storage Virtual Machine (SVM), are stored in AWS secrets manager as {"username":"vsadmin","password":"…"}.

Trident Storage Class

Finally, create a Kubernetes storage class for Trident, to be used by any Persistent Volume Claims (PVCs) that need Trident:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fsx

parameters:
  fsType: nfs
  backendType: ontap-nas
  storagePools: fsx:.*

allowVolumeExpansion: true
volumeBindingMode: Immediate
provisioner: csi.trident.netapp.io

Replace NFS PVCs with Trident PVCs

This is the challenging part. You cannot just replace an NFS PVC’s storage class name with Trident’s. Most PVC fields are immutable after creation. The only way is to delete & recreate the PVC. Also, PVC deletion is blocked until all pods using it, are also deleted. We chose to tackle this as follows:

  • First, create a Kyverno policy in EKS that intercepts PVC create/update & modifies the PVC to use Trident instead of NFS CSI
  • Next, run a one-time Bash script that iterates through all EKS apps, scaling them down or just deleting their pods, followed by recreating their PVCs, which Kyverno would convert to Trident PVCs on the fly

Ideally, if you had the time, you would rewrite all PVC manifests in all microservice Helm charts to be Trident-compliant, followed by Helm upgrade of all Helm releases in EKS, with the right Helm values to replace NFS CSI PVCs with Trident PVCs.

Instead, we chose to build the Kyverno layer that ensures that all existing & new PVCs, even those coming in from future Helm upgrades, auto-convert to Trident seamlessly on create/update.

We were also migrating from static PVs in NFS CSI to dynamic PVs in Trident. So the Kyverno policy needed to take care of that as well. To understand how it works, first look at this PV-PVC pair using NFS CSI:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs

spec:
  resources:
    requests:
      storage: 1Gi

  volumeName: nfs
  storageClassName: nfs
  volumeMode: Filesystem
  accessModes: [ ReadWriteMany ]
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs

spec:
  storageClassName: nfs
  volumeMode: Filesystem
  accessModes: [ ReadWriteMany ]

  capacity:
    storage: 1Gi
  claimRef:
    name: nfs
    kind: PersistentVolumeClaim

  csi:
    driver: nfs.csi.k8s.io
    volumeHandle: fs-<random-ID>
    volumeAttributes:
      share: /nfs # FSx volume name
      server: svm-<random-ID>.fs-<random-ID>.fsx.<AWS-region>.amazonaws.com

The equivalent Trident PVC for this would be:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs

  annotations:
    trident.netapp.io/importBackendUUID: <trident-backend-ID>
    trident.netapp.io/importOriginalName: nfs # FSx volume name
    volume.kubernetes.io/storage-provisioner: csi.trident.netapp.io

spec:
  resources:
    requests:
      storage: 1Gi

  storageClassName: fsx
  volumeMode: Filesystem
  accessModes: [ ReadWriteMany ]

The annotations are key here. Normally with dynamic provisioning, Trident would create both a new FSx volume & a new PV for a PVC. The annotations above ensure that Trident creates a dynamic PV but not a new FSx volume. Instead, Trident “imports” the existing FSx volume & starts managing it.

The Kyverno policy has 2 rules: 1 for PVC create & 1 for PVC update. In case of create, it reads the PVC’s PV, extracts the FSx volume name & sets it in the Trident PVC’s import annotation:

kind: ClusterPolicy
apiVersion: kyverno.io/v1
metadata:
  name: convert-nfs-pvc-to-trident-pvc
spec:
  webhookConfiguration:
    # Don't block create / update of PVC if this policy fails
    failurePolicy: Ignore

  rules:
  - name: convert-nfs-pvc-to-trident-pvc-on-create
    match:
      any:
      - resources:
          operations: [ CREATE ]
          kinds: [ PersistentVolumeClaim ]

    context:
    - name: csiDriver
      apiCall:
        urlPath: /api/v1/persistentvolumes/{{request.object.spec.volumeName || ''}}
        jmesPath: spec.csi.driver
    - name: fsxVolume
      apiCall:
        urlPath: /api/v1/persistentvolumes/{{request.object.spec.volumeName || ''}}
        jmesPath: spec.csi.volumeAttributes.share
    - name: tridentBackendUUID
      apiCall:
        urlPath: /apis/trident.netapp.io/v1/namespaces/trident/tridentbackendconfigs/fsx
        jmesPath: status.backendInfo.backendUUID

    preconditions:
      all:
      - key: '{{csiDriver}}'
        operator: Equals
        value: nfs.csi.k8s.io

    mutate:
      patchStrategicMerge:
        metadata:
          annotations:
            +(trident.netapp.io/importBackendUUID): '{{tridentBackendUUID}}'
            +(trident.netapp.io/importOriginalName): "{{trim_prefix('{{fsxVolume}}','/')}}"
        spec:
          volumeName: ''
          storageClassName: fsx
          accessModes: [ ReadWriteMany ]

  - name: convert-nfs-pvc-to-trident-pvc-on-update
    match:
      any:
      - resources:
          operations: [ UPDATE ]
          kinds: [ PersistentVolumeClaim ]

    preconditions:
      all:
      - key: '{{request.oldObject.spec.storageClassName}}'
        operator: Equals
        value: fsx
      - key: '{{request.object.spec.storageClassName}}'
        operator: NotEquals
        value: fsx

    mutate:
      patchStrategicMerge:
        spec:
          storageClassName: fsx
          accessModes: [ ReadWriteMany ]
          volumeName: '{{request.oldObject.spec.volumeName}}'

All subsequent NFS CSI PVC creates & updates, either via Helm upgrades or manually, will be seamlessly converted to Trident PVCs by this policy!

With this Kyverno policy deployed in EKS, run a script to re-create all NFS CSI PVCs as Trident PVCs:

NS=my-namespace
for PVC in $(kubectl get pvc -n "$NS" --no-headers -o custom-columns=:.metadata.name 2>/dev/null); do

  PV=$(kubectl get pvc -n "$NS" "$PVC" -o jsonpath='{.spec.volumeName}' 2>/dev/null)
  [ -z "$PV" ] && continue

  CSI=$(kubectl get pv "$PV" -o jsonpath='{.spec.csi.driver}' 2>/dev/null || true)
  if [ "$CSI" == 'nfs.csi.k8s.io' ]; then

    PVC_YAML=$(kubectl get pvc -n "$NS" "$PVC" -o yaml)
    echo Deleting NFS PVC: "$PVC"
    kubectl delete --wait=false -n "$NS" pvc/"$PVC"

    PODS=$(kubectl get pods -n "$NS" -o json | jq -r --arg pvc "$PVC" \
    '.items[] | select(.spec.volumes[]? | select(.persistentVolumeClaim.claimName == $pvc)) | .metadata.name')

    if [ -z "$PODS" ]; then
      echo No pods are mounting this PVC
    else
      echo Deleting pods mounting this PVC:
    fi

    for POD in $PODS; do
      kubectl delete --wait=false -n "$NS" pod/"$POD" || true
    done

    echo Waiting for PVC to delete...
    while kubectl get pvc -n "$NS" "$PVC" >/dev/null 2>&1; do true; done

    echo Creating Trident PVC: "$PVC"
    PVC_YAML=$(echo "$PVC_YAML" | yq eval 'del(.metadata.annotations."pv.kubernetes.io/bind-completed")' -)
    PVC_YAML=$(echo "$PVC_YAML" | yq eval 'del(.metadata.annotations."pv.kubernetes.io/bound-by-controller")' -)
    PVC_YAML=$(echo "$PVC_YAML" | yq eval 'del(.metadata.annotations."volume.kubernetes.io/storage-provisioner")' -)
    PVC_YAML=$(echo "$PVC_YAML" | yq eval 'del(.metadata.annotations."volume.beta.kubernetes.io/storage-provisioner")' -)
    echo "$PVC_YAML" | kubectl create -f -
  fi
done

All new Trident PVCs should now be bound. Look for any pending Trident PVCs & fix as needed:

kubectl get pvc -Ao go-template='{{range .items}}{{if and (eq .status.phase "Pending") (eq .spec.storageClassName "fsx")}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | sort | uniq

This concludes the CSI migration. All NFS CSI PVCs have now converted to Trident PVCs. The static NFS CSI PVs are now “released” & can be deleted.

Leave a Reply

Your email address will not be published. Required fields are marked *