Migrating OpenShift Templates to Helm Charts for Amazon EKS: Design Guidelines & Lessons Learned

Table of Contents

Introduction

If you’ve been following our blog for a while, you know that we recently completed a large-scale migration of hundreds of containerized microservices from OpenShift to Amazon EKS.

Here are the relevant articles, if you’d like a refresher:

The bulk of this migration was writing Helm charts that would be deployable on both OpenShift & EKS. In the process, we discovered the best way to do many things in Helm charts, some of which are standard practice in the industry, others that are unique to our use cases. This article is a summary of what to do & what not to do in Helm charts.

Avoid Helm Sub-Charts

Sub-charts in Helm are a great concept & very useful in many scenarios (like umbrella charts). If however, you find yourself in a case like ours where, a number of Helm charts come together incrementally, to deploy an entire product suite, it’s better to stay away from sub-charts.

You see, the way we instruct our end users to deploy our product suite is by creating a single, global Helm values YAML file containing overrides for values of an entire product & then use the same YAML file to install the charts that make up the product, one by one.

In such a case, the following attributes of Helm sub-charts were more of an inconvenience for us, than a desired feature:

  • Sub-charts are standalone charts in their own right & as such, come with all the overhead of a chart, like its own, separate versioning scheme.
  • Overridden values from the helm install command line are not passed to sub-charts, unless they’re either under a global section or a section named after the sub-chart.
  • If you try to make it work with sub-charts & global sections, you inevitably end up writing overly complicated Helm templates & utilities to work around the complexity.

Due to all these reasons & some other minor ones, it was much easier & cleaner for us to group similar manifests in a sub-directory under the templates directory of the Helm chart, instead of creating a dedicated sub-chart for it.

Use Subdirectories Under templates

A typical Helm chart has a directory structure like this:

my-chart
├── Chart.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

When you have Helm charts made up of distinct groups of manifests, like multiple, related microservices, create subdirectories under templates to keep things clean:

my-chart
├── Chart.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── microservice-1
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   ├── microservice-2
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── serviceaccount.yaml
└── values.yaml

Keep common objects outside the subdirectories, like the service account & helpers above.

This directory structure doesn’t have any side-effects: you don’t have to change the hierarchy of values in values.yaml or anything else like that. It just works!

Keep All Parameters in One values.yaml

There is only one values.yaml (at the top-level of the chart). This contains all values needed by all microservices in the chart. Within values.yaml, you can create hierarchies to segregate microservice-specific values and keep common parameters at the top level.

Document All Parameters in values.yaml

For every parameter in values.yaml, describe the purpose of the parameter in a comment just before that parameter. A consumer of the Helm chart will see them in the helm show values command. This makes it much easier for an end user to consume a chart, without having to find & dig through out-of-band parameter documentation.

Rename / Restructure Conflicting Parameters

Since all values from all microservices share a file (values.yaml), conflicting param names must either be renamed or restructured in their values hierarchy to coexist, & their references updated in the Helm charts.

Call-out Mandatory Parameters

Values that don’t have defaults & must be provided by the chart installer, must be marked as mandatory using Helm’s required function:

{{ .Values.mandatory_value | required "mandatory_value is required" }}

Design Self-Sufficient Helm Charts

As far as possible, all Helm charts should be installable without providing --values or --set at the helm install command line. All parameters required by the chart should have sensible defaults built into the chart’s own internal values.yaml.

Always test your chart by helm installing it without --values or --set. The only exception to this rule are parameters that are mandatory (must be provided at the helm install command line) & cannot have sensible defaults, like external database endpoints.

Strive for Minimal External Dependencies

As much as possible, avoid external dependencies, even on library charts. Although the use of a library is a best practice for reusable code, it can sometimes be avoided by adopting a better design. Strive to keep external dependencies to a minimum for better maintainability.

Namespace is NOT a Parameter

None of the templates in the chart should have the metadata.namespace field defined, unless they’re creating objects that are meant to be deployed into a specific namespace, like kube-system.

Moreover, namespace should not be a parameter expected from --values or --set at the helm install command line. Namespaces should only be specified by the chart installer using the --namespace flag to the helm install command.

If you need the namespace in your Helm templates, use {{ .Release.Namespace }} as described at Helm | Built-in Objects.

Standardize File Extensions

This is just to keep things clean. All YAML files end with the .yaml extension, not .yml.

Standardize File Names

Design a naming convention & have everyone in your team follow it. For example:

These are the only file names allowed: service.yaml, deployment.yaml, secret.yaml, ingress.yaml, config-map.yaml, service-account.yaml & so on.

Note how they’re all lowercase, use hyphens between words & never use camel case.

Always Parameterize the Replica Count

Never hardcode the replica count of a Deployment / StatefulSet. Always use a parameter for it with a default value in values.yaml, so it can be overridden on-site, if required.

Use stringData Instead of data in Secrets

Unless you’re using an external secret store for Kubernetes secrets, there’s not much value in base64 encoding secret data in your manifests.

It’s much easier to use stringData instead of data as much as possible, to avoid the overhead of manually base64 encoding/decoding secrets when working with secret manifests.

This also avoids confusion for the end user about whether to provide plaintext secrets or base64 encode them before adding them to --values or --set.

Avoid More Than One YAML Doc in One File

As much as possible, avoid putting more than one YAML document (separated by ---) in one YAML file. However, there are some valid use cases to do this, like creating a collection of very similar PVCs or config maps.

Parameterize All Container Resource Requests & Limits

The chart’s internal values.yaml should declare default values for resource requests & limits of all containers in that chart. End users should be able to override them.

Ensure StorageClass is Specified

Always provide the storageClassName in objects like PVCs & StatefulSets. Without this, they would use the cluster’s default StorageClass, which isn’t ideal.

Parameterize StorageClass Names

Never hardcode StorageClass names in PVCs or StatefulSets or anywhere else. Always ensure they’re parameterized (like .Values.storage_classes.gold.name), so they can be overridden by helm install.

Ensure All PVCs Have volumeNames

If you provision your PVs manually, ensure all PVCs have spec.volumeName! Without this, a PVC can bind with a PV created for another workload/namespace!

Conditional spec.VolumeName in PVCs

If like us, you have a need to write Helm charts that need to be deployable to environments with both static & dynamic PVs, here’s a tip:

The presence of spec.VolumeName in PVCs should be conditionally based on whether the static PV required by the PVC already exists in the cluster.

As you gradually move from static to dynamic PV provisioning mechanisms, PVCs that had their PVs statically provisioned thus far, would be expected to dynamically provision their own PVs henceforth.

As such, PVCs that need to specify a static PV name, must do so like this:

spec:
  {{ if lookup "v1" "PersistentVolume" "" "my-static-pv" }}
  volumeName: my-static-pv
  {{ end }}

Learn more about the lookup function at Helm | Template Functions and Pipelines.

Retain PVCs After Helm Uninstall

Once you migrate from static to dynamic PVs, the only way to control the lifecycle of a PV is via its PVC. If you delete the PVC, its PV is also deleted & you lose its data, assuming the StorageClass is configured to do so. So when you uninstall a Helm chart that created PVCs, it will delete its PVCs, which in turn will delete its PV & data.

To prevent this, add the helm.sh/resource-policy: keep annotation to all PVCs (both standalone & in StatefulSets). When you uninstall a chart that contains objects annotated like this, those objects will be retained upon chart uninstall:

$ helm uninstall my-chart

These resources were kept due to the resource policy:
[PersistentVolumeClaim] my-pvc

release "my-chart" uninstalled

If you reinstall the chart with the same name again, it will seamlessly retake control of this PVC. If however, you don’t need the PVC & its data, manually delete the PVC after uninstalling the chart to perform a complete cleanup.

For more info, see Helm | Chart Development Tips and Tricks.

Ensure High-Availability

All (critical) microservices should have at least 3 replicas & replicas should be evenly distributed across availability zones.

Conclusion

This was a short list of some best practices we picked up migrating our OpenShift templates to Helm charts, targeted for Amazon EKS clusters. I’m sure there are many more that can be added to this list. 😊

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Leave a Reply

Your email address will not be published. Required fields are marked *