Table of Contents

• Attribute-Based Instance Type Selection
• Karpenter Provisioners
• ABS in Karpenter
• CPU & Memory-Based Instance Selection
• Caveat: Setting Inclusive Limits
• Test ABS in Karpenter
• Conclusion
• Resources
• About the Author ✍🏻

This post assumes you’re familiar with Kubernetes, Amazon EKS, & Karpenter.

Attribute-Based Instance Type Selection

Attribute-based instance type selection (ABS) lets you express your instance requirements as a set of attributes, such as vCPU, memory, processor type, & storage. Your requirements are translated by ABS to all matching instance types, simplifying the creation & maintenance of instance type configurations. This also allows you to automatically use newer generation instance types when they are released & access a broader range of capacity via EC2 spot instances. In this post, you will learn how to configure ABS in Karpenter so it can select & launch instances that fit the specified attributes, removing the need to manually pick instance types.

Karpenter Provisioners

A Karpenter provisioner sets constraints on the nodes that can be created by Karpenter & the pods that can run on them. Here is a sample Karpenter provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
  consolidation:
    enabled: true
  requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["c", "r"]
  - key: kubernetes.io/arch
    operator: In
    values: ["arm64"]
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot"]

The instance requirements here are limiting Karpenter to ARM spot nodes in C & R instance families.

ABS in Karpenter

Kubernetes defines a number of well-known labels, which cloud providers like AWS implement, so we can use them in places like Karpenter provisioner’s requirements section. In addition to these well-known labels, Karpenter supports several AWS-specific labels for more advanced scheduling. In this post, I’ll demonstrate ABS using 2 of these labels: CPU & memory. The same approach can be applied to other labels as well, to achieve ABS using them.

CPU & Memory-Based Instance Selection

Consider a generic workload like a web server, that scales horizontally proportional to the volume of incoming traffic. When it creates more pods, Karpenter provisions nodes to accommodate the scale up. Instead of configuring Karpenter to find specific types of nodes like r5.large or c5.xlarge, you can provide a range of acceptable vCPU & memory requirements, within which, Karpenter can look for any available node type & use it to optimize for node utilization or cost.

The exact labels to use for this scenario, are karpenter.k8s.aws/instance-cpu & karpenter.k8s.aws/instance-memory. And since we wish to provide a range of acceptable values, not an exact value, we’ll use the greater than (Gt) & less than (Lt) operators. Here is an example:

- key: karpenter.k8s.aws/instance-cpu
  operator: Gt
  values: ['4']
- key: karpenter.k8s.aws/instance-cpu
  operator: Lt
  values: ['16']
- key: karpenter.k8s.aws/instance-memory
  operator: Gt
  values: ['8192'] # 8G
- key: karpenter.k8s.aws/instance-memory
  operator: Lt
  values: ['32768'] # 32G

Caveat: Setting Inclusive Limits

The example above, looks for nodes with vCPU >4 <16 & memory >8 <32. Although the Gt & Lt operators, when used in a pod’s nodeAffinity act as “greater than or equal to” & “less than or equal to” (reference), Karpenter treats them strictly as greater than & less than. So to include nodes with 4 or 16 vCPUs or 8G or 32G memory, you must adjust your values accordingly. For example:

- key: karpenter.k8s.aws/instance-cpu
  operator: Gt
  values: ['3']
- key: karpenter.k8s.aws/instance-cpu
  operator: Lt
  values: ['17']
- key: karpenter.k8s.aws/instance-memory
  operator: Gt
  values: ['7168'] # 7G
- key: karpenter.k8s.aws/instance-memory
  operator: Lt
  values: ['33792'] # 33G

Test ABS in Karpenter

After creating a provisioner with the above requirements, try deploying a sample workload to see the node type Karpenter provisions:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      containers:
      - name: inflate
        image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
        resources:
          requests:
            cpu: 1

This would most likely bring up a 4 vCPU 8G node. Try changing the CPU request (last line in above deployment) to 5 & Karpenter should launch something like an 8 vCPU 16G node.

Conclusion

In this article, you learnt how to configure Karpenter to intelligently select EC2 instance types based on your resource requirements without having to manually provide a list of instance types to Karpenter. For even more attributes you can use to influence Karpenter attribute-based node selection, like OS, CPU architecture, network bandwidth, & GPU, see the complete list of attributes here.

Resources

To learn more about ABS, check out these resources:

• Attribute-Based Instance Type Selection for EC2 Fleet — EC2 User Guide
• Implementing Attribute-Based Instance Type Selection using Terraform — AWS Compute Blog
• Attribute-Based Instance Type Selection for EC2 Auto Scaling & EC2 Fleet — AWS News Blog

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

AWS DevOps EKS Karpenter