Cutting EKS spend 60%+ with Karpenter and dynamic Spot

Most EKS bills are big for two boring reasons: nodes are on-demand, and they're half empty. Cluster Autoscaler grows pre-defined node groups but is bad at picking the right instance and worse at giving capacity back. The result is a fleet of oversized on-demand boxes running at 40% utilization.

Karpenter attacks both problems: it provisions right-sized nodes per pending pod directly against EC2, prefers Spot, and continuously consolidates — replacing or removing nodes as workloads shrink. On a typical fleet the combination of Spot pricing (~70% off) and consolidation (reclaiming the idle half) lands in the 60–75% compute reduction range. Below is how we roll it out without waking anyone up at 3 a.m.

The core switch: NodePool + EC2NodeClass

Karpenter (v1 API) replaces node groups with two objects: a NodePool (what to schedule and how to disrupt) and an EC2NodeClass (how to build the node). The single most important field for savings is the diversified requirement set — give Karpenter a wide menu of instance types and both capacity types, and let it choose the cheapest that fits.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]      # Spot first; on-demand as fallback
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]          # allow Graviton (see below)
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]                        # modern generations only
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h                         # recycle nodes every 30 days
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m                        # give back idle capacity fast
  limits:
    cpu: "2000"                                 # a safety ceiling for the pool

The wider that instance menu, the better Spot behaves — Karpenter's Spot allocation uses the price-capacity-optimized strategy, so dozens of eligible types across AZs means it can always find a deep, cheap pool and reprovision instantly when one gets reclaimed. Narrow menus are the number-one cause of Spot pain.

The EC2NodeClass defines the node itself — AMI, IAM role, subnets, and, importantly for cost visibility, the tags stamped onto every instance Karpenter launches:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: "KarpenterNodeRole-prod"
  subnetSelectorTerms:
    - tags: { "karpenter.sh/discovery": "prod" }
  securityGroupSelectorTerms:
    - tags: { "karpenter.sh/discovery": "prod" }
  tags:                                   # cost-allocation tags on every node
    team: platform
    cost-center: "cc-4417"
    environment: prod
    provisioner: karpenter

The rule that makes bin-packing work: CPU requests, memory limits

Karpenter bin-packs nodes based on pod requests. If requests are wrong, everything downstream is wrong — you either waste money (over-request) or get OOM churn (under-request). The discipline we enforce on every workload:

• Set CPU requests, but no CPU limits.
• Set memory requests equal to memory limits.

resources:
  requests:
    cpu: "500m"        # used for scheduling + Karpenter bin-packing
    memory: "1Gi"
  limits:
    memory: "1Gi"      # == request. NOTE: no cpu limit on purpose.

Why this asymmetry? Because the two resources fail differently:

• CPU is compressible. A CPU limit is enforced by the CFS quota, which will throttle a container even when the node has idle cores sitting right there — wrecking tail latency for no reason. Requests already guarantee a fair share under contention, so a limit only adds artificial throttling. Drop it.
• Memory is incompressible. You can't "throttle" memory; the only way to reclaim it is to kill something. Without a limit, a leaky pod grows until the node OOMs and takes innocent neighbors with it. A memory limit equal to the request makes the offender the one that gets OOMKilled, and keeps the node's schedulable memory honest so Karpenter packs it correctly.

Rightsizing requests is the same lever as the node bill. Tighten requests to real usage (VPA in recommend-mode or Prometheus p95 is your friend) and Karpenter consolidation converts the slack directly into fewer nodes.

Spot without the pandemonium

Karpenter handles Spot interruptions natively when you give it an interruption queue (an SQS queue fed by EC2 Spot interruption, rebalance-recommendation, and health events). On the 2-minute warning it cordons and drains the doomed node and has replacement capacity coming up before the pods are evicted:

# Helm values — wire Karpenter to the interruption SQS queue
settings:
  clusterName: prod
  interruptionQueue: prod-karpenter-interruption

With a diversified NodePool this is a non-event for stateless services. The pods that can't tolerate a 2-minute eviction are the real work — that's the next two sections.

Stateful workloads: keep them off the churn

Spot + aggressive consolidation is fantastic for stateless deployments and hostile to databases, brokers, and anything backed by an EBS volume. The volume is pinned to one AZ, reattachment takes time, and a mid-write eviction is exactly what you don't want. Our approach is to isolate stateful workloads onto a stable, on-demand NodePool and tell Karpenter to leave them alone.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: stateful
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]              # no Spot for stateful
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: workload
          value: stateful
          effect: NoSchedule                  # only tolerating pods land here
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmpty            # only reclaim truly empty nodes
    consolidateAfter: 30m                     # slow, gentle

Then the stateful pods opt in with a matching toleration and node selector, and — the key bit — the do-not-disrupt annotation so Karpenter never voluntarily evicts them for consolidation or drift:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"   # exclude from voluntary disruption
    spec:
      nodeSelector:
        karpenter.sh/capacity-type: on-demand
      tolerations:
        - key: workload
          operator: Equal
          value: stateful
          effect: NoSchedule
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: postgres }

For stateful services that can tolerate controlled failover (a quorum of 3+ replicas, e.g. Kafka, etcd, a replicated database), you can run them on Spot to save more — but only behind a strict PodDisruptionBudget and zonal spread so you never lose quorum to a single interruption wave.

PodDisruptionBudgets: the safety interlock

Karpenter honors PDBs during all voluntary disruption (consolidation, drift, expiration). A PDB is how you cap how much of a service Karpenter may take down at once. Every meaningful workload should have one — without it, a consolidation pass can drain replicas faster than they reschedule.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 80%          # Karpenter may disrupt at most 20% at a time
  selector:
    matchLabels: { app: web }

One caveat worth knowing: a PDB only governs voluntary disruption. An actual Spot reclaim is involuntary and does not respect the PDB — which is exactly why quorum-sensitive systems still need replica counts and zonal spread that survive losing a node outright, not just a well-set PDB.

Scheduling: taints, tolerations, and dedicated pools

Beyond stateful isolation, taints/tolerations plus per-pool requirements let you carve the fleet into intent-specific lanes without static node groups:

• Batch / preemptible — a Spot-only pool tainted workload=batch; jobs tolerate it and get the cheapest capacity, and you don't care if they're reclaimed.
• GPU — a pool with a GPU instance category and a nvidia.com/gpu taint so only GPU pods (which tolerate it) ever pull those expensive nodes.
• System / add-ons — CoreDNS, ingress controllers, and the like on a small on-demand pool, tainted so app workloads can't crowd them out.
• startupTaints — set for things like a CNI or security agent that must be ready before normal pods schedule; the agent removes the taint when it's up.

Pods select their lane with nodeSelector/nodeAffinity on Karpenter's well-known labels — karpenter.sh/capacity-type, kubernetes.io/arch, node.kubernetes.io/instance-type, karpenter.k8s.aws/instance-family, and topology.kubernetes.io/zone — the same keys you use in NodePool requirements.

Two chip architectures, one fleet: Graviton

Graviton (arm64) instances are typically ~20% cheaper than the x86 equivalent and often faster per dollar — stacking on top of the Spot and consolidation savings. Because the default NodePool above already allows kubernetes.io/arch: [amd64, arm64], Karpenter will pick Graviton whenever it's the cheapest fit. The only prerequisite is that your images are multi-arch.

# Build and push a multi-arch manifest (one tag, both architectures)
docker buildx create --use
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t 1234567890.dkr.ecr.us-east-1.amazonaws.com/web:1.8.0 \
  --push .

# Verify the manifest actually carries both arches
$ docker buildx imagetools inspect .../web:1.8.0 | grep -A1 Platform
  Platform:  linux/amd64
  Platform:  linux/arm64

With multi-arch images, no pod changes are needed — the kubelet pulls the matching arch automatically. For workloads that are not arm-ready yet (a vendored x86 binary, an incompatible dependency), pin just those to x86 and let everything else float:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/arch
              operator: In
              values: ["amd64"]     # this one workload stays x86

A pragmatic rollout: allow both arches fleet-wide, watch for exec format error crashes (that's a single-arch image on an arm node), multi-arch or pin those few, and let the rest migrate to Graviton on its own.

What it looks like on the bill

Representative before/after for a ~180-node stateless-heavy fleet we migrated. Each row stacks on the one above it:

You can watch the mechanics live. Karpenter surfaces exactly what it launched and why:

# Node inventory by capacity type + arch after migration
$ kubectl get nodes -L karpenter.sh/capacity-type,kubernetes.io/arch \
      -L node.kubernetes.io/instance-type
  NAME              STATUS   CAPACITY-TYPE   ARCH    INSTANCE-TYPE
  ip-10-0-12-4      Ready    spot            arm64   c7g.2xlarge
  ip-10-0-33-9      Ready    spot            amd64   m6a.xlarge
  ip-10-0-41-2      Ready    spot            arm64   r7g.xlarge
  ip-10-0-8-51      Ready    on-demand       amd64   m6i.large     # stateful pool

# Karpenter's own view of provisioning + consolidation decisions
$ kubectl get nodeclaims -o wide
$ kubectl logs -n kube-system deploy/karpenter | grep -Ei 'launched|consolidat|disrupt'
  ... "launched nodeclaim" ... capacity-type=spot instance-type=c7g.2xlarge
  ... "disrupting via consolidation delete" ... nodes=3
  ... "disrupting via consolidation replace" ... reason=underutilized

Takeaways

• Diversify hard. Wide instance menus across families, sizes, and AZs are what make Spot boring and consolidation effective.
• CPU requests, no CPU limits; memory requests == limits. This single policy fixes both throttling and bin-packing accuracy.
• Isolate state. On-demand pool, taints, do-not-disrupt, and PDBs keep databases off the churn while everything else rides Spot.
• PDBs are mandatory, not optional — but remember they don't stop involuntary Spot reclaims, so design replica counts to survive real node loss.
• Turn on Graviton. Multi-arch images plus an arch-flexible NodePool is ~20% on top, nearly for free.

If your EKS bill looks like a fleet of half-empty on-demand nodes, this is a two-to-four week engagement that usually pays for itself in the first month. Get in touch.