AWS EKS in Production — Node Groups, Karpenter, and the Operational Gotchas

Introduction

AWS EKS abstracts Kubernetes control plane management, but node orchestration remains your responsibility. Managed node groups simplify node provisioning, but lack sophistication for cost optimization. Karpenter, AWS's intelligent autoscaling solution, replaces Cluster Autoscaler with performance and cost improvements. Add to this EKS add-ons for networking and DNS, IAM Roles for Service Accounts (IRSA) for pod-level permissions, cluster upgrade complexity, and the operational surface area grows. This post covers node group strategies, Karpenter, add-on management, IRSA, upgrades, and cost optimization.

Managed Node Groups vs Self-Managed
Karpenter for Intelligent Node Provisioning
EKS Add-Ons Management
IAM Roles for Service Accounts (IRSA)
EKS Cluster Upgrade Strategy
Cluster Autoscaler vs Karpenter
Cost Optimization with Spot Instances
Checklist
Conclusion

Managed Node Groups vs Self-Managed

Managed node groups: EKS handles EC2 instance provisioning and lifecycle. Simpler, recommended for most teams.

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main-nodes"
  node_role_arn   = aws_iam_role.node_role.arn
  subnet_ids      = var.private_subnet_ids

  scaling_config {
    desired_size = 3
    max_size     = 20
    min_size     = 2
  }

  instance_types = ["t3.large", "t3a.large", "m5.large"]
  disk_size      = 100

  labels = {
    workload = "general"
  }

  tags = {
    Name        = "eks-main"
    Environment = "prod"
  }

  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

The ignore_changes on desired_size prevents Terraform from fighting with autoscaler.

Self-managed nodes: You create Launch Templates and ASGs. More control, more operational burden.

resource "aws_launch_template" "node" {
  name_prefix = "eks-node-"

  image_id      = data.aws_ami.amazon_linux_2_ecs.id
  instance_type = "t3.large"

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size           = 100
      volume_type           = "gp3"
      delete_on_termination = true
      encrypted             = true
    }
  }

  iam_instance_profile {
    name = aws_iam_instance_profile.node.name
  }

  vpc_security_group_ids = [aws_security_group.node.id]

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "eks-node"
    }
  }

  user_data = base64encode(templatefile("${path.module}/user_data.sh", {
    cluster_name        = aws_eks_cluster.main.name
    cluster_endpoint    = aws_eks_cluster.main.endpoint
    cluster_ca          = aws_eks_cluster.main.certificate_authority[0].data
  }))
}

resource "aws_autoscaling_group" "node" {
  name                = "eks-node-asg"
  vpc_zone_identifier = var.private_subnet_ids
  min_size            = 2
  max_size            = 20
  desired_capacity    = 3

  launch_template {
    id      = aws_launch_template.node.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "eks-node"
    propagate_at_launch = true
  }

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

Karpenter for Intelligent Node Provisioning

Karpenter replaces Cluster Autoscaler with a more efficient provisioner. It groups pods by scheduling requirements, right-sizes instances, and terminates underutilized nodes faster.

Install Karpenter:

helm repo add karpenter https://charts.karpenter.sh
helm install karpenter karpenter/karpenter \
  --namespace karpenter --create-namespace \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT_ID:role/karpenter-controller \
  --set settings.aws.clusterName=my-cluster \
  --set settings.aws.interruptionQueue=my-cluster-queue

Karpenter Provisioner:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        pool: default
    spec:
      nodeClassRef:
        name: default
      expireAfter: 2592000s  # 30 days
  limits:
    resources:
      cpu: "1000"
      memory: "1000Gi"
  consolidationPolicy:
    nodes: "WhenUnderutilized"
  disruption:
    consolidateAfter: 30s
    consolidateOnDeletion: true
    budgets:
    - nodes: "10%"
      duration: 5m
      schedule: "0 9 * * mon-fri"
      timezone: "America/New_York"
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: KarpenterNodeRole
  subnetSelector:
    karpenter.sh/discovery: "true"
  securityGroupSelector:
    karpenter.sh/discovery: "true"
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      volumeSize: 100Gi
      volumeType: gp3
      iops: 3000
      throughput: 125
  tags:
    ManagedBy: Karpenter
    Environment: production

Karpenter vs Cluster Autoscaler:

Feature	Karpenter	Cluster Autoscaler
Instance right-sizing	Yes (groups pods by resource reqs)	No (fixed instance types)
Consolidation	Yes (packs pods tighter)	No
Spot instances	Native	Via ASG tags
Scheduling awareness	Yes	No
Cold start latency	Minimal	Higher
Cost	Lower (better bin-packing)	Higher (fragmentation)

EKS Add-Ons Management

EKS add-ons are managed services for essential cluster components.

resource "aws_eks_addon" "vpc_cni" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "vpc-cni"
  addon_version            = "v1.14.1-eksbuild.1"
  resolve_conflicts_on_update = "OVERWRITE"

  service_account_role_arn = aws_iam_role.vpc_cni_role.arn
}

resource "aws_eks_addon" "coredns" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "coredns"
  addon_version            = "v1.9.3-eksbuild.2"
  resolve_conflicts_on_update = "OVERWRITE"
}

resource "aws_eks_addon" "kube_proxy" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "kube-proxy"
  addon_version            = "v1.27.9-eksbuild.2"
  resolve_conflicts_on_update = "OVERWRITE"
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "aws-ebs-csi-driver"
  addon_version            = "v1.24.0-eksbuild.1"
  service_account_role_arn = aws_iam_role.ebs_csi_role.arn
}

Always pin add-on versions. Unspecified versions auto-update, which can break workloads.

IAM Roles for Service Accounts (IRSA)

IRSA grants pods IAM permissions without requiring EC2 instance role. Fine-grained, secure.

# Create trust relationship between ServiceAccount and IAM role
data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    effect = "Allow"
    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${aws_iam_openid_connect_provider.eks.url}"]
    }
    action = "sts:AssumeRoleWithWebIdentity"
    condition {
      test     = "StringEquals"
      variable = "${aws_iam_openid_connect_provider.eks.url}:sub"
      values   = ["system:serviceaccount:production:s3-reader"]
    }
  }
}

resource "aws_iam_role" "s3_reader" {
  name               = "eks-s3-reader"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}

resource "aws_iam_role_policy" "s3_read" {
  name   = "s3-read"
  role   = aws_iam_role.s3_reader.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ]
        Resource = [
          "arn:aws:s3:::data-bucket",
          "arn:aws:s3:::data-bucket/*"
        ]
      }
    ]
  })
}

Kubernetes ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/eks-s3-reader
---
apiVersion: v1
kind: Pod
metadata:
  name: s3-reader-pod
  namespace: production
spec:
  serviceAccountName: s3-reader
  containers:
  - name: reader
    image: amazon/aws-cli:latest
    command:
    - /bin/sh
    - -c
    - aws s3 ls s3://data-bucket

EKS Cluster Upgrade Strategy

Upgrading EKS involves updating control plane, then nodes. Plan for downtime.

# Update Kubernetes version
resource "aws_eks_cluster" "main" {
  name    = "my-cluster"
  version = "1.28"  # Update this field

  vpc_config {
    subnet_ids = var.subnet_ids
  }
}

# After control plane updates, update node groups
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main"

  # Update to match cluster version
  version = "1.28"
}

Upgrade process:

Plan: Review changelogs, test in staging
Control Plane: AWS updates control plane (5-10 min downtime per AZ)
Nodes: Update node groups progressively (max_unavailable = 1)
Rollback: Keep previous AMI available for quick rollback

Progressive node update:

resource "aws_eks_node_group" "main" {
  update_config {
    max_unavailable_percentage = 20  # Update 20% of nodes at a time
  }
}

Cluster Autoscaler vs Karpenter

Cluster Autoscaler: Kubernetes' built-in autoscaler. Simpler, but less efficient.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.27.0
        name: cluster-autoscaler
        command:
        - ./cluster-autoscaler
        - --cloud-provider=aws
        - --nodes=2:20:my-asg-name

Cluster Autoscaler scales one node at a time. Karpenter groups pods and provisions right-sized instances in bulk.

Cost comparison (real example):

Cluster Autoscaler: 15 t3.large nodes (underutilized due to fragmentation)
Karpenter: 8 t3.xlarge + 2 m5.large nodes (better bin-packing)
Monthly savings: $300-400 (20% reduction)

Cost Optimization with Spot Instances

Spot instances are 70-90% cheaper than On-Demand but can be interrupted.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: spot-pool
spec:
  template:
    spec:
      nodeClassRef:
        name: spot
      taints:
      - key: karpenter.sh/capacity-type
        value: spot
        effect: NoSchedule
  limits:
    resources:
      cpu: "500"
  disruption:
    budgets:
    - nodes: "5%"
      duration: 5m
  providerRef:
    name: spot
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: spot
spec:
  amiFamily: AL2
  subnetSelector:
    karpenter.sh/discovery: "true"
  securityGroupSelector:
    karpenter.sh/discovery: "true"
  capacityType: "spot"
  tags:
    CapacityType: Spot

Pod tolerations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-processor
spec:
  template:
    spec:
      tolerations:
      - key: karpenter.sh/capacity-type
        operator: Equal
        value: spot
        effect: NoSchedule
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - batch-processor
              topologyKey: kubernetes.io/hostname

Batch and non-critical workloads tolerate Spot. Critical workloads use On-Demand.

Checklist

Conclusion

EKS manages Kubernetes control plane, but node orchestration complexity remains. Managed node groups provide a good starting point. Graduate to Karpenter for intelligent bin-packing and Spot cost optimization. Use IRSA to grant fine-grained IAM permissions. Pin add-on versions to prevent surprise updates. Plan cluster upgrades as major operational events. Combine On-Demand and Spot instances for cost efficiency without sacrificing reliability. With these practices, EKS clusters become cost-effective, reliable, and operationally manageable at scale.