AWS EKS Upgrade 1.18 to 1.21

Introduction 

Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes service offering from AWS. We rely on EKS to run our APIs because of the security, reliability, and scalability that EKS offers. It provides high availability and scalability for the Control Plane to run across multiple availability zones (thus eliminating a single point of failure situation). 

Since EKS is a fully managed service, EKS Updates deliver new features, design updates, and bug fixes. So, it is always recommended to update the EKS cluster every six months, when a new EKS version is released, unless the application needs to run on a specific version of Kubernetes. 

This post explains the steps to upgrade your EKS cluster from v1.18 to v1.21 with zero downtime. 

EKS Upgrade (v1.18 to v1.19) – Steps 

Below are the detailed steps to upgrade the cluster as well as migration of applications to worker nodes from v1.18 to v1.19: 

 1. Upgrade Control Plane 

2. Upgrade kube-proxy version 

3. Upgrade CoreDns version 

4. Upgrade Amazon VPC CNI version 

5. Create PDB for all services 

6. Edit Deployment for application 

7. Update Node Group in AWS 

Step 1: Update Control Plane 

Before updating the Control plane, check the version of your Kubernetes cluster and worker node. If your worker node version is older than your current Kubernetes version, then you must update your worker node first then only proceed with updating your control plane. 

Check Kubernetes Control Plane version and worker node from the following command: 

kubectl version –short 

kubectl get nodes 

Since the Pod Security Policy(PSP) admission controller is enabled by default from 1.13 and later versions of Kubernetes, we need to make sure that a proper pod security policy is in place, before updating the Kubernetes version on the Control Plane. 

Check the default security policy using the command below: 

kubectl get psp eks.privileged 

After applying all the changes, now update the cluster from the AWS console. 

1. Login to AWS console and click on update now in the EKS cluster which you want to update. During the process of updating, you cannot make any changes on your cluster and no scheduling of the nodes will take place. So, it is preferred to do this activity during non-peak, non-business hours. 

2. After the update, the IP of the API server endpoint will change so, if there is any dependency on deployment with these IPs then you must change old IPs with New IPs. 

Once it is upgraded, update the EKS version in Terraform YAMLs. 

Step 2: Update kube-proxy version 

We will patch the kube-proxy daemonset to use the image that corresponds to the cluster’s Region and current Kubernetes version. Retrieve the kube-proxy version: 

kubectl get daemonset kube-proxy –namespace kube-system -o=jsonpath='{$.spec.template.spec.containers[:1].image}’

Update the kube-proxy: 

For 1.19: 

kubectl set image daemonset.apps/kube-proxy -n kube-system kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.19.6-e 

For 1.20: 

kubectl set image daemonset.apps/kube-proxy -n kube-system kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.20.4-e 

For 1.21: 

kubectl set image daemonset.apps/kube-proxy -n kube-system kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.21.2-e 

Ref: https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html 

Step 3: Upgrade Core DNS version 

Check if the cluster is running on CoreDNS and its version: 

kubectl get pod -n kube-system -l k8s-app=kube-dns 

Describe CoreDNS version: 

kubectl describe deployment coredns –namespace kube-system | grep Image | cut -d “/” -f 3 

Retrieve CoreDNS version and URL using the command below: 

kubectl get deployment coredns –namespace kube-system -o=jsonpath='{$.spec.template.spec.containers[:1].image}’ 

Replace CoreDNS version and URL 

For 1.19: 

   kubectl set image –namespace kube-system deployment.apps/coredns coredns=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.0-e

For 1.20: 

kubectl set image –namespace kube-system deployment.apps/coredns coredns=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.3-e For 1.21: 

 kubectl set image –namespace kube-system deployment.apps/coredns coredns=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.4-e Ref: https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html 

Step 4: Upgrade Amazon VPC CNI version 

Check the version of the VPC CNI: 

kubectl describe daemonset aws-node –namespace kube-system | grep Image | cut -d “/” -f 2 

The output should be like this: 

amazon-k8s-cni-init:v1.7.5-eksbuild.1 

amazon-k8s-cni:v1.7.5-eksbuild.1 

Update VPC CNI version: 

curl -o aws-k8s-cni.yaml https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.10/config/master/aws-k8s-cni.yaml 

For Oregon region: 

kubectl apply -f aws-k8s-cni.yaml 

For Singapore region: 

sed -i -e ‘s/us-west-2/ap-southeast-1/’ aws-k8s-cni.yaml (if region is other than us-west-2) 

kubectl apply -f aws-k8s-cni.yaml 

Ref: https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html 

Step 5: Create PDB for all the services 

Create pdb.yml file same as below for all services: 

apiVersion: policy/v1beta1 

kind: PodDisruptionBudget 

metadata: 

name: <service-name>-pdb 

namespace: default 

spec: 

minAvailable: 100% 

selector: 

matchLabels: 

app.kubernetes.io/name: <service-name> 

Step 6: Edit Deployment for application 

We can use open-source tool called kubent for checking which deployment Yaml needs to be updated for the versions we are going to upgrade to: 

sh -c “$(curl -sSL https://git.io/install-kubent)” 

kubent –target-version 1.19.6 

Change the API Version for all the services in their deployment Yaml and deploy it and recheck using the same command till it shows Deprecated APIs. 

Ref: https://github.com/doitintl/kube-no-trouble 

Step 7: Update Node Group in AWS 

Go to the EKS cluster in the AWS console and under Configuration → Compute, click update now Under Node Group and use Rolling update Under Update Strategy and wait for the upgrade to be completed. 

If you get the error PodEvictionFailure , then it is because PodDisruptionBudget is not configured properly. 

EKS Upgrade (v1.19 to v1.20) – Steps 

Follow the same steps described above for v1.18 to v1.19 

EKS Upgrade (v1.20 to v1.21) – Steps 

Follow the same steps described above for v1.18 to v1.19 

References 

https://blogs.halodoc.io/aws-eks-upgrade/ 

https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html 

https://docs.aws.amazon.com/eks/latest/userguide/eks-linux-ami-versions.html 

https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html 

https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html