Your first Kubernetes cluster with kubeadm

The kubeadm tool makes it possible to bootstrap a minimum viable Kubernetes cluster. Its advantages include simplicity, availability on almost any system, and integration with automated provisioning and cluster lifecycle management. It also supports consistent, interoperable clusters within the Certified Kubernetes Conformance Program.

In general, creating and operating a Kubernetes cluster involves more than just managing its individual components: it requires attention to several details and an ordered set of instructions, often scattered across different documentation. This guide walks through the full bootstrapping process using the following components and versions:

• Ubuntu 22.04.3 LTS
• Docker Engine 24.0.6
• cri-dockerd 0.3.4
• Kubernetes 1.27.5
• Calico 3.26.1

This post assumes you’re already familiar with Kubernetes cluster components and is intended as a hands-on guide.

Before starting

Before you begin, make sure every node that will join the cluster has the required settings in place.

Forwarding IPv4 and letting iptables see bridged traffic

Pods must communicate transparently across the cluster, regardless of the node on which they run, so each node needs traffic forwarding enabled. You must also ensure that the filesystem overlay used to compose container layers is available.

Create a file containing the names of the kernel modules that should be loaded at boot time:

1
2
3
4

cat << EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

Then, load the kernel modules:

1
2

sudo modprobe overlay
sudo modprobe br_netfilter

Verify that the modules are loaded:

1
2

lsmod | grep br_netfilter
lsmod | grep overlay

Now create a file containing the system variable configuration to load at boot time:

1
2
3
4
5

cat << EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

Apply the kernel parameters at runtime so that the changes above take effect without rebooting:

1

sudo sysctl --system

Finally, verify that these system variables are set to 1:

1

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Verifying MAC and product_uuid uniqueness

Some components use the network adapter ID (MAC address) and/or the motherboard ID (product_uuid) to uniquely identify the nodes.

Verify the MAC address is unique for every node:

1

ip link

Verify the product_uuid is unique for every node:

1

sudo cat /sys/class/dmi/id/product_uuid

Checking required ports

When running Kubernetes in an environment with strict network boundaries, be aware of the ports and protocols used by its components, and verify that they are open to allow communication between them.

On control-plane / master node(s):

1

nc -zv 127.0.0.1 6443 2379-2380 10250 10259 10257

On worker node(s):

1

nc -zv 127.0.0.1 10250 30000-32767

Disabling swap memory

Cluster components are designed primarily for performance and reliability. Using swap under disk pressure makes their behavior less predictable.

Disable swapping on all known swap devices and files:

1

sudo swapoff -a

Verify that swap has been disabled:

1

free -h

Remove the unnecessary swap file:

1

sudo rm /swap.img

Finally, comment out the relevant line in fstab to prevent swap from being mounted at startup:

1
2
3

sudo vi /etc/fstab
... 
# /swap.img       none    swap    sw      0       0

Installing the container runtime

Installing Docker Engine

Next, install a container runtime that conforms to the Container Runtime Interface (CRI) so that pods and containers can run on each node in the cluster.

Since we are installing Docker Engine, add Docker’s official GPG key:

1
2
3

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Set up the repository:

1
2
3
4

echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update the APT package index and install the packages required to use a repository over HTTPS:

1

sudo apt update && sudo apt install -y ca-certificates curl gnupg

Install Docker Engine, containerd, and Docker Compose:

1

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

To install a specific version, first list the available versions for each package:

1 2 3 4 5

apt list -a docker-ce | awk '{print $2}' apt list -a docker-ce-cli | awk '{print $2}' apt list -a containerd.io | awk '{print $2}' apt list -a docker-buildx-plugin | awk '{print $2}' apt list -a docker-compose-plugin | awk '{print $2}'

Then specify the version in the installation command:

1	sudo apt install -y docker-ce=<version> docker-ce-cli=<version> containerd.io=<version> docker-buildx-plugin=<version> docker-compose-plugin=<version>

Add your user to the docker group to manage Docker as a non-root user:

1

sudo usermod -aG docker $USER

Refresh your group membership without logging out and back in:

1

newgrp docker

Finally, verify that you can run Docker commands without sudo:

1

docker run hello-world

You should see output similar to this:

1
2
3

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

Configure Docker to start on boot:

1
2

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Installing cri-dockerd

Originally, Docker Engine was integrated directly into the kubelet code. When Kubernetes moved to use the CRI layer, a temporary adapter called dockershim was added between the CRI and Docker Engine. With Kubernetes 1.24, dockershim was removed from the Kubernetes core, so users need to install the third-party cri-dockerd adapter to integrate Docker Engine with Kubernetes.

Use the prebuilt cri-dockerd package to install the binary and set up the system to run it as a service:

1
2

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb -P /tmp
sudo apt install -y /tmp/cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

Verify that the service is running and listening on unix:///var/run/cri-dockerd.sock, the default endpoint socket:

1
2

systemctl status cri-docker.service
systemctl status cri-docker.socket

Installing Kubernetes tools

Next, install the cluster-bootstrap tool (kubeadm), the component that starts pods and containers (kubelet), and the command-line tool used to communicate with the cluster (kubectl).

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control-plane you want kubeadm to install for you.

Update the APT package index and install the packages required to use the Kubernetes APT repository:

1

sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl

Download the Google Cloud public signing key:

1

curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

Add the Kubernetes apt repository:

1

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update the APT package index and install the latest versions of kubelet, kubeadm, and kubectl:

1

sudo apt update && sudo apt install -y kubelet kubeadm kubectl

To install a specific version, first list the available package versions:

1	curl -s https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages | grep Version | awk '{print $2}' | sort -V | uniq

Then specify the version in the installation command:

1	sudo apt install -y kubelet=<version> kubeadm=<version> kubectl=<version>

Pin the kubelet, kubeadm, and kubectl versions:

1

sudo apt-mark hold kubelet kubeadm kubectl

Enable kubectl autocompletion:

1

echo 'source <(kubectl completion bash)' >> ~/.bashrc

Extend shell completion to work with the k alias (optional):

1
2

echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -F __start_kubectl k' >> ~/.bashrc

Reload .bashrc so that the new configuration takes effect in the current session:

1

source ~/.bashrc

kubectl is supported within one minor version (older or newer) of kube-apiserver.

To check the versions of kubectl and kube-apiserver currently in use, inspect the client and server versions in the output of the following command:

1	kubectl version

Installing the cluster control-plane components

The control-plane node is where the cluster’s main decisions are made. It runs components such as etcd, the cluster database, and the API server, which the kubectl command-line tool communicates with.

Initialize the control-plane node by specifying a suitable CIDR block for the CNI-based pod network add-on and the container runtime endpoint:

1

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock

Make sure that the pod network does not overlap with any host network.

Copy the kubeadm join command, including its token and discovery-token-ca-cert-hash, so that you can add nodes to the cluster later:

1	kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Make kubectl work for your non-root user:

1
2
3

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Installing the pod network add-on

Now deploy a Container Network Interface (CNI)-based pod network add-on so that pods can communicate with one another.

This guide uses Calico, a widely adopted, battle-tested, open-source Kubernetes networking solution that provides two major services for cloud-native applications:

• Network connectivity between workloads.
• Network security policy enforcement between workloads.

Requirements

Create the following configuration file to prevent NetworkManager from interfering with the interfaces:

1
2
3
4
5

sudo mkdir -p /etc/NetworkManager/conf.d/
cat << EOF | sudo tee /etc/NetworkManager/conf.d/calico.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
EOF

Installing Calico with the operator

Get the latest stable version tag from the projectcalico/calico repository:

1

VERSION=$(curl -sL https://api.github.com/repos/projectcalico/calico/releases/latest | jq -r ".name")

Install the Calico operator and the custom resource definitions (CRDs):

1

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/$VERSION/manifests/tigera-operator.yaml

Download the custom resources needed to configure Calico, then customize the manifest for the 10.244.0.0/16 CIDR:

1
2

curl https://raw.githubusercontent.com/projectcalico/calico/$VERSION/manifests/custom-resources.yaml -O
sed -i 's/cidr:.*/cidr: 10\.244\.0\.0\/16/' custom-resources.yaml

Finally, apply the manifest to install Calico:

1

kubectl create -f custom-resources.yaml

Wait until each pod is running:

1

watch kubectl get pods -n calico-system

Checking the installation

Cluster DNS (CoreDNS) starts only after a network is installed correctly.

Verify that CoreDNS pods are running:

1

kubectl get pods --all-namespaces

Verify that the node is also ready:

1

kubectl get nodes -o wide

Remove the control-plane taint so that you can schedule pods on it:

1

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

If you plan to keep the control-plane node dedicated to control-plane workloads, you can skip the previous step. However, at least one worker node must join the cluster later so that you can deploy your applications.

Installing calicoctl

The calicoctl command-line tool manages Calico network and security policies, endpoint configuration, and Calico node instances.

To install calicoctl as a binary on a single host:

1
2
3
4

POD=$(kubectl -n calico-system get pod -l k8s-app=calico-kube-controllers -o jsonpath="{.items[0].metadata.name}")
VERSION=$(kubectl -n calico-system describe pod $POD | grep Image: | cut -d ':' -f3)
sudo curl -L https://github.com/projectcalico/calico/releases/download/$VERSION/calicoctl-linux-amd64 -o /usr/local/bin/calicoctl
sudo chmod +x /usr/local/bin/calicoctl

Verify that the command was installed correctly:

1

calicoctl version

Make sure you always install the version of calicoctl that matches the version of Calico running on your cluster.

Joining a new worker node

Whether or not you allow pods to run on the control-plane node, run this command on every machine you want to join to the existing cluster:

1

sudo kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Verify that the new node was added successfully by running the following command on the control-plane node:

1

kubectl get nodes

If you need to retrieve the token, run the following command on the control-plane node:

1	kubeadm token list

If you need to retrieve the discovery-token-ca-cert-hash, run the following command on the control-plane node:

1 2	openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'

If you are joining a node after the token has expired, run the following command on the control-plane node:

1	kubeadm token create --print-join-command

Deprovisioning a node cleanly

Everything that has a beginning has an end. If you want to deprovision a node and cleanly revert the changes made by kubeadm, first drain the node and make sure it is empty, then deconfigure it.

To mark the node as unschedulable, prevent new pods from arriving, and evict or delete existing pods, run:

1

kubectl drain <node_name> --delete-emptydir-data --force --ignore-daemonsets

Then reset the state created by kubeadm:

1

sudo kubeadm reset 

If you wish to reset the iptables rules manually, run:

1

sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X 

If you wish to reset the IPVS tables manually, run:

1

sudo ipvsadm -C

You can now safely remove the node from the cluster:

1

kubectl delete node <node_name>

You can now start again with kubeadm init or kubeadm join, using the appropriate arguments.

Controlling your cluster from machines other than the control-plane node

To use kubectl from another computer, copy the administrator kubeconfig file from the control-plane node to your workstation.

Assuming root SSH access is disabled on the control-plane node, copy the kubeconfig file to your home directory and make your user its owner:

1
2

sudo cp /etc/kubernetes/admin.conf $HOME/.
sudo chown $USER: admin.conf

Copy the kubeconfig file from the remote control-plane node and give it a name of your choice:

1
2

scp $USER@<control-plane-host>:~/admin.conf ~/.kube/
mv ~/.kube/admin.conf ~/.kube/config-new

Append the new kubeconfig file to your current KUBECONFIG environment variable and update your kubeconfig file:

1
2

export KUBECONFIG=~/.kube/config:~/.kube/config-new
kubectl config view --flatten > ~/.kube/config

Check that the new cluster is listed as a Kubernetes context and make it the current context:

1
2

kubectl config get-contexts
kubectl config set-context <cluter_name>

Conclusion

At this point, you should have a fully functional Kubernetes cluster up and running, with a working control plane, a healthy node registration process, and a kubectl setup that allows you to interact with it remotely. The steps covered in this post provide the foundation for deploying applications, testing workloads, and continuing your learning journey with more advanced topics such as networking, storage, ingress, and cluster hardening.

If you are setting up a personal lab or a small development environment, this configuration gives you a solid starting point that can be extended as your needs grow. In future posts, I’ll walk through practical examples such as deploying services, configuring networking, and exploring more advanced Kubernetes concepts.