Embedded etcd (HA) may have performance issues on slower disks such as Raspberry Pis running with SD cards.
Why An Odd Number Of Server Nodes?
HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum. For a cluster with n servers, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail.
An HA K3s cluster with embedded etcd is composed of:
- Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore.
- Optional: Zero or more agent nodes that are designated to run your apps and services
- Optional: A fixed registration address for agent nodes to register with the cluster
Provision K3s Control Plane Node
Requirements. Execute on All nodes.
cat<<EOF >> /etc/hosts
10.4.19.21 raja-k3s-server1 raja-k3s-server1.rjhaikal.my.id
10.4.19.22 raja-k3s-server2 raja-k3s-server2.rjhaikal.my.id
10.4.19.23 raja-k3s-server3 raja-k3s-server3.rjhaikal.my.id
10.4.19.24 raja-k3s-agent1 raja-k3s-agent1.rjhaikal.my.id
10.4.19.25 raja-k3s-agent2 raja-k3s-agent2.rjhaikal.my.id
10.4.19.26 raja-k3s-agent3 raja-k3s-agent3.rjhaikal.my.id
10.4.19.27 raja-lb1 raja-lb1.rjhaikal.my.id
10.4.19.28 raja-lb2 raja-lb2.rjhaikal.my.id
10.4.19.29 raja-nfs
10.4.19.100 raja-k3s-vip.rjhaikal.my.id
EOF
timedatectl set-timezone Asia/Jakarta
apt update -y ; apt upgrade -y
To get started, first launch a server node with the cluster-init flag to enable clustering and a token that will be used as a shared secret to join additional servers to the cluster.
Execute on server1 node
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server \
--cluster-init \
--tls-san=raja-k3s-server1.rjhaikal.my.id # Optional, needed if using a fixed registration address
After launching the first server, join the second and third servers to the cluster using the shared secret:
Execute on server2 & server3 node
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server \
--server https://raja-k3s-server1.rjhaikal.my.id:6443 \
--tls-san=raja-k3s-server2.rjhaikal.my.id # Optional, needed if using a fixed registration address
Check to see that the second and third servers are now part of the cluster:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
raja-k3s-server1 Ready control-plane,etcd,master 10m v1.30.5+k3s1 10.4.19.21 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-server2 Ready control-plane,etcd,master 5m47s v1.30.5+k3s1 10.4.19.22 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-server3 Ready control-plane,etcd,master 6m4s v1.30.5+k3s1 10.4.19.23 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
There are a few config flags that must be the same in all server nodes:
- Network related flags: --cluster-dns, --cluster-domain, --cluster-cidr, --service-cidr
- Flags controlling the deployment of certain components: --disable-helm-controller, --disable-kube-proxy, --disable-network-policy and any component passed to --disable
- Feature related flags: --secrets-encryption
Setup Load Balancer
HAProxy is an open source option that provides a TCP load balancer. It also supports HA for the load balancer itself, ensuring redundancy at all levels. See HAProxy Documentation for more info.
Additionally, we will use KeepAlived to generate a virtual IP (VIP) that will be used to access the cluster. See KeepAlived Documentation for more info.
Execute on lb1 and lb2 node.
1. Install HAProxy and KeepAlived:
sudo apt-get install haproxy keepalived
2. Add the following to /etc/haproxy/haproxy.cfg on lb-1 and lb-2:
frontend k3s-frontend
bind *:6443
mode tcp
option tcplog
default_backend k3s-backend
backend k3s-backend
mode tcp
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s
server server-1 10.4.19.21:6443 check
server server-2 10.4.19.22:6443 check
server server-3 10.4.19.23:6443 check
3. Add the following to /etc/keepalived/keepalived.conf on lb-1 and lb-2:
global_defs {
enable_script_security
script_user root
}
vrrp_script chk_haproxy {
script 'killall -0 haproxy' # faster than pidof
interval 2
}
vrrp_instance haproxy-vip {
interface ens3 # match to your interface
state <STATE> # MASTER on lb-1, BACKUP on lb-2
priority <PRIORITY> # 200 on lb-1, 100 on lb-2
virtual_router_id 51
virtual_ipaddress {
10.4.19.100/24
}
track_script {
chk_haproxy
}
}
4. Restart HAProxy and KeepAlived on lb-1 and lb-2:
systemctl restart haproxy
systemctl restart keepalived
Join Worker Nodes
On agent-1, agent-2, and agent-3, run the following command to install k3s and join the cluster:
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - agent --server https://raja-k3s-vip.rjhaikal.my.id:6443
You can now use kubectl from server node to interact with the cluster.
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
raja-k3s-agent1 Ready <none> 72s v1.30.5+k3s1 10.4.19.24 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-agent2 Ready <none> 34s v1.30.5+k3s1 10.4.19.25 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-agent3 Ready <none> 30s v1.30.5+k3s1 10.4.19.26 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-server1 Ready control-plane,etcd,master 28m v1.30.5+k3s1 10.4.19.21 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-server2 Ready control-plane,etcd,master 24m v1.30.5+k3s1 10.4.19.22 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
raja-k3s-server3 Ready control-plane,etcd,master 24m v1.30.5+k3s1 10.4.19.23 <none> Ubuntu 22.04.5 LTS 5.15.0-122-generic containerd://1.7.21-k3s2
Reference
