Kubernetes vs K3s at the Edge: When Lightweight Wins

TL;DR:

• Full Kubernetes needs 800MB–1.1GB RAM just for the control plane — most edge hardware can’t clear that bar
• K3s replaces etcd with SQLite, cuts control plane RAM to ~512MB, and ships as a single binary
• Use K3s for multi-container fleets with teams that know Kubernetes; use systemd for single-purpose nodes

The question of Kubernetes at the edge gets answered poorly from both sides. Cloud-native teams say “use it everywhere.” Embedded teams say “never.” The right answer is narrower: K3s solves a real problem for a specific class of edge deployment, and systemd is better for everything else.

Why Full Kubernetes Fails at the Edge

Full Kubernetes — kubeadm-installed with etcd, API server, scheduler, controller manager, kubelet, and kube-proxy — has a resource floor that most edge hardware doesn’t clear.

Idle RAM by component:

Under load, control plane consumption reaches 2–4GB before your workload uses a single byte.

etcd is especially problematic at the edge. It’s designed for data centre conditions — low-latency SSD storage, stable network connectivity. SD cards and eMMC have high write latency and limited endurance. etcd’s write-ahead log pattern will produce operational failures on flash storage within months. If you’ve ever had an edge device slowly degrade and eventually stop working for no obvious reason, there’s a reasonable chance etcd was involved.

What K3s Changes

K3s (Rancher/CNCF) ships as a single ~100MB binary. Key changes: etcd is replaced with SQLite (better on flash storage, smaller RAM footprint; embedded dqlite available for HA configurations), legacy and alpha APIs are removed (reduces API server memory), and external components are bundled — containerd, Flannel, CoreDNS, Traefik, and local-path-provisioner included.

The result: K3s server (control plane) runs in ~512MB RAM. K3s agent (worker-only) uses ~75MB.

When K3s Makes Sense

K3s solves a real operational problem: managing containerised workloads on a fleet of edge nodes, across sites, with rolling updates, health checks, and a consistent deployment model.

Use K3s when a single node runs multiple containers (MQTT broker, time-series database, processing pipeline, dashboard) that need ordered startup and independent updates. When your team already uses Kubernetes — K3s lets you reuse kubectl, Helm, and Kustomize without retraining. When workloads need zero-downtime rolling updates. When you’re managing 50+ sites and want GitOps with Rancher Fleet. These are real use cases, and K3s handles them well.

When to Skip Orchestration Entirely

Over-engineering the deployment is a genuine failure mode. Use systemd when the node runs one container — 20 lines of systemd unit configuration beats a full Kubernetes cluster. When the device has under 1GB RAM. When your ops team knows Linux but not Kubernetes. When the node boots a custom Yocto image with exactly one application.

A systemd unit for a containerised edge application handles restart-on-failure, logs via journalctl, and is understood by any Linux administrator:

[Unit]
Description=Edge Data Collector
After=network.target docker.service

[Service]
Restart=always
RestartSec=10
ExecStart=/usr/bin/docker run --rm \
  --name edge-collector \
  --env-file /etc/edge-collector/env \
  -v /data/collector:/data \
  --network host \
  myregistry.example.com/edge-collector:1.4.2

[Install]
WantedBy=multi-user.target

Simple, readable, maintainable. Don’t add complexity you don’t need.

Deploying K3s on Ubuntu 22.04 arm64

This walkthrough uses a Raspberry Pi 5 running Ubuntu 22.04 LTS Server arm64.

Prerequisites: Ubuntu 22.04 arm64, booted from NVMe or USB SSD (not SD card), static IP or DHCP reservation, cgroups v2 (default on Ubuntu 22.04).

Install K3s server:

curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --node-name edge-node-01

--disable traefik saves ~50MB RAM if you don’t need an ingress controller. Allow 30–60 seconds for the control plane to initialise, then verify: kubectl get nodes.

Join an agent node:

curl -sfL https://get.k3s.io | K3S_URL=https://192.168.10.10:6443 \
  K3S_TOKEN=<node-token> sh -s - --node-name edge-node-02

Multi-Site Networking

Tailscale creates a mesh WireGuard VPN automatically — each node appears on the Tailnet regardless of physical location or NAT. It’s particularly useful for UK deployments spread across multiple sites. Pass the Tailscale IP to K3s:

curl -sfL https://get.k3s.io | sh -s - \
  --node-ip 100.64.1.1 \
  --advertise-address 100.64.1.1 \
  --node-name site-a-server

This gives you a geographically distributed K3s cluster without complex site-to-site VPN configuration.

Handling Intermittent Connectivity

K3s tolerates node disconnections — worker nodes keep running while disconnected from the server. Tune eviction grace periods to avoid premature pod eviction on cellular-connected nodes:

--kube-controller-manager-arg=node-monitor-grace-period=2m
--kube-controller-manager-arg=pod-eviction-timeout=5m

This gives nodes 5 minutes of offline time before pods are evicted — appropriate for typical cellular reconnect times on UK rural deployments.

K3s Limitations

SQLite is single-writer, so K3s isn’t for multi-thousand-node clusters. Embedded dqlite scales to tens of nodes comfortably, not hundreds. HA beyond 5 server nodes requires an external datastore (PostgreSQL, MySQL, or etcd), adding operational complexity. And K3s isn’t for real-time workloads — Kubernetes scheduling is not deterministic in timing. For sub-millisecond jitter requirements, run the real-time process outside the container runtime.

The Bottom Line

K3s is the right Kubernetes distribution for edge deployments where Kubernetes is the right choice at all. The SQLite datastore, single-binary install, and 512MB RAM footprint make it practical on the hardware that actually exists in the field. But the more important question is whether you need orchestration at all — if the node runs one process, a systemd unit is simpler, more reliable, and requires less expertise to operate.