Containerization at the Edge: K3s, KubeEdge, and OTA Updates

TL;DR:

• Full Kubernetes needs 800MB–1.1GB RAM for the control plane alone — impractical on most edge hardware
• K3s (512MB footprint) and KubeEdge (cloud-managed edge nodes with offline autonomy) solve different problems
• Container image distribution at the edge requires local registries or image pre-pulling — don’t assume bandwidth for on-demand pulls

Containerisation at the edge sounds straightforward — containers are portable, containers simplify deployments, therefore run containers at the edge. The practical complications start with RAM. Full Kubernetes is a data centre technology; its resource floor is defined by data centre hardware. Edge devices — industrial gateways, retail kiosks, factory floor boxes — often run on 2–4GB RAM with flash storage. The question isn’t whether to containerise, but which orchestration layer (if any) makes sense for your hardware and operational model.

Why Full Kubernetes Fails at the Edge

The full Kubernetes control plane — etcd, API server, scheduler, controller manager, kubelet — needs 800MB–1.1GB at idle. On a 4GB device, that’s 25% of memory gone before your application starts. On a 2GB device, under 1GB is left for workloads. And etcd is specifically hostile to flash storage: its write-ahead log creates continuous small writes that degrade SD cards and eMMC within months. For devices running years without maintenance, that’s a real operational problem, not a theoretical one.

K3s: Kubernetes for Edge

K3s (Rancher, now CNCF) replaces etcd with SQLite, strips legacy and alpha APIs, and bundles everything into a single ~100MB binary. The result:

• Control plane RAM: ~512MB
• Worker-only agent RAM: ~75MB
• Install time: 30 seconds
• Includes: containerd, Flannel, CoreDNS, Traefik (optional), local-path-provisioner

K3s is right when a device runs multiple containers and your team wants a familiar Kubernetes API. Helm charts, kubectl skills, and GitOps workflows transfer directly. The SQLite backend handles hundreds of nodes; beyond that, plug in an external datastore (PostgreSQL, MySQL).

One limitation to be aware of: K3s is a single-cluster model. Every edge node that’s a K3s server maintains an independent cluster. Managing hundreds of independent K3s clusters requires a fleet management layer — Rancher Fleet, ArgoCD, or Flux.

MicroK8s: The Snap Alternative

MicroK8s (Canonical) is K3s’s main alternative — a single-package Kubernetes distribution that installs via snap on Ubuntu and is the recommended Kubernetes distribution for Ubuntu-based edge hardware. It uses the same etcd as upstream Kubernetes but with sensible defaults for constrained environments.

MicroK8s’s addon system (microk8s enable dns storage) simplifies configuration but adds startup overhead. Because it uses etcd internally, the flash storage concern applies — use NVMe or SSD, not SD cards, for MicroK8s nodes.

KubeEdge: Cloud-Managed Edge Nodes

KubeEdge takes a fundamentally different architecture. Rather than running an independent Kubernetes cluster at the edge, it extends a central cloud Kubernetes cluster to edge nodes via a lightweight edge agent.

KubeEdge’s offline autonomy is its defining feature. When an edge node loses connectivity to the cloud control plane, it keeps running its current workloads — pod specs and config are cached locally, and the agent reconciles state when connectivity restores. This is critical for cellular or satellite-connected deployments, where intermittent connectivity is the norm.

The tradeoff: KubeEdge requires a healthy cloud Kubernetes cluster as its control plane. If that goes down, you can’t deploy or change edge workloads even if edge nodes are fine. For air-gapped or cloud-independent deployments, K3s is the better fit.

Container Registries at the Edge

Cloud container registries assume fast internet access. Edge deployments often have slow or metered connections — pulling a 2GB image over cellular is expensive and slow.

Local registry mirroring is the standard solution. Run a registry mirror (Harbor, or Docker’s registry:2 image) on an edge server or local fog node. Configure containerd to pull from the local mirror first, falling back to the upstream registry:

# /etc/rancher/k3s/registries.yaml
mirrors:
  "docker.io":
    endpoint:
      - "http://registry.local:5000"
      - "https://registry-1.docker.io"

Image pre-pulling uses DaemonSets or node provisioning scripts to pull required images before workloads are scheduled. K3s supports image pre-loading via tarball (k3s ctr images import image.tar), so you can distribute images via USB or local network without internet access.

Spegel (peer-to-peer image distribution for Kubernetes) lets nodes pull images from each other — useful for fleets of similar devices on a local network where one node has already pulled the image.

OTA Updates for Edge Containers

OTA update strategies differ by orchestration model:

K3s with Rancher Fleet is GitOps-based — Fleet agents pull desired state from Git and apply updates when manifests change.

KubeEdge pushes updates from the cloud Kubernetes API; rolling update strategies work as in standard Kubernetes.

Mender.io handles OS-level updates (full image or delta) and container workload updates independently, with rollback and staged rollouts.

Balena manages containerised applications, OS updates, and device health monitoring together — well suited to IoT fleets where you want everything in one place.

Always implement staged rollouts (10% → 25% → 100%) and automated rollback on health check failure for any fleet with reliability requirements.

The Bottom Line

Use K3s when you need multi-container orchestration on fleet-managed edge nodes and your team knows Kubernetes. Use KubeEdge when you need a central control plane managing many edge nodes with intermittent connectivity. Skip orchestration entirely (systemd + Docker) when a node runs one container and operational simplicity matters more than deployment automation. Whatever you choose, solve the image distribution problem before deploying at scale — on-demand registry pulls over slow links are a production failure waiting to happen.