KEDA — Kubernetes Event-Driven Autoscaling — solves a problem that standard Kubernetes Horizontal Pod Autoscaler cannot: scaling workloads based on the depth of a message queue, the lag on a Kafka topic, or the number of rows in a database table. For IoT data pipelines and edge Kubernetes clusters, where work arrives in bursts and idle periods need to cost nothing, KEDA’s scale-to-zero capability is the enabling technology that makes event-driven architectures on Kubernetes practical.
This guide covers KEDA’s architecture, the scalers most relevant to IoT and edge deployments, and practical configuration for common patterns.
What KEDA Does
Standard Kubernetes HPA scales pods based on CPU and memory metrics. This works for request-driven web services but fails for queue consumers: if your pod isn’t processing anything, CPU is idle regardless of how many messages are queued. HPA sees low CPU and does nothing. The queue backs up.
KEDA introduces a third scaling mode alongside HPA and VPA: event-driven scaling. A KEDA ScaledObject watches an external source — an MQTT subscription, a Kafka consumer group, an Azure Service Bus queue, a Prometheus metric, a PostgreSQL query result — and scales the target deployment based on what it finds there.
Crucially, KEDA scales to zero when the source is empty. When messages arrive, it scales from zero to the configured minimum. This eliminates the cost of idle consumer pods, which is particularly valuable at the edge where compute resources are constrained and billing is often consumption-based.
Architecture
KEDA runs as two components in your cluster:
- keda-operator: watches
ScaledObjectandScaledJobresources and manages the HPA that drives scaling - keda-metrics-apiserver: exposes external metric values to Kubernetes as a custom metrics API so HPA can act on them
[External Source (MQTT/Kafka/DB)]
↓
[KEDA keda-operator polls source]
↓
[Writes metric to keda-metrics-apiserver]
↓
[HPA reads external metric via Custom Metrics API]
↓
[Scales target Deployment/StatefulSet]KEDA installs via Helm:
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda \
--namespace keda \
--create-namespaceScaledObject Basics
A ScaledObject binds a scaling target (a Deployment, StatefulSet, or custom resource) to a trigger (the event source). Here’s the structure:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: iot-processor-scaledobject
spec:
scaleTargetRef:
name: iot-data-processor # Deployment name
minReplicaCount: 0 # Scale to zero when idle
maxReplicaCount: 20 # Never exceed 20 pods
pollingInterval: 15 # Check source every 15 seconds
cooldownPeriod: 60 # Wait 60s after last event before scaling down
triggers:
- type: mqtt # Trigger type
metadata:
...minReplicaCount: 0 is the scale-to-zero setting. When the trigger reports zero events pending, KEDA removes all pods. When new events arrive, KEDA scales from zero to one pod (bypassing HPA’s minimum) and then lets HPA scale further based on the metric.
MQTT Trigger for IoT Sensor Data
MQTT is the dominant protocol for IoT sensor data, and KEDA’s MQTT scaler monitors topic subscription depth. This is particularly useful when sensor devices publish at variable rates and you want processing pods to appear only when data is waiting.
The MQTT scaler requires a ScaledObject and a TriggerAuthentication for broker credentials:
apiVersion: v1
kind: Secret
metadata:
name: mqtt-secret
stringData:
username: "keda-consumer"
password: "your-mqtt-password"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: mqtt-trigger-auth
spec:
secretTargetRef:
- parameter: username
name: mqtt-secret
key: username
- parameter: password
name: mqtt-secret
key: password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sensor-processor-scaler
spec:
scaleTargetRef:
name: sensor-data-processor
minReplicaCount: 0
maxReplicaCount: 10
pollingInterval: 10
cooldownPeriod: 120
triggers:
- type: mqtt
authenticationRef:
name: mqtt-trigger-auth
metadata:
brokerList: "tcp://mosquitto.iot-system.svc.cluster.local:1883"
topic: "sensors/temperature"
qos: "1"
lagThreshold: "5" # Scale up when >5 messages are pending per replicalagThreshold controls the scaling sensitivity. With lagThreshold: "5", KEDA targets one replica per 5 pending messages — a queue of 25 messages triggers 5 replicas.
Kafka Scaler for High-Throughput IoT Streams
For high-throughput IoT scenarios using Kafka (or Redpanda) as the message bus, the Kafka scaler measures consumer group lag — the difference between the latest offset and the consumer’s current offset.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: iot-kafka-processor
spec:
scaleTargetRef:
name: kafka-stream-processor
minReplicaCount: 0
maxReplicaCount: 30
pollingInterval: 15
triggers:
- type: kafka
metadata:
bootstrapServers: "kafka.iot-platform.svc.cluster.local:9092"
consumerGroup: "iot-data-processors"
topic: "raw-sensor-data"
lagThreshold: "100" # One replica per 100 messages of lag
offsetResetPolicy: "latest"For Kafka deployments with TLS and SASL authentication, KEDA’s TriggerAuthentication supports the full range of Kafka authentication mechanisms.
The Kafka scaler is well-suited to edge Kubernetes clusters running near IoT gateways, where you want processors to consume sensor telemetry within seconds of arrival but idle to zero when sensor networks are quiet (overnight, weekend, planned maintenance windows).
PostgreSQL / TimescaleDB Trigger
Many IoT platforms land sensor data in a time-series database (TimescaleDB, regular PostgreSQL with partitioning, or InfluxDB) and then need batch processing or alerting workers to process unprocessed rows. The PostgreSQL scaler runs a query against the database and scales based on the returned value.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: timescale-auth
spec:
secretTargetRef:
- parameter: connection
name: timescale-secret
key: connectionString
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: anomaly-detector-scaler
spec:
scaleTargetRef:
name: anomaly-detection-worker
minReplicaCount: 0
maxReplicaCount: 5
pollingInterval: 30
triggers:
- type: postgresql
authenticationRef:
name: timescale-auth
metadata:
query: |
SELECT COUNT(*) FROM sensor_readings
WHERE processed = false
AND reading_time > NOW() - INTERVAL '1 hour'
targetQueryValue: "50" # One replica per 50 unprocessed rows
dbName: "iot_platform"
sslmode: "require"This pattern is useful for anomaly detection and alerting workers that should only run when there’s a meaningful backlog of unprocessed readings, and should idle to zero otherwise.
KEDA on k3s at the Edge
KEDA works well with k3s, the lightweight Kubernetes distribution popular for edge and IoT deployments. k3s runs on hardware as modest as Raspberry Pi 4 and ARM-based industrial gateways, and KEDA’s memory footprint is compatible with constrained edge nodes.
For an edge k3s cluster processing sensor data from a factory floor:
# Install k3s on edge node
curl -sfL https://get.k3s.io | sh -
# Install KEDA on the edge cluster
helm install keda kedacore/keda \
--namespace keda \
--create-namespace \
--set operator.resources.requests.memory=64Mi \
--set operator.resources.limits.memory=256Mi \
--set metricsServer.resources.requests.memory=64Mi \
--set metricsServer.resources.limits.memory=256MiReducing resource requests for edge deployments ensures KEDA doesn’t crowd out workloads on nodes with limited RAM (4–8GB is common on edge hardware).
ScaledJob for Batch Processing
Beyond scaling Deployments, KEDA supports ScaledJob — which creates Kubernetes Jobs rather than scaling existing pods. This is ideal for IoT batch processing where each unit of work should run in an isolated, short-lived container.
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: firmware-analysis-jobs
spec:
jobTargetRef:
template:
spec:
containers:
- name: firmware-analyzer
image: your-registry/firmware-analyzer:latest
env:
- name: MQTT_BROKER
value: "tcp://mosquitto:1883"
restartPolicy: Never
triggers:
- type: mqtt
metadata:
brokerList: "tcp://mosquitto.iot-system.svc.cluster.local:1883"
topic: "firmware/updates/pending"
lagThreshold: "1" # One job per pending message
maxReplicaCount: 10
rollout:
strategy: gradual # Don't delete running jobs when scaling downScaledJob with lagThreshold: "1" creates exactly one Job per pending message — each firmware update is processed in its own isolated container, and the containers disappear when done.
Observability
KEDA exposes Prometheus metrics at the /metrics endpoint on the operator and metrics server pods. The most useful metrics for IoT deployments are:
keda_scaled_object_errors_total— trigger evaluation failureskeda_resource_totals— count of ScaledObjects by state- External metric values (source-specific, e.g. Kafka lag, queue depth) are visible as Kubernetes custom metrics
For real-time visibility into scaling behaviour, the KEDA dashboard (available as a Grafana dashboard at dashboard ID 16907) shows target replicas, current replicas, and trigger values over time.
Common Pitfalls
Scale-to-zero and cold start latency: When a deployment scales from zero, the time to schedule a pod, pull the container image, and start the process adds latency to the first message processed after an idle period. For latency-sensitive IoT scenarios, consider minReplicaCount: 1 to keep at least one warm pod rather than scaling to zero.
Polling interval vs. latency: KEDA checks the trigger source on pollingInterval. A 30-second polling interval means up to 30 seconds between message arrival and scale-up initiation. For time-sensitive processing, reduce pollingInterval to 5–10 seconds — but be aware this increases the number of API calls to the trigger source.
Cooldown and flapping: Set cooldownPeriod long enough that brief idle gaps between message bursts don’t cause constant scale-up/scale-down cycles. For IoT workloads with periodic sensor bursts, 120 to 300 seconds is a reasonable starting point.