If you’re running IoT deployments at any meaningful scale — a thousand devices, ten thousand, more — you’ve probably already hit the wall that MQTT alone puts up. MQTT brokers like Mosquitto or HiveMQ are excellent for device-to-cloud communication, but they’re not designed to be the backbone of a high-throughput data pipeline. That’s where Apache Kafka comes in.

The MQTT-to-Kafka pattern is now fairly standard in industrial IoT architectures, and for good reason: it combines MQTT’s lightweight, protocol-efficient device communication with Kafka’s durable, replayable, horizontally scalable event streaming. Getting it right, though, takes more thought than plugging them together and hoping for the best.

Why you need both

MQTT is a publish-subscribe protocol designed for constrained devices and unreliable networks. It handles QoS levels, last-will messages, persistent sessions for reconnecting devices, and all the edge-case device connectivity behaviour you need when you’re talking to hardware over cellular or LoRaWAN. What it doesn’t do is durable storage, stream replay, consumer group scaling, or the kind of backpressure management you need when a downstream processor falls behind.

Kafka does all of that. Topics in Kafka are partitioned, replicated logs — messages are retained for a configurable period (days or weeks in most IoT deployments), consumers can replay from any offset, and you can add consumers independently without affecting producers. When you have multiple downstream systems consuming the same telemetry — a time-series database, a real-time alerting engine, a machine learning pipeline — Kafka lets them all read at their own pace from the same topic.

The combination gives you what neither provides alone: efficient device-side communication from MQTT, and durable, replayable, scalable streaming from Kafka.

The bridge pattern

Connecting MQTT to Kafka requires a bridge component. The most widely used approaches are:

Confluent’s MQTT Source Connector (part of Kafka Connect) is the production-grade option for teams already running Confluent Platform. It subscribes to MQTT topics, transforms messages, and writes to Kafka topics. It handles reconnection, offset tracking, and schema mapping, and it integrates with Confluent Schema Registry for Avro or Protobuf serialisation.

For teams not on Confluent, HiveMQ’s Kafka Extension does the same job within the HiveMQ broker itself, which eliminates a network hop. EMQX, another popular MQTT broker, has built-in Kafka integration in its Enterprise edition.

For lightweight deployments or custom requirements, a simple bridge using a Python consumer (Paho MQTT + confluent-kafka) is perfectly reasonable and gives you full control over transformation logic.

Partitioning: getting it right for IoT

This is where most IoT-Kafka deployments make a consequential early decision that’s painful to undo. The standard advice is to partition by device ID. This ensures that all messages from a given device land on the same partition, preserving ordering per device — which matters if you’re doing state machine processing, anomaly detection, or any logic that depends on the sequence of readings from a single sensor.

What this means practically: your partition count should be at least as large as your anticipated peak consumer parallelism, but your partition key should be device_id (or a derivative of it). For a deployment with 50,000 sensors and an expectation of reading up to 20 Kafka consumers in parallel, 50+ partitions with device_id as the key is a reasonable starting point.

A common mistake is partitioning by topic name or message type instead. That seems sensible from a logical organisation standpoint, but it means all readings from the same device can end up on different partitions, in different orders, which makes per-device stream processing significantly more complex.

Schema management

IoT data tends to evolve. A firmware update on a sensor fleet might add new fields, change units, or restructure the message. Without schema management, your consumers eventually break when they encounter messages that don’t match their expectations.

Using a Schema Registry (Confluent’s is the most common; there are open-source alternatives) with Avro or Protobuf serialisation solves this. Messages are serialised with a schema ID embedded; consumers can fetch the schema for any message they receive, and schema evolution rules (backward, forward, full compatibility) are enforced centrally. It adds a small amount of operational overhead but saves a lot of pain as your device firmware evolves.

Do you actually need Kafka?

Here’s the honest question: not every IoT deployment needs Kafka. If you have fewer than a few hundred devices, moderate message rates, and a single downstream consumer, a managed MQTT broker and a cloud-native stream service (AWS Kinesis, Azure Event Hubs, Google Pub/Sub) may be simpler and cheaper to operate.

Kafka makes sense when you have: high message volumes (tens of thousands of messages per second); multiple independent downstream consumers; a requirement for stream replay and durable retention; or complex stream processing using Kafka Streams or Apache Flink.

The operational cost of a Kafka cluster — even managed via Confluent Cloud or AWS MSK — is real, and it deserves honest evaluation against a simpler architecture before you commit. The principle of not adding infrastructure complexity you haven’t earned applies as much to Kafka as anything else.

That said, for industrial IoT deployments at scale, Kafka’s combination of durability, throughput, and ecosystem integration (Flink, Spark, dbt, the full modern data stack) makes it hard to replace. The MQTT-to-Kafka bridge is, at this point, the standard architecture for a reason.