There’s a useful way to think about the history of IoT intelligence. In the first generation, devices collected data and sent it somewhere else to be understood. In the second, you pushed pre-built models down to devices and ran inference locally — TensorFlow Lite on a Raspberry Pi, TinyML on a Cortex-M microcontroller. That was already a meaningful improvement. But you were still treating ML as something bolted onto existing hardware.

The third generation, which is genuinely arriving now in 2026 rather than being perpetually “just around the corner,” is hardware where the AI capability is designed in from the start. The NPU isn’t an afterthought — it’s a first-class block on the system-on-chip, sized to handle the inference workloads the device is expected to run, integrated with the memory architecture to support them efficiently. The software stack assumes ML is happening, not accommodates it reluctantly.

This matters for anyone designing IoT systems, because it changes what’s possible, what’s economic, and what you should be specifying.

What changed in the silicon

The shift became visible in 2024–2025 and is fully mainstream for new chip designs in 2026. A few concrete examples.

The Qualcomm QCS series (QCS6490, QCS8550) are now shipping in industrial camera and computer vision modules with 12–73 TOPS of NPU performance alongside their application processor cores. These aren’t phone chips — they’re variants tuned for industrial temperature ranges, longer product lifecycles, and the power budgets relevant to always-on monitoring applications.

The MediaTek Genio series targets AIoT specifically, with Genio 700 and Genio 1200 offering integrated APU performance alongside display, camera, and connectivity subsystems. Widely used by ODMs building smart retail devices, digital signage, and industrial HMIs.

At the low end, the ESP32-P4 (Espressif) adds an AI accelerator alongside the existing Xtensa cores — not enough for complex vision models, but sufficient for keyword spotting, anomaly detection on sensor time-series, and audio event detection. At roughly US$3–5 in volume, this changes the economics for sensors that previously had no AI capability at all.

The pattern is consistent across price points: dedicated inference acceleration is appearing everywhere, from $3 microcontrollers to $100+ industrial SoCs.

What this unlocks in practice

The most immediate change is that the use cases that previously required a local edge server or a cloud connection now fit on the sensor itself.

Vibration-based predictive maintenance is the clearest example. A motor health monitor that detects anomalies in vibration signature needs to process high-frequency sensor data and run a model against it in real time. Previously that meant a gateway or a cloud upload. Now a sub-£20 module with an integrated NPU can do it locally, with no connectivity required for the detection step — connectivity only kicks in when an anomaly is found and needs to be reported.

Audio event detection — detecting machinery sounds that indicate problems, or monitoring for safety events like alarms or falls in healthcare settings — similarly fits on-device at the sensor level. The model runs continuously, draws maybe 50–100mW, and only wakes the communications subsystem when something worth reporting happens. Battery-powered deployment becomes viable.

Computer vision on the sensor is the most dramatic shift. Smart cameras with integrated NPUs can run object detection, classification, or counting models locally, reporting only the semantic output (count of vehicles, presence of PPE, detection of anomaly) rather than streaming video. The implications for bandwidth, privacy, and connectivity cost are substantial.

What this means for system architecture

If you’re designing IoT systems now, or evaluating hardware for deployments that will run for five to ten years, a few things follow from this shift.

The gateway tier is shrinking for many applications. The traditional pattern — sensor to gateway to cloud — made sense when sensors were dumb and gateways were where intelligence lived. If your sensors can run inference, the gateway becomes a connectivity aggregator and data router rather than a processing node. For some applications it disappears entirely.

Model management becomes a device-level concern. If you’re running models on 10,000 sensors, you need OTA model updates, version management, and rollback — at the sensor tier, not just the gateway tier. This is a non-trivial operational problem. The platforms that handle it well (Mender, balena, AWS IoT Greengrass at the edge) are increasingly important.

Power budgeting needs to account for inference. An NPU draws power even when it’s running a lightweight model. For battery-powered deployments, you need to model the duty cycle of inference carefully — continuous inference at even 50mW is a significant drain over months. The hardware that gets this right uses aggressive clock gating and power domains so the NPU draws near-zero when it’s not actively running.

Quantisation matters more than model accuracy in isolation. A model with 94% accuracy that fits in 2MB of flash and runs in 8ms on the target NPU beats a model with 97% accuracy that needs 20MB and takes 40ms. The ML workflow for edge deployment has to include quantisation and profiling on actual target hardware, not just benchmark numbers from the chip datasheet.

Specifying new hardware: questions worth asking

When evaluating chips or modules for a new design, ask the following.

What’s the NPU topology, and which operation types does it accelerate? Not all NPUs are equal — some are optimised for convolutions (computer vision), others for recurrent or transformer operations (audio, time-series), others for a mix. Match the NPU to your workload.

What’s the memory bandwidth to the NPU? Inference is often memory-bandwidth-bound, not compute-bound. A chip with a fast NPU but shared memory bandwidth with the application processor will underperform its TOPS figure.

What’s the supported model format? ONNX, TFLite, and vendor-specific formats are all common. Check that your training framework exports to something the target NPU runtime supports without major conversion work.

What’s the thermal envelope for continuous inference? At industrial temperature ranges, the sustained power dissipation under continuous inference load should be in the datasheet. If it isn’t, ask — it’s the number that determines whether your enclosure design is adequate.

The era of treating AI as a cloud feature that IoT devices access remotely is ending. The era of treating it as a hardware capability that sensors have natively has arrived.