The argument for running AI in the cloud was always simplicity: unlimited compute, no deployment headaches, pay per inference. The argument against it, which is winning more industrial debates in 2026, is latency, connectivity, and cost at scale. If you’ve got 500 cameras on a factory floor and you’re sending every frame to a cloud model, you’re paying for bandwidth, waiting for round-trip latency, and creating a single point of failure. Running the model at the edge solves all three — if you can get the deployment right.

That “if” is doing a lot of work. Edge AI inference in industrial settings involves hardware constraints that don’t exist in a data centre, and getting a model from training to reliable production on a constrained device is a genuinely different skill set from cloud deployment.

Picking the Right Model Size

The first and most important decision is model selection. The best model for an edge application is not the most accurate model — it’s the most accurate model that runs within your hardware and latency budget.

In practice, this usually means starting with the nano or small variants of modern detection architectures. YOLOv8n runs at 120–140 FPS on a Raspberry Pi 5 with a Hailo-8L NPU. YOLOv8s drops to 60–80 FPS. If your camera is running at 30 FPS and you need near-real-time detection, the nano variant gives you 4x headroom for burst handling; the small variant still works. Move up to YOLOv8m only when the accuracy gap matters and your hardware budget allows.

For classification tasks (quality pass/fail on a conveyor, equipment state detection), MobileNetV2 or EfficientNet-Lite models are dramatically smaller than detection architectures while achieving production-quality accuracy for many industrial use cases. If you’re not doing localisation — you just need to know what is in the frame, not where — a classification model will be faster, smaller, and easier to deploy than a detection model.

The decision framework: specify your latency requirement first (how many ms per frame), then your accuracy floor (what false-positive and false-negative rates are operationally acceptable), then select the smallest model architecture that meets both constraints on your target hardware.

Quantisation and Format Conversion

Models trained in PyTorch or TensorFlow need to be converted and typically quantised before deployment to most edge hardware. Quantisation — converting 32-bit floating point weights to 8-bit integers — reduces model size by roughly 4x and inference time significantly, with usually acceptable accuracy loss for computer vision tasks.

The standard export pipeline:

  1. Train in PyTorch (FP32)
  2. Export to ONNX (portable interchange format)
  3. Convert to target runtime format (TFLite, HEF for Hailo, TensorRT for Jetson, OpenVINO for Intel)
  4. Validate accuracy on representative test data post-quantisation
  5. Deploy to edge device
# Export YOLOv8 model to ONNX
from ultralytics import YOLO
model = YOLO("yolov8n.pt")
model.export(format="onnx", imgsz=640, dynamic=False)

The quantisation step can surprise you. Some models lose significant accuracy when quantised, particularly if they were trained on narrow datasets or have unusual activation distributions. Always benchmark post-quantisation accuracy on data that represents your actual production environment, not just the standard benchmark datasets.

For Jetson devices (Orin Nano, Orin NX), TensorRT is the target runtime and the conversion adds further optimisations for the NVIDIA architecture. For Hailo NPUs, the hailomz compile toolchain handles ONNX-to-HEF conversion. For Intel Neural Compute Stick or OpenVINO-compatible hardware, the OpenVINO Model Optimizer is the tool.

Deployment Architecture Patterns

There are three common architecture patterns for industrial edge AI, and they suit different scenarios.

Standalone edge device — a single compute unit (Jetson, industrial PC with NPU) handles camera input, inference, and local logic. Output goes to a local dashboard, PLC signal, or local logging system. This is the right pattern for latency-critical applications (under 50ms response time) and environments with unreliable connectivity. The operational downside is that each device is managed independently.

Edge-to-cloud hybrid — the edge device runs inference and acts on results locally, but streams aggregated data (not raw video) to a cloud platform for monitoring, model versioning, and fleet management. Azure IoT Hub, AWS Greengrass, and Balena Cloud all support this pattern. Most production industrial deployments settle here — you get the latency and cost benefits of edge inference with the operational visibility of cloud management.

Fleet-managed edge — a Kubernetes distribution (K3s is the practical choice for industrial edge) orchestrates model deployment across many edge devices, handling rolling updates and health monitoring. The model is packaged as a container, versioned like software. When you retrain, you push a new container version through the fleet. This is more infrastructure overhead upfront but dramatically reduces operational cost at scale (50+ devices).

Monitoring Production Inference

Edge AI deployments fail silently in ways that don’t happen in cloud deployments. A cloud model returns an error when something goes wrong. An edge device might just slow down, start skipping frames, or produce degraded predictions if the camera angle shifts, lighting changes seasonally, or the device overheats in summer.

The minimum viable monitoring stack for production edge AI:

  • Inference latency: Track per-frame inference time. Spikes indicate thermal throttling or resource contention.
  • Confidence distribution: Monitor the distribution of model confidence scores. Sudden drops signal environmental changes (lighting, camera drift, new product variants not in training data).
  • Frame drop rate: If you’re not inferencing every frame, track the ratio. Rising drop rates indicate hardware saturation.
  • Hardware temperature: Industrial edge devices in non-air-conditioned environments regularly hit thermal throttle thresholds.

Prometheus with a Grafana dashboard is the standard stack. Even without a full Kubernetes setup, a lightweight MQTT-published metrics stream that feeds into your industrial monitoring infrastructure covers the essentials.

The gap between “model works in testing” and “model works reliably in production over 18 months in an industrial environment” is where most edge AI projects fail. Budget time for operational hardening — thermal management, model drift monitoring, camera maintenance schedules — alongside the initial development effort, and the projects that survive to deliver their promised ROI are the ones that plan for this from the start.