The default assumption when you’re building something with an LLM is that the model lives in the cloud. You send a request to an API, it comes back with a completion, you bill it by the token. That model made sense when edge hardware couldn’t run anything interesting and when the main LLMs were tens of billions of parameters. Both of those things are changing fast.

In 2026, you can run a genuinely capable 7B or 13B parameter model on a machine that costs less than £500, draws 15 watts of power, and sits in a retail back office, a factory floor, or a telecom edge node. Whether you should do that depends on your latency requirements, your data sensitivity, and whether cloud inference costs are becoming a real line item on your operations budget.

Why run at the edge at all?

Three reasons come up consistently in real deployments.

Latency. Round-trip to a cloud inference endpoint adds 200–500 milliseconds even under good network conditions. For applications where an AI response is part of an interactive loop — a voice assistant, a real-time quality check on a production line, a customer-facing kiosk — that latency is noticeable. Local inference on edge hardware running a quantised model can respond in under 100 milliseconds for modest-length outputs.

Data sensitivity. Manufacturing firms running AI-assisted inspection don’t always want their machine vision frames, defect images, or production telemetry leaving the factory. Healthcare settings with patient interaction can’t route PII through third-party cloud APIs without careful legal review. Edge inference keeps data local and makes the compliance story simpler.

Cost at scale. Cloud inference pricing works fine at low volume. At millions of queries a month across hundreds of edge sites, the maths changes. Akamai’s published analysis of their edge small language model deployments showed a 76 per cent reduction in cost-per-token compared to equivalent cloud inference — the fixed hardware cost amortises quickly at that query volume.

The hardware picture

The Raspberry Pi is not what you should be looking at here. A Pi 4 can run a 7B model in 4-bit quantisation, but it’s slow — expect 2–4 tokens per second on CPU. Fine for experimentation, not for production.

NVIDIA Jetson remains the most capable option for edge AI inference. The Jetson Orin NX (16GB) can run a 7B quantised model at 15–25 tokens per second, which is usable for most applications. The Orin AGX at 64GB can handle 13B models comfortably. Jetson also has the advantage of strong framework support: TensorRT, CUDA, and compatibility with the full NVIDIA inference stack.

Apple Silicon Mac Mini is an underrated edge inference option. The M4 Mac Mini starts at £699, has unified memory that makes large model loading efficient, and supports Metal Performance Shaders for inference acceleration. It’s quiet, reliable, and runs standard tools without fuss. Not suited for environments that need fanless or industrial temperature ratings, but for offices, retail, and similar environments it’s practical.

Qualcomm AI-oriented SoCs — including the Snapdragon X series and dedicated edge AI platforms — are showing up in thin client and edge devices. Qualcomm’s Hexagon NPU is specifically designed for transformer inference, and OEMs are building edge AI appliances around it.

The software stack

Ollama is the easiest path to getting a model running locally. It handles model download, quantisation format support, and exposes an OpenAI-compatible API. You point your application at localhost:11434 instead of api.openai.com and most code changes are minimal.

# Install Ollama and pull a model
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b-instruct-q4_K_M

# Test
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b-instruct-q4_K_M",
  "prompt": "Summarise the following defect report:",
  "stream": false
}'

llama.cpp gives more control over quantisation and hardware acceleration. It runs on CPU, CUDA, Metal, and Vulkan, and it’s what Ollama uses under the hood. If you’re building a production system and need to optimise performance for specific hardware, llama.cpp gives you more knobs to turn.

vLLM is the choice when you’re running inference at scale on more powerful edge nodes and need production-grade batching, continuous batching, and PagedAttention. It requires a GPU but is significantly more efficient than naive inference when serving multiple concurrent requests.

Which models to run

The 4-bit quantised variants of Llama 3.1 8B, Mistral 7B, and Phi-3 Mini are the practical working options for constrained edge hardware. They’re capable enough for summarisation, classification, structured data extraction, and question-answering over retrieved context. They’re not going to match GPT-4o on complex reasoning tasks, but for bounded, well-defined edge applications they don’t need to.

For specific domains, fine-tuned variants often outperform general models at a fraction of the size. A 3B model fine-tuned on manufacturing defect descriptions will beat a general 7B model on that task while running twice as fast. If you’re deploying at scale to a specific use case, fine-tuning for that use case is worth the investment.

When not to run at the edge

Complex, multi-step reasoning tasks still benefit from the largest models — GPT-4o, Claude Opus — which aren’t going to run on edge hardware any time soon. If your edge application has a reasoning step that’s genuinely hard, consider a hybrid: handle most queries at the edge, route hard cases to cloud inference.

Also consider the operational overhead. A fleet of edge nodes each running an LLM needs model update management, monitoring for inference failures, and hardware that’s robust enough for the environment. If you’re running ten edge sites, this is manageable. If you’re running thousands, your edge AI deployment needs the same operational rigour as any other fleet management problem — K3s, centralised logging, and a proper update pipeline.

Edge inference is a real option in 2026 in a way it wasn’t in 2023. Whether it’s the right option for your use case depends on latency, data sovereignty, and whether the economics work at your query volume. For many industrial, retail, and healthcare edge deployments, they do.