Running LLMs at the Edge: llama.cpp, Ollama, and Local AI Inference in 2026

TL;DR:

• Modern quantized LLMs (1.5B–8B parameters) run on edge hardware with 4–16GB RAM at practical inference speeds — from NVIDIA Jetson Orin to Raspberry Pi 5
• llama.cpp and Ollama are the two primary inference engines for edge LLM deployment, with llama.cpp offering more control and Ollama providing simpler server-mode operation
• The use cases unlocking value at the edge — industrial NLP, local voice processing, anomaly description, document analysis — don’t require frontier model capability, making smaller models a genuine fit

Language model inference has been a cloud workload since the first public LLM APIs appeared in 2020. That assumption is now breaking down at the edges — literally. The combination of increasingly capable small models, hardware-accelerated inference (GGML/CUDA/Metal), and aggressive quantization has made it practical to deploy LLMs directly on edge devices in 2026.

This matters for a class of applications where sending data to a cloud API is problematic: manufacturing environments with proprietary process data, healthcare deployments with patient information, defence and public safety applications with restricted networks, and latency-sensitive applications where cloud round-trips are too slow.

The Edge Hardware Landscape

The hardware available for edge LLM inference spans several categories:

NVIDIA Jetson Orin NX / AGX Orin: 8–64GB unified memory with GPU/CPU/deep learning accelerators in a compact module. Capable of running 7B models at 15–25 tokens/second with INT8 quantization. Purpose-built for on-device AI inference and the most capable edge option for medium to large models.

Raspberry Pi 5 (8GB): Runs small quantized models (1.5B–3B parameters) at 5–8 tokens/second via CPU. Sufficient for lightweight text analysis, local chatbot interfaces, and command parsing. Cost-effective at under £100.

Apple Silicon embedded (M-series Mac mini / MacBook as edge server): Excellent unified memory bandwidth makes the M4 Mac mini a capable edge server for running 7B–13B models. Common in retail and office edge deployments where a small form-factor server makes sense.

ARM Cortex-A based single-board computers (SBCs): Orange Pi 5 (RK3588, 16GB), Rock 5B — capable of running 1.5B–3B models with reasonable performance at lower cost than Jetson.

x86 mini PCs: Intel NUC / Beelink / Minisforum with 16–32GB RAM can run 7B models on CPU. Slow (4–8 tok/sec) but familiar to deploy and maintain.

llama.cpp: The Foundation Layer

llama.cpp is the C/C++ inference library that made edge LLM deployment practical. It supports:

• GGUF format (quantized model weights in 2-bit through 8-bit precision)
• CPU inference on any platform (x86, ARM, RISC-V)
• GPU acceleration via CUDA (NVIDIA), Metal (Apple), ROCm (AMD), and Vulkan
• A REST API server mode compatible with OpenAI’s API format

Building for Jetson (CUDA)

# On NVIDIA Jetson Orin
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Download a model (7B quantized to Q4_K_M — ~4.1GB)
wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

# Run the server (OpenAI-compatible API on port 8080)
./build/bin/llama-server \
  -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -n 2048 \
  --gpu-layers 99  # Offload all layers to GPU

Building for Raspberry Pi 5 (CPU + ARM NEON)

# On Raspberry Pi 5 (ARM64)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j4

# Use a small, fast model for Pi
wget https://huggingface.co/bartowski/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf

./build/bin/llama-server \
  -m Qwen2.5-1.5B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -n 1024 \
  --threads 4

Ollama: Simpler Deployment with Server Management

Ollama wraps llama.cpp with automatic model management, a simpler CLI, and background service management — useful when you want model management without the build-from-source workflow.

# Install on Linux edge devices
curl -fsSL https://ollama.com/install.sh | sh

# Ollama supports ARM64 — works on Jetson and Pi
# Pull and run a model
ollama pull llama3.2:3b          # 3B model, ~2GB download
ollama pull qwen2.5:1.5b         # 1.5B, fast and capable
ollama pull gemma3:4b            # Google's Gemma 3 4B

# Start API server (background service)
ollama serve   # Runs on localhost:11434 by default

# Make it accessible on the local network
OLLAMA_HOST=0.0.0.0:11434 ollama serve

For a systemd service (recommended for production edge deployments):

# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Server
After=network.target

[Service]
ExecStart=/usr/local/bin/ollama serve
Environment=OLLAMA_HOST=0.0.0.0:11434
Environment=OLLAMA_KEEP_ALIVE=24h
Restart=always
User=ollama

[Install]
WantedBy=multi-user.target

Model Selection for Edge Deployment

Quantization guide:

• Q8_0: Near full precision, best quality, uses ~1 byte per parameter
• Q4_K_M: Good quality/size balance, ~0.5 bytes per parameter (recommended starting point)
• Q3_K_S or IQ2_XXS: Aggressive quantization for memory-constrained devices, quality suffers

Edge LLM Application Patterns

1. Industrial Anomaly Description

import httpx
import json

# llama.cpp server running locally on edge device
BASE_URL = "http://localhost:8080"

def describe_anomaly(sensor_data: dict) -> str:
    """Generate a human-readable anomaly description from sensor readings."""
    prompt = f"""You are an industrial equipment monitoring system.
Analyze these sensor readings and describe the anomaly in plain English for a maintenance technician.

Sensor readings:
{json.dumps(sensor_data, indent=2)}

Normal operating ranges:
- temperature: 40-60°C
- vibration: 0.5-2.0 mm/s
- pressure: 3.5-4.5 bar

Provide a brief, specific description of what is abnormal and potential causes."""

    response = httpx.post(
        f"{BASE_URL}/v1/chat/completions",
        json={
            "model": "local",
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 150,
            "temperature": 0.1,
        },
        timeout=30,
    )
    return response.json()["choices"][0]["message"]["content"]

# Example usage
anomaly_readings = {
    "temperature": 78.3,
    "vibration": 4.2,
    "pressure": 3.8,
    "timestamp": "2026-06-06T09:15:00Z"
}

description = describe_anomaly(anomaly_readings)
print(description)
# "Temperature is 78.3°C — 18°C above normal operating range. 
#  Vibration at 4.2 mm/s is double the normal maximum, suggesting...

2. Local Document Classification

def classify_document(text: str, categories: list[str]) -> str:
    """Classify a document into predefined categories."""
    category_list = "\n".join(f"- {c}" for c in categories)
    
    response = httpx.post(
        f"{BASE_URL}/v1/chat/completions",
        json={
            "model": "local",
            "messages": [{
                "role": "user",
                "content": f"Classify this document into exactly one category.\n\nCategories:\n{category_list}\n\nDocument:\n{text[:2000]}\n\nRespond with only the category name, nothing else."
            }],
            "max_tokens": 20,
            "temperature": 0,
        }
    )
    return response.json()["choices"][0]["message"]["content"].strip()

Resource Management on Constrained Devices

Edge devices have limited memory and compute. Key considerations:

Context window: Reduce --ctx-size in llama.cpp from the default (2048-4096) to what your use case actually needs. Smaller context = less memory and faster inference.

Continuous batching: llama.cpp server supports batch processing of concurrent requests, which is more efficient than sequential single requests for high-throughput applications.

Model caching: Once a model is loaded into RAM/VRAM, inference is fast. Keep the model loaded between requests (Ollama’s OLLAMA_KEEP_ALIVE setting) rather than loading/unloading per request.

Graceful fallback: For edge deployments where connectivity is intermittent, design your application to use the local model when available and fall back to cloud when the local device is overloaded or unavailable.

The edge LLM deployment space is evolving rapidly. The models available in Q4_K_M quantization today that fit on a Jetson Orin NX are meaningfully more capable than those from 18 months ago, and the gap between edge-deployable and frontier models continues to narrow for many practical tasks.