Most enterprise AI conversations still assume a large model behind an API in someone’s cloud. For many workloads that is the right answer. For a growing set it is not.
Small language models, from a few hundred million to a few billion parameters, can now run on laptops, phones, industrial gateways and modest edge servers. They are not as broadly capable as frontier models. They do not need to be. For well-defined tasks they can be fast, private and cheap to run.
Why run a model at the edge
The reasons are usually practical rather than fashionable.
- Latency — a model on the device answers without a network round trip
- Connectivity — factories, vehicles, field sites and clinics do not always have a reliable link
- Data residency and privacy — sensitive data never leaves the device or site
- Cost at volume — high-frequency, low-complexity calls add up quickly on a metered API
- Resilience — the feature keeps working when the cloud service or network does not
If none of these apply, a hosted model is probably simpler. If one or more is a hard constraint, the edge deserves a serious look.
What small models do well
Small models are strongest on narrow, repeatable tasks with clear inputs and outputs:
- Classifying and routing text, tickets or events
- Extracting structured fields from forms, logs or short documents
- Summarising short content in a fixed format
- Intent detection for voice or command interfaces
- Drafting short responses from templates and local context
They are weaker on open-ended reasoning, long documents and tasks needing broad world knowledge. Plan the use case around that, not against it.
Fine-tuning and distillation
A general small model is a starting point. Performance on your task usually improves with:
- Fine-tuning on curated examples from the real workflow
- Distillation, using a larger model to generate or label training data
- Constrained output, such as JSON schemas or grammars, so results are easy to validate
The data work matters more than model choice. A clean, representative set of examples and a solid evaluation set will do more than switching between similar models.
Hardware and runtime choices
Edge deployment is an engineering problem as much as a modelling one. Decisions include:
- Quantisation — lower-precision weights reduce memory and speed up inference, with some quality cost you must measure
- Runtime — options range from mobile and browser runtimes to optimised inference servers on edge GPUs or NPUs
- Memory and power budgets — especially on battery or fanless hardware
- Thermal behaviour under sustained load, not just in a quick benchmark
Benchmark on the actual target hardware with realistic inputs. Desktop numbers rarely transfer cleanly.
Hybrid is often the answer
Edge and cloud are not either-or. A common pattern is:
- A small local model handles the frequent, simple cases
- Uncertain or complex cases escalate to a larger hosted model, where policy allows
- Results and feedback sync back when connectivity returns
This keeps most traffic local while still covering the long tail. It does require clear routing rules and confidence thresholds, which belong in evaluation.
Operating models you cannot see
The hardest part of edge AI is often operations. Models spread across hundreds or thousands of devices still need:
- Versioned rollout with staged deployment and rollback
- Telemetry on latency, errors and confidence, sent within privacy limits
- Evaluation before every model update, on the same test set
- Security for model files and the device itself
Treat a model update like a firmware update. It needs the same care.
Where to start
- Pick one task where latency, connectivity, privacy or cost is a hard constraint
- Build an evaluation set from real examples before choosing a model
- Benchmark two or three small models, quantised, on the target hardware
- Design the escalation path to a larger model, if policy allows one
- Plan rollout, telemetry and rollback before the first device ships
Small models will not replace large ones. They will take over a lot of work large ones were never the best fit for.