TinyML Overview
What is TinyML?
TinyML is the practice of training, compressing, and deploying machine learning models on microcontrollers (MCUs) — devices with as little as 2 KB of RAM and 16 KB of Flash, running on microwatt budgets. The goal is always-on, battery-powered intelligence at the sensor.
- Target: Cortex-M, RISC-V, AVR, Xtensa MCUs
- RAM budget: 2 KB – 512 KB
- Flash budget: 16 KB – 2 MB
- Power: typically < 1 mW during inference
- No operating system required
MCUs are the most abundant compute platform on Earth — billions ship per year in sensors, appliances, wearables, and industrial equipment. Bringing ML to MCUs unlocks intelligence at zero connectivity cost, long battery life, and privacy-preserving local inference.
- Cost: $0.50 – $5 per unit
- Battery: months to years on a coin cell
- Always-on keyword detect at <1 mW
- No cloud dependency or data egress
- Deterministic, hard real-time capable
Almost all TinyML work runs inference only on the MCU — training happens on a workstation or cloud with standard frameworks. A few research projects explore on-device fine-tuning (backprop on MCU) but RAM demands make full training impractical today.
- Inference: fully practical on MCU
- Fine-tuning: possible on larger MCUs (>256 KB RAM)
- Full training: not feasible on MCU
- Transfer learning done off-device
- Personalization via federated or split learning
Every TinyML design navigates a three-way tension. Shrinking the model to fit RAM reduces accuracy. Reducing quantization bit-width saves memory and speeds up integer ops but hurts accuracy. Understanding your accuracy floor and latency ceiling determines which compression techniques to apply.
- Profile target MCU RAM/Flash first
- Set minimum acceptable accuracy threshold
- Establish latency budget from application
- Iterate: quantize → prune → distill as needed
- Benchmark on real hardware, not simulator
The TinyML Constraint Triangle
| Constraint | Typical Budget (MCU) | Typical Budget (Mobile) | Primary Lever |
|---|---|---|---|
| Flash (model storage) | 16 KB – 1 MB | 10 – 200 MB | Quantization, pruning |
| RAM (activations) | 2 KB – 256 KB | 256 MB – 4 GB | Layer-by-layer scheduling, streaming |
| Power (inference) | 1 µW – 1 mW | 100 mW – 2 W | Clock gating, duty cycling, int ops |
| Latency | 1 ms – 500 ms | <100 ms | Model architecture, CMSIS-NN kernels |
| Accuracy | Moderate (task-specific) | High | Data quality, distillation |
TinyML vs Edge AI vs Cloud AI
| Dimension | TinyML (MCU) | Edge AI (SBC / NPU) | Cloud AI |
|---|---|---|---|
| Compute | MHz CPUs, no OS | GHz CPUs + NPU/GPU | Datacenter GPUs/TPUs |
| Power | µW – mW | 0.5 – 10 W | Hundreds of watts |
| Battery life | Months – years | Hours – days | N/A (plugged) |
| Model size | KB range | MB – hundreds MB | GB – TB (LLMs) |
| Connectivity required | None | Optional | Always |
| Latency | ms (local) | ms (local) | 50 – 500 ms (network) |
| Unit cost | $1 – $10 | $10 – $200 | Pay-per-inference |
TinyML Design Principles
Size the model to the RAM constraint before optimizing for accuracy. Activation memory (intermediate tensors) often exceeds weight memory. Use streaming/tiling to process inputs in chunks.
- Profile activation RAM, not just weights
- Depthwise-separable convolutions save RAM
- Single-buffer scheduling reuses memory
- Avoid residual connections that double buffer
Most MCUs lack an FPU or have a very slow one. INT8 inference runs 4–8× faster than FP32 and consumes significantly less power. Quantization-aware training preserves accuracy while enabling integer-only kernels.
- INT8 is the TinyML workhorse
- INT4 / binary for extreme constraints
- CMSIS-NN provides optimised ARM kernels
- No FP required at inference time
Run a tiny always-on detector (keyword spotter, motion trigger) that wakes a more capable stage only when needed. Two-stage inference lets the system sleep >99% of the time and dramatically extends battery life.
- Stage 1: µW always-on detector
- Stage 2: full inference on trigger
- Sensor hardware gating (PDM mic, accelerometer)
- Sleep modes between inference windows
Co-design the model architecture with the target hardware. Use Neural Architecture Search (NAS) constrained by MCU FLOP and memory budgets. Train with quantization from the start rather than applying it post-hoc.
- MCUNet / MicroNAS for automated search
- Quantization-aware training (QAT)
- Target FLOP count from day one
- Validate on-device, not just simulator