Model Compression
Quantization
Convert a trained FP32 model to INT8 without retraining. Calibrate scale factors using a representative dataset (100–1000 samples). Fast, but accuracy can drop 1–5% on sensitive tasks.
- No retraining required
- Calibration dataset: 100–1000 samples
- Weight + activation quantization
- Dynamic (weights only) or static (both)
- TFLite converter, ONNX quantize tool
Insert fake-quantization nodes during training so the model learns to tolerate integer rounding. Typically recovers 1–3% accuracy versus PTQ. Required when PTQ accuracy is unacceptable.
- Fake-quant ops simulate INT8 rounding
- Straight-through estimator for gradients
- TF: tf.quantization.quantize_and_dequantize
- PyTorch: torch.ao.quantization FX API
- Best accuracy at given bit-width
Quantize only the weight tensors, keeping activations in FP. Reduces model size for storage and transfer. Useful when activation ranges are highly dynamic (e.g., audio processing).
- INT4/INT8 weights, FP32 activations
- No calibration dataset needed
- Dequantize weights at runtime
- Less speedup than full INT8
- Good for memory-limited Flash, not RAM
Extreme quantization: weights and activations restricted to {-1, +1} (binary) or {-1, 0, +1} (ternary). Replace MACs with XNOR + popcount. 32× smaller, 58× faster on capable hardware, but significant accuracy cost.
- XNOR-Net, BinaryNet architectures
- popcount replaces multiply-accumulate
- Typical accuracy gap: 5–15% vs FP32
- Effective for keyword spotting tasks
- Custom CMSIS kernels or Larq library
Quantization Schemes Compared
| Scheme | Bits | Size vs FP32 | Accuracy Drop | MCU Support | Best For |
|---|---|---|---|---|---|
| FP32 | 32 | 1× (baseline) | None | Poor (no FPU on most MCUs) | Training, reference |
| FP16 | 16 | 2× smaller | Negligible | Limited (Cortex-M33 FPU) | Edge servers |
| INT8 | 8 | 4× smaller | 0.5–2% | Excellent (CMSIS-NN) | General TinyML |
| INT4 | 4 | 8× smaller | 2–5% | Good (custom kernels) | Very constrained MCUs |
| Binary (BNN) | 1 | 32× smaller | 5–15% | Possible (XNOR ops) | Ultra-tiny classifiers |
| Mixed-precision | 4–8 | 4–8× smaller | Minimal | Good | Sensitive layer protection |
Pruning
Zero out individual weights below a magnitude threshold. Creates a sparse weight tensor. Size reduction requires sparse storage formats; actual speedup requires sparse compute support (rare on MCUs).
- Magnitude-based: remove small weights
- Iterative pruning + fine-tuning
- Lottery ticket hypothesis
- CSR/CSC sparse formats for storage
- Limited MCU speedup without sparse HW
Remove entire filters, channels, or attention heads. Produces a smaller dense model — no sparse compute required. Can directly reduce RAM and latency on any MCU without special hardware support.
- L1-norm filter ranking
- Activation-based importance scores
- Remove whole output channels
- Dense model — standard runtime
- 20–50% channel removal with ~2% accuracy loss
Knowledge Distillation
Train a small student model to mimic the soft output probabilities of a large teacher model, not just the hard labels. The teacher's probability distribution carries richer signal about class relationships.
- Soft targets: teacher's probability vector
- Temperature T softens distributions
- Loss = α·CE(student, hard) + β·KL(student, teacher)
- Student can be 10–100× smaller than teacher
- No teacher needed at inference time
Distill intermediate layer activations (feature maps, attention maps) in addition to logits. Helps the student learn internal representations. Important when student and teacher differ strongly in architecture.
- FitNets: match intermediate feature maps
- Attention Transfer: match attention maps
- RKD: relational knowledge distillation
- Requires projection layers when channels differ
- Stronger than logit-only for TinyML
Recommended Compression Pipeline
| Step | Technique | Tool | Expected Gain | Notes |
|---|---|---|---|---|
| 1 | Architecture choice | MobileNet, MCUNet, DS-CNN | 10–100× vs ResNet baseline | Start small, not compressed large |
| 2 | Quantization-aware training | TF QAT, PyTorch FX | 4× size, minimal accuracy loss | Fake-quant during training |
| 3 | Structured pruning | TF Model Optimization, torch.nn.utils.prune | 2–5× FLOP reduction | After QAT, before export |
| 4 | Knowledge distillation | Custom training loop | Recover 1–3% accuracy | Optional; helps after aggressive pruning |
| 5 | Convert to runtime format | TFLite converter, microTVM compile | Further kernel fusion | Validate accuracy post-conversion |
| 6 | On-device profiling | TFLite Micro benchmark, Edge Impulse profiler | Confirm RAM/Flash/latency | Always measure on real hardware |