Inference Optimization
Model Optimization Techniques
Reduce numerical precision of weights and activations from FP32 to FP16, INT8, or INT4 for faster inference and smaller models.
- 2-8x model size reduction
- 2-4x inference speedup
- Lower memory requirements
- Minimal accuracy loss (INT8)
- Post-training or quantization-aware training
- LLM deployment (GPTQ, AWQ)
- Edge device inference
- Cost reduction in production
- Mobile ML applications
Remove unnecessary weights or entire neurons/layers from model to reduce size and computational requirements.
- Structured pruning (remove channels/layers)
- Unstructured pruning (remove individual weights)
- Magnitude-based or gradient-based
- Can achieve 50-90% sparsity
- Requires fine-tuning
- Mobile deployment
- Latency-critical applications
- Resource-constrained devices
- Model compression
Train smaller 'student' model to mimic larger 'teacher' model's behavior, preserving performance with lower complexity.
- Transfer knowledge from large to small model
- Soft labels from teacher model
- Often matches teacher performance
- Complementary to quantization/pruning
- Popular for BERT → DistilBERT, TinyBERT
- Deploy lighter models (DistilBERT)
- Edge/mobile inference
- Latency optimization
- Cost reduction
Combine multiple operations into single kernel to reduce memory access overhead and improve computational efficiency.
- Fuse elementwise ops (add, relu, etc.)
- Reduce memory bandwidth
- Lower kernel launch overhead
- Automatic in TensorRT, ONNX Runtime
- 10-30% speedup typical
- GPU inference optimization
- TensorRT deployments
- High-throughput scenarios
- ONNX Runtime inference
Compile models to optimized low-level code for specific hardware using compilers like TVM, XLA, or TensorRT.
- Hardware-specific optimization
- Automatic kernel selection
- Graph-level optimization
- Cross-platform (TVM)
- Significant speedups (2-10x)
- Production deployment
- Custom hardware targets
- Multi-backend support
- Performance-critical systems
Split large models across multiple GPUs/devices using tensor, pipeline, or sequence parallelism for scale.
- Tensor parallelism (split layers)
- Pipeline parallelism (split stages)
- Sequence parallelism (split sequences)
- Necessary for LLMs >100B parameters
- DeepSpeed, Megatron-LM, FSDP
- Large language models
- Multi-GPU inference
- Billion-parameter models
- High-throughput serving
Quantization Precision Comparison
| Precision | Bits | Size Reduction | Speed Improvement | Quality Loss | Use Case |
|---|---|---|---|---|---|
| FP32 (Full) | 32-bit | Baseline (1x) | Baseline (1x) | None | Training, highest accuracy needs |
| FP16 | 16-bit | 2x smaller | 1.5-2x faster | Minimal | Standard inference, training |
| BF16 | 16-bit | 2x smaller | 1.5-2x faster | Minimal | Training, wider range than FP16 |
| INT8 | 8-bit | 4x smaller | 2-4x faster | Low | Production inference, edge devices |
| INT4 | 4-bit | 8x smaller | 3-5x faster | Moderate | LLMs, memory-constrained devices |
| FP8 | 8-bit | 4x smaller | 2-4x faster | Very Low | H100 GPUs, modern accelerators |
| Mixed Precision | Variable | 2-4x smaller | 2-3x faster | Minimal | Balance speed and accuracy |
Quantization Methods
Post-Training Quantization (PTQ)
Approach: Quantize trained model without retraining
Calibration: Use small calibration dataset
Speed: Fast (minutes)
Quality: Good for INT8, moderate for INT4
Tools: TensorRT, ONNX Runtime, PyTorch
Quantization-Aware Training (QAT)
Approach: Simulate quantization during training
Calibration: Not needed
Speed: Slow (full retraining)
Quality: Best, especially for INT4
Tools: TensorFlow Lite, PyTorch, QAT frameworks
LLM Quantization Methods
GPTQ: 4-bit quantization for LLMs
AWQ: Activation-aware weight quantization
GGUF/GGML: CPU-optimized formats
bitsandbytes: 8-bit and 4-bit quantization
Tools: AutoGPTQ, llama.cpp, HF Transformers
Runtime Optimization Techniques
Batching Strategies
- Static Batching: Fixed batch size
- Dynamic Batching: Variable sizes, timeout-based
- Continuous Batching: LLM-optimized (vLLM)
- Sequence Packing: Pack multiple sequences
- Trade-off: Throughput vs latency
Memory Optimization
- KV Cache Management: PagedAttention (vLLM)
- Flash Attention: Memory-efficient attention
- Gradient Checkpointing: Trade compute for memory
- Model Offloading: CPU/disk swapping
- Memory Pooling: Reuse allocations
Kernel Optimization
- TensorRT: NVIDIA optimized kernels
- ONNX Runtime: Cross-platform optimization
- Custom CUDA Kernels: Hand-optimized ops
- Operator Fusion: Combine operations
- Graph Optimization: Computation graph simplification
Caching Strategies
- Response Caching: Cache common outputs
- Embedding Caching: Reuse embeddings
- Prompt Caching: Cache system prompts (50% savings)
- KV Cache: Reuse key-value pairs
- Semantic Caching: Similar queries → same answer
Hardware Acceleration Options
| Hardware | Best For | Advantages | Limitations |
|---|---|---|---|
| NVIDIA GPUs (CUDA) | General ML, training, inference | Mature ecosystem, flexible, widely supported | Expensive, power-hungry |
| AWS Inferentia/Trainium | High-throughput inference, training | 70% cost savings, purpose-built | AWS-only, framework limitations |
| Google TPUs | Large-scale training, TensorFlow | Fast matrix operations, cost-effective | GCP-only, TensorFlow-optimized |
| Intel Habana Gaudi | Training, cost-conscious workloads | Good price/performance, PyTorch support | Smaller ecosystem |
| CPUs (x86, ARM Graviton) | Latency-sensitive, small models | Ubiquitous, good for quantized models | Slower than GPUs for large models |
| Edge Devices (Jetson, Coral) | On-device inference, IoT | Low latency, privacy, offline | Limited compute, requires optimization |
Inference Optimization Workflow
1. Benchmark Baseline
Measure current latency, throughput, memory, cost
2. Apply Optimizations
Quantization, pruning, batching, kernel optimization
3. Validate Quality
Ensure accuracy remains acceptable for use case
4. Measure Gains
Compare metrics vs baseline, iterate if needed