Controls randomness in token selection by scaling logits before softmax. Lower values (0.1-0.3) produce deterministic, focused outputs ideal for factual tasks; higher values (0.7-1.0) increase creativity and diversity for brainstorming. Temperature 0 always picks the highest probability token, while temperature 1+ can produce incoherent or repetitive text.

Samples from the smallest set of tokens whose cumulative probability exceeds threshold p. At p=0.9, only tokens comprising 90% of probability mass are considered. More adaptive than Top-K: when the model is confident, fewer tokens are sampled; when uncertain, more variety is allowed. Commonly combined with temperature for fine-grained control.

Splits individual layers (weight matrices) across multiple GPUs horizontally. Each GPU computes a portion of each layer, requiring all-reduce communication between layers. Reduces per-GPU memory linearly but adds latency from synchronization. Best within a single node with fast NVLink; cross-node adds significant overhead.

Distributes model layers across GPUs vertically. Each GPU handles a subset of layers, passing activations forward. Lower communication than tensor parallelism (only at stage boundaries), but creates pipeline bubbles where GPUs idle waiting for dependencies. Micro-batching helps fill bubbles. Works well across nodes.

Replicates the full model on each GPU and distributes different batches across devices. Gradients are synchronized via all-reduce after each step. Scales batch size linearly with GPU count. Simple to implement but requires each GPU to hold the entire model. For inference, enables higher throughput with independent replicas.

Distributes long input sequences across devices, each GPU processing a portion. Enables context lengths beyond single-GPU memory (e.g., 1M+ tokens). Ring attention passes KV slices between GPUs in a ring topology. Particularly important for long-document understanding and retrieval-augmented generation with large contexts.

Uses a small draft model (e.g., 1B params) to generate K candidate tokens quickly, then verifies all K in a single forward pass of the large model. Accepted tokens are kept; rejected ones trigger resampling. Achieves 2-3x speedup without changing output distribution. Most effective when draft model closely matches target model's distribution.

Dynamically adds new requests to running batches as slots become available, rather than waiting for entire batches to complete. A request finishing after 50 tokens immediately frees its slot for a new request, even while others generate 500+ tokens. Increases throughput 2-10x over static batching and reduces average latency significantly.

Splits long prompt processing into chunks (e.g., 512 tokens), interleaving prefill with ongoing decode operations. Without chunking, a 10K token prompt blocks all other requests for seconds. Chunking allows decode iterations to proceed between prefill chunks, reducing P99 latency dramatically for mixed workloads with varying prompt lengths.

C++/Python library for efficient Transformer inference with layer fusion, batch reordering, and KV caching. Supports INT8/INT4 quantization, runs on CPU (x86, ARM) and NVIDIA GPUs. Often 2-4x faster than HuggingFace Transformers with lower memory. Supports Llama, Mistral, Whisper, T5, BERT and more.

Microsoft's optimization library for training and inference at scale. ZeRO optimizer eliminates memory redundancy, enabling trillion-parameter models. DeepSpeed-Inference provides optimized kernels, tensor parallelism, and dynamic batching. Used to train MT-NLG 530B, BLOOM 176B. Integrates with HuggingFace, PyTorch Lightning, and Accelerate.