Each layer contains causal self-attention followed by a feed-forward network, wrapped with layer normalisation and residual connections. There is no encoder and no cross-attention. The same architecture handles both understanding and generation by treating every task as text completion. Modern variants use pre-norm (normalise before each sub-layer) and SwiGLU activations.

The decoder-only design is remarkably versatile. By framing every task as "continue this text", a single architecture handles translation, summarisation, question answering, code generation, and reasoning without task-specific modifications.

A triangular mask is applied to the attention scores before softmax, setting all positions to the right of the current token to negative infinity. After softmax, these positions receive zero weight. At position t, the model sees tokens 0 through t and nothing beyond. This mask is applied identically during both training and inference.

Causal masking makes autoregressive generation possible. The model learns to predict the next token using only preceding context, which matches the generation setting exactly. Without this constraint, the model would "cheat" during training by looking at future tokens.

GPT-1 and GPT-2 use learned position embeddings (a separate embedding table indexed by position). Each token's input is the sum of its token embedding and position embedding. GPT-3 uses the same approach. Context length is fixed at training time (512 for GPT-1, 1024 for GPT-2, 2048 for GPT-3, 8K/32K/128K for GPT-4).

Learned position embeddings are simple and effective but limit the model to the context length seen during training. This motivated later innovations like ALiBi and RoPE in non-GPT models, though GPT-4 substantially extended context length through engineering.

The last transformer layer's output is projected through a linear layer with dimensions (hidden_size x vocabulary_size). Softmax converts the logits into a probability distribution over the entire vocabulary. During training, cross-entropy loss compares this distribution to the actual next token. Weight tying shares the embedding matrix with this output projection, reducing parameters.

The simplicity of next-token prediction as the sole training objective is what makes GPT's approach powerful. A single objective, applied to massive text corpora, produces models that learn grammar, world knowledge, reasoning patterns, and even code without any task-specific engineering.

Collect a dataset of prompts paired with ideal responses, often written by human annotators. Fine-tune the pre-trained model on this data using the same next-token prediction objective, but only on the response tokens (the prompt tokens contribute to context but not to the loss). Typical datasets range from tens of thousands to hundreds of thousands of examples.

SFT bridges the gap between a base model (which completes text) and an assistant (which follows instructions). A base model given "What is photosynthesis?" might continue with another question. After SFT, it provides a clear, helpful explanation.

First, train a reward model on human preference data (annotators compare two model outputs and pick the better one). Then use the reward model's scores as a signal to optimise the language model with Proximal Policy Optimisation (PPO). A KL penalty prevents the model from drifting too far from the SFT baseline. The reward model learns nuanced preferences that are hard to specify as rules.

RLHF teaches models to be helpful, harmless, and honest in ways that SFT alone cannot capture. It handles subjective preferences (tone, detail level, safety) where there is no single "correct" answer. InstructGPT showed that RLHF with a small model could outperform a 100x larger base model.

Use the same human preference data (pairs of outputs with a preferred choice) but optimise the language model directly. DPO reformulates the RLHF objective into a classification loss on preference pairs, proving mathematically that the optimal policy can be extracted without training a reward model. The model learns to increase the probability of preferred responses relative to dispreferred ones.

DPO significantly simplifies the alignment pipeline. No reward model training, no PPO instability, no KL tuning. It produces comparable results to RLHF with less compute, fewer hyperparameters, and more stable training. It has become the preferred approach for many open-source model developers.