What a Model Does
By the end of this module
- Describe what a language model returns when you give it text
- Explain why the same prompt can produce two different answers
- Recognise the mechanism behind a confidently invented fact
Definition
A language model takes text and returns a probability for every token that could come next. That is the whole operation. It does not look anything up, it does not consult a record of what is true, and it does not decide what to say before saying it. It scores candidates, something picks one, and the picked token is added to the text. Then it runs again.
Everything a model appears to do sits on top of that loop. Answering a question, writing code, refusing a request: all of it is the same operation repeated until a stopping condition is met.
Tokens
Models do not work in words. They work in tokens, which are fragments of text: common words are usually one token, rarer words split into several, and punctuation and spaces are tokens of their own. The word "unbelievable" might arrive as three pieces. This is why a model can miscount the letters in a word it can spell perfectly: it never saw the letters.
Sampling
Below is one prompt and the distribution over what comes next. Run it on Always the most likely and the sentence is the same every time. Switch to Sample the distribution and run it several times.
Probability of the next token, out of 5 candidates
Notice what the model did at the first step. It had no access to any refund policy, so it put most of its probability on "30", because across everything it was trained on, "30 days" follows that phrase more often than anything else. It is not reporting a policy. It is reporting the shape of a sentence.
That is the mechanism behind a confidently wrong answer. There is no separate step where the model checks whether a claim is true, so there is no step that can fail and warn you. A fabricated number and a correct one are produced by exactly the same process, which is why they read with the same confidence.
Consequences
Three properties follow directly from the mechanism, and the rest of this course keeps returning to them.
| Property | Why it follows |
|---|---|
| No separate truth step | Correct and fabricated output are produced by one process, so fluency is not evidence of accuracy and the model cannot flag the difference. |
| The prompt is the only ground | Anything the answer must respect has to be in the text the model is given, or it will be supplied from the general case instead. |
| Variation by default | Picking from a distribution means the same input can answer differently on each call, which is the subject of module three. |
A model answers a question about your company's refund window and gets it wrong, with no hedging. What happened?
Discussion
Before the next module, settle one question in the room: of the things your team currently asks a model to do, which ones depend on it knowing a fact nobody put in the prompt?