Non-Determinism
By the end of this module
- List the sources of variation in a model-backed system
- Judge when varying output is a correctness problem
- Choose between constraining, verifying and measuring
Definition
Call a model twice with identical input and you can get two different answers. This follows from the mechanism in module one: the model returns a distribution and something draws from it.
Teams usually meet this as an annoyance in a demo, and meet it again later when something downstream depends on the answer being the same.
Sources of Variation
Sampling is the source everyone knows about and the easiest to turn down. The others have no temperature setting.
| Source | What changes | Under your control |
|---|---|---|
| Sampling | Which token is drawn at each step. | Yes, through temperature and truncation. |
| Model version | The distribution itself, when a provider updates a model behind an alias. | Partly. Pin a version and migrate deliberately. |
| Context assembly | What is retrieved, what fits, what order it arrives in. | Yes, and it is usually the largest effect. |
| Tool results | What the world said when the agent asked it. | No. The data is live by design. |
| Hardware and batching | Low-order bits in the arithmetic, which can flip a close decision. | No, on a hosted endpoint. |
Classification Variance
The panel below asks for a routing decision rather than prose. Generate several times at temperature 0, then move the slider up and generate several more.
Classify the ticket into exactly one of: Billing, Technical, Account. Reply with the label only.
Ticket: My payouts have been failing for 3 days and I can't see why in the dashboard.
Nothing generated yet.
At 0 the answer repeats. At 1.2 the same ticket reaches three different teams.
A varying label sends the ticket to a different team and writes a different record, so any report built on those records is wrong. A varying sentence costs the reader nothing. The same variation is harmless in one place and a correctness bug in another, so the response belongs to the decision rather than to the system.
Failure Modes
| Assumption | How it fails |
|---|---|
| Tests asserting an exact output | They pass when written and fail on a rerun, so they get marked flaky and stop protecting anything. |
| Caching on the input | A cached answer is one sample presented as the answer, which hides variation rather than removing it. |
| Reproducing a reported bug | The failing run cannot be replayed unless the trace was recorded at the time. |
| Anything parsing the output | A shape that held for a thousand calls fails on the next one, in production, far from the prompt. |
| Approval of one sample | Signing off an output approves one draw, not the distribution. |
Responses
- Constrain. Remove the freedom. Lower the temperature, restrict the output to a fixed set of options, or move the decision to a model that cannot answer off the list. The next module covers that class of model.
- Verify. Let it vary and check the result before acting on it, with a schema check, a rule, a second model or a person. The check has to be cheaper than the thing it guards.
- Accept and measure. Where variation is harmless, measure the distribution over a set of cases instead of testing for a fixed answer. Unit six covers how.
A nightly job summarises the day's tickets for a dashboard. Summaries are worded differently each night. Which response fits?
Discussion
For each use of a model in your team's work, say which of the three responses it currently gets, and whether anyone decided that.