Temperature, Top-p & Sampling

Here is the one idea that makes every knob in this module make sense.

A language model does not choose a word. It publishes odds.

Give it The capital of France is and it doesn't reach for "Paris". It scores every token in its vocabulary — somewhere between 50,000 and 200,000 of them, depending on the model — and hands back the whole ranked list with a probability attached to each one:

• Paris — 92%
• the — 3%
• located — 2%
• now — 0.4%
• ...and a very long tail of near-zero no-hopers

That list is the model's entire output. Every time. It is a forecast, not a decision.

The jar of raffle tickets

Picture a jar. Every candidate token gets raffle tickets in proportion to its probability — Paris gets 92,000 tickets, now gets 400, and thousands of hopeless tokens get one or two each.

To produce a word, something reaches in and draws a ticket.

That draw is sampling. It happens after the model has finished thinking, it is a completely separate step, and it is the only place randomness lives. Same input, same odds — the forward pass is deterministic. The draw is not.

Which gives you the sentence to carry through this topic:

Temperature and top-p never change what the model believes. They change how the jar is stocked before you draw.

Every setting below is jar management. None of it is model management.