From scores to a chosen token, and temperature: Sampling and decoding

Why a model can answer the same question differently each time, and the three words that explain it.

The problem: the same question, different answers

Ask a chat model the same question twice and you may get two different answers. Is it moody? No. At every step the model does not choose one token. It produces a score for every token it knows, and a separate step turns the scores into a choice. That second step is called decoding, and you can control it with a few settings.

Logit

The raw, unbounded score the model gives each possible next token. Higher means more likely. Logits are not probabilities and do not add up to 1.

Softmax

The function that turns logits into probabilities that are positive and add up to 1: raise e (about 2.718) to each score and divide by the total. A gap of one point in logits multiplies the odds by about 2.7.

Decoding

The whole procedure of generating text one token at a time: compute the logits, choose a token, append it, repeat until a stop token or a length limit.

The example code (code/mini-sampling) uses five tokens that might follow "The capital of France is". The scores are invented; no model produced them.

Token	Paris	Lyon	France	a	banana
Logit	5.0	3.0	2.5	1.0	-1.0
Probability (T = 1)	0.808	0.109	0.066	0.015	0.002

The function:

def softmax(logits, temperature=1.0):
    z = np.asarray(logits, dtype=float) / temperature
    z = z - z.max()          # numerical safety; does not change the result
    e = np.exp(z)
    return e / e.sum()

The line z - z.max() only keeps the numbers small so that exp does not overflow; it does not change the answer. The division by temperature is the setting explained next.

Choosing: greedy or sampled

Greedy decoding

Always take the highest-probability token. It is fast and repeatable, but one locally best choice can lead to dull or repetitive text, and it can get stuck in loops.

Instead of always taking the top token, you can sample: pick a token at random, with chances equal to the probabilities. With the table above, "Paris" is chosen about 81% of the time, but not always. Here are 1,000 real draws with a fixed random seed:

Paris: 795   Lyon: 125   France: 64   a: 15   banana: 1

So "banana" was chosen once in a thousand times. That is why the same question can give different answers, and why a small chance of something odd is added on every step. Greedy is what "temperature 0" means in many APIs: the limit as temperature approaches zero. (Check your provider's documentation for how it treats 0.)

Check yourself

What is the difference between a logit and a probability?
With these probabilities, how often would sampling choose "Lyon" in 1,000 draws, roughly?

Temperature

A number that divides the logits before softmax. Below 1 it sharpens the distribution, so the leader gets more of the probability. Above 1 it flattens it, giving more variety and more risk. It never changes which token is ranked first.

Try it by hand with three tokens and logits [2, 1, 0]. At temperature 1 the probabilities are 0.665, 0.245 and 0.090. At temperature 0.5 the logits become [4, 2, 0] and the probabilities 0.867, 0.117 and 0.016. At temperature 2 the logits become [1, 0.5, 0] and the probabilities flatten to 0.506, 0.307 and 0.186. The order never changed.

Same five scores at four temperatures. Real output from the example code.

The same effect on the Paris example, with a measure of how spread out the distribution is, called entropy (in bits; 0 means one certain choice, higher means more uncertainty):

Temperature	P(Paris)	Entropy (bits)
0.2	1.000	0.00
0.5	0.975	0.19
1.0	0.808	0.97
2.0	0.544	1.78
5.0	0.330	2.21

At temperature 5 "Paris" has only a third of the probability, and even "banana" gets about 10%. That is what "too hot" looks like.

Check yourself

Does a low temperature change which token is ranked first?
What happens to the entropy as temperature rises?