Pretraining and the base model

During pretraining, the model sees a text and, at every position, is scored on how much probability it gave to the true next token. Repeat this across trillions of tokens and the model absorbs grammar, facts, code and style, because all of them help to predict what comes next.

You can feel the idea with a counting model. The example trains an n-gram language model: it only counts which word follows which short context. It is not a neural network, but "learn next-word probabilities from text, then pick from them" is the same loop.

base = NGramLM()
base.train(PRETRAIN)                       # count what follows what
base.generate("what is an agent ?", 18)    # greedy: pick the most likely next token each time

Greedy means always taking the most likely next token. Real output:

what is an agent ? many people ask what is an agent . the question is how an agent decides what to do

We asked a question and the model did not answer. It continued the text the way text of that kind usually continues. In this model's training text, "what is an agent ?" was followed by "many people ask what is an agent .", which is discussion about the question, not an answer.

A different question behaves differently:

what is a token ? a token is a piece of text . what is an agent ? many

Here the model did produce an answer, "a token is a piece of text .", and then rambled on. The reason is not understanding. The training text happened to contain that exact question followed by that answer. Two lessons: a base model's "answers" are accidents of what followed similar text, and it does not know when to stop.

Perplexity is the standard score of how surprised a language model is by a text. Low means "not surprised". On text from its own training set the example model scores about 1.1. Text it has never seen hits the floor of its scoring, so the huge number you get for reversed text (1,000,000) is that floor and not a meaningful measurement. A test asserts that familiar text scores better than shuffled text.