How a model writes: one token at a time

A language model never writes a whole answer at once. It predicts the next token, adds it to the text, and repeats.

The idea

A language model is a very good next-token predictor. Given the text so far, it gives every possible next token a probability. One token is chosen, added to the text, and the longer text goes back in for the next prediction. A whole answer is just this loop running many times, and it stops when the model picks a special end token or hits a length limit.

Code

public interface ILanguageModel
{
    Dictionary<string, double> NextTokenProbabilities(IReadOnlyList<string> context);
}

public static string Generate(ILanguageModel model, List<string> context, int maxTokens)
{
    for (int i = 0; i < maxTokens; i++)
    {
        var probs = model.NextTokenProbabilities(context);
        string next = probs.MaxBy(p => p.Value).Key;
        if (next == "<end>") break;
        context.Add(next);
    }
    return string.Join(" ", context);
}

Tips

• Interview line: "An LLM generates text autoregressively: each new token is predicted from all the tokens before it, including the ones it just wrote."
• The model does not look up a stored answer. It computes probabilities fresh at every step, which is why the same question can be answered in different words.
• This demo always takes the most likely token (greedy). Real systems usually sample from the probabilities, which the temperature lesson shows.
• Longer answers take more time because the loop runs once per output token.

In the .NET library

In .NET you would normally stream these tokens to the user as they arrive (for example by reading the HTTP response as it comes in) so the first words appear immediately instead of waiting for the whole loop to finish.