Tokens and the context window

Models read tokens, not words, and can only look at a limited window of them at once. Watch old tokens fall out.

The idea

A model does not see letters or words. Text is first cut into tokens: common words, pieces of longer words, punctuation. Everything is measured in tokens: the size of your prompt, the length of the answer, and the context window, which is the most tokens the model can consider in one request. The demo uses a tiny window so you can watch it fill and the oldest tokens drop out.

Code

public sealed class ContextWindow
{
    private readonly Queue<string> _tokens = new();
    private readonly int _capacity;

    public ContextWindow(int capacity) => _capacity = capacity;
    public IEnumerable<string> Visible => _tokens;

    public void Add(string token)
    {
        _tokens.Enqueue(token);
        if (_tokens.Count > _capacity)
            _tokens.Dequeue();
    }
}

public static string WhatTheModelSees(ITokenizer tokenizer, string text, int capacity)
{
    var tokens = tokenizer.Split(text);
    var window = new ContextWindow(capacity);
    foreach (var t in tokens) window.Add(t);
    return string.Join(" ", window.Visible);
}

Tips

• A rough rule of thumb for English is a few characters per token, or a bit less than one word per token. It varies by model and language, so count with the provider’s own tools when it matters.
• The window holds your prompt and the reply together. A long prompt leaves less room for the answer.
• The model has no memory outside the window. If something fell out, or was never sent, the model cannot use it.
• Tokens are also how usage is measured and limited, so shorter prompts are cheaper and faster.

In the .NET library

This demo window drops the oldest token. Real apps decide what to keep on purpose: they trim or summarize old conversation turns and always keep the system prompt, instead of letting important text fall out by accident.