Temperature and sampling

The same scores give a sharp or a flat probability list depending on temperature. Low means predictable, high means varied.

The idea

Before the probabilities are formed, the model produces a raw score (a logit) for each candidate token. Temperature divides those scores. A low temperature widens the gaps, so the top token takes almost all the probability and the output is predictable. A high temperature shrinks the gaps, so more tokens have a real chance and the output is more varied. Sampling then rolls a random number and walks down the cumulative probabilities.

Code

public static class Sampling
{
    public static double[] Softmax(double[] logits, double temperature)
    {
        var scaled = logits.Select(l => l / temperature).ToArray();
        double max = scaled.Max();
        var weights = scaled.Select(s => Math.Exp(s - max)).ToArray();
        double sum = weights.Sum();
        return weights.Select(w => w / sum).ToArray();
    }

    public static int Greedy(double[] probs) =>
        Array.IndexOf(probs, probs.Max());

    public static int Sample(double[] probs, Random rng)
    {
        double roll = rng.NextDouble();
        double cumulative = 0;
        for (int i = 0; i < probs.Length; i++)
        {
            cumulative += probs[i];
            if (roll < cumulative) return i;
        }
        return probs.Length - 1;
    }
}

Tips

• Interview line: "Temperature controls randomness. Low is good for extraction, classification and code. Higher is better for brainstorming and creative writing."
• Temperature 0 cannot divide by zero, so APIs treat it as greedy: always take the top token. It is mostly repeatable, but do not assume it is perfectly identical on every run.
• Top-p (nucleus) sampling is a related control: only sample from the smallest group of tokens whose probabilities add up to p.
• The allowed temperature range differs between providers, so check your API’s documentation. 2.0 here is just to make the effect easy to see.

In the .NET library

Treat temperature as a request setting, not a code change. Keep it in configuration so you can use a low value for a classification endpoint and a higher one for a brainstorming endpoint without redeploying.