Evaluations: test cases, scores, regressions

Prompts change behaviour in ways you cannot see by eye. Run fixed test cases against each version and compare case by case.

The idea

Changing a prompt can fix one problem and quietly break another, and model output varies, so looking at a few answers is not enough. An evaluation is a fixed set of test cases (an input and what a good answer looks like), a scoring rule, and a score for each prompt version. Run both versions on the same cases, then look at the overall score and at the individual cases that got worse. Those are your regressions.

Code

public record TestCase(string Name, string Input, string Expected);

public static class Evals
{
    public static async Task<Dictionary<string, bool>> EvaluateAsync(string prompt, List<TestCase> cases)
    {
        var results = new Dictionary<string, bool>();
        foreach (var c in cases)
        {
            string output = await RunModelAsync(prompt, c.Input);
            results[c.Name] = output.Trim() == c.Expected;
        }
        return results;
    }

    public static async Task CompareAsync(string promptV1, string promptV2, List<TestCase> cases)
    {
        var v1 = await EvaluateAsync(promptV1, cases);
        var v2 = await EvaluateAsync(promptV2, cases);
        var regressions = cases.Where(c => v1[c.Name] && !v2[c.Name]).ToList();
        Console.WriteLine($"v1 {v1.Values.Count(ok => ok)}/{cases.Count}, v2 {v2.Values.Count(ok => ok)}/{cases.Count}");
    }
}

Tips

• Interview line: "I treat prompts like code: versioned, with an eval suite that runs on every change, and a gate on regressions."
• A higher total score can hide a regression. Always list the cases that passed before and fail now.
• Whenever a real user finds a failure, add it as a new test case so it can never silently come back.
• Exact match works for labels. For open-ended answers use checks (contains the key fact, valid JSON, within length) or a second model as a grader, and spot-check that grader.
• Use a low temperature for evals and re-run flaky cases, because one run can mislead. Five cases is a demo; real suites have dozens to hundreds.

In the .NET library

Run evals as a test project (xUnit, NUnit or MSTest) or a console tool in CI. A build that fails on regressions catches prompt changes the same way a failing unit test catches code changes.