II · THE IDEA · ARTIFICIAL INTELLIGENCE
Evaluating Models
▶ Listen · narrated
Benchmarks turn vague claims about capability into comparable numbers, but the tests themselves are artefacts with histories, biases, and an awkward habit of leaking into training data.
At a glance
- What they measure
- Task-specific accuracy, human preference rankings, or both
- MMLU coverage
- 57 subjects, from elementary maths to professional law
- Contamination risk
- Test questions circulate publicly; models may train on them
- LLM-as-judge
- Using one model to score another's outputs against criteria
Imagine you want to know which of two calculators is better. You could give them both the same hundred maths problems and count how many each gets right. That is what a benchmark does for language models: it gives them all the same test and compares the scores. MMLU is a test with thousands of multiple-choice questions across dozens of subjects. A model that scores ninety per cent got nine out of ten questions right. But the test has problems. If a calculator had seen the answer key before the exam, its high score would not mean it is good at maths—it would mean it memorised the answers. That is contamination, and it happens to language models because the tests are public and the training data is scraped from the internet. The other problem is that multiple choice is not how you use a model in practice. Being good at picking the right answer from a list is not the same as being good at writing a coherent paragraph or solving a novel problem.
A benchmark defines a task, a dataset of examples, and a scoring function. MMLU uses 15,908 multiple-choice questions across fifty-seven subjects, split into development, validation, and test sets. The model is prompted with a few-shot prefix—typically five examples from the development set—then asked to predict the correct option (A, B, C, or D) for each test question. Accuracy is computed per subject and averaged. Contamination occurs when test examples, or near-duplicates, appear in the training corpus. Exact n-gram matching can detect some contamination, but paraphrases and embeddings of questions in analysis or discussion evade it. Some labs now report overlap metrics, but no standard methodology exists. MT-Bench uses eighty multi-turn prompts across eight categories (writing, reasoning, maths, coding, extraction, STEM, humanities, roleplay). A judge model—GPT-4 in the original work—scores each response on a ten-point scale using a rubric. The judge's prompt includes the conversation history, the evaluation criteria, and sometimes reference answers. Scores are averaged across prompts. The method assumes the judge's preferences align with human preferences, which holds approximately but not exactly: judge models favour longer responses and confident phrasing, even when accuracy suffers. Chatbot Arena collects pairwise human preferences and computes Elo ratings using the Bradley-Terry model. The system is vulnerable to selection bias in both prompts (user-submitted) and raters (self-selected volunteers).
Look closer
MMLU is multiple choice across fifty-seven subjects
The Massive Multitask Language Understanding benchmark presents a model with questions spanning elementary mathematics, US history, professional law, computer science, and moral reasoning, among others. Each question offers four options; the model's task is to select one. The breadth is deliberate: a model scoring well must hold knowledge across domains that humans typically specialise in separately. Scores are reported per subject and as an aggregate, and that aggregate has become a standard line in model announcements.
Contamination means the test was in the training set
Because benchmark datasets circulate openly, they can end up in the corpora used for pre-training or fine-tuning. A model may then score well not because it has learned the underlying skill, but because it has seen these specific questions before. Detecting contamination is harder than it sounds: exact string matching misses paraphrases, and a question reworded slightly can still trigger memorisation. Some labs now report contamination checks, but the methods vary and no standard has settled.
MT-Bench uses a model to judge multi-turn conversations
Where multiple-choice tests reduce everything to a single correct letter, MT-Bench asks models to hold conversations across two turns, then uses another model—often GPT-4 in the original work—to score the quality of the responses. The judge model assigns ratings based on helpfulness, relevance, and coherence. This shifts the bottleneck: instead of designing questions and answers by hand, you design the judging criteria and accept that the judge itself has biases. The method scales better than human evaluation, but it imports the judge's preferences wholesale.
The story
Benchmarks exist because vague claims are cheap. Saying a model is good at reasoning or fluent in conversation costs nothing and means little until you define a task, a dataset, and a scoring rule. MMLU, introduced by Hendrycks and colleagues, became widely used because it offered breadth and a clear number: accuracy across fifty-seven subjects, from abstract algebra to virology. The questions are multiple choice, which makes scoring mechanical. A model either selects the correct letter or it does not.
The appeal is obvious. You can compare models trained by different labs, on different data, with different architectures, and produce a single percentage that fits in a table. Leaderboards followed. So did the problems.
First, multiple choice is a constrained format. It tells you whether a model can recognise the right answer among four options, not whether it can generate that answer unprompted, explain its reasoning, or apply the knowledge in a novel context. A model might score well by ruling out implausible distractors without fully understanding the question. The format also makes contamination easier to exploit: a memorised question-answer pair is worth full marks.
Contamination is the second problem, and it is structural. MMLU's questions are public. They live in GitHub repositories, academic papers, and blog posts analysing model performance. Web scrapers harvest all of this. If a model's training corpus includes the benchmark, either directly or embedded in discussion, the model may have seen the test before it sat the exam. Some labs filter known benchmarks from their training data; others report contamination metrics. But detection is imperfect, and incentives point the wrong way: a contaminated model scores higher, and a higher score drives adoption.
The third problem is that a single number compresses fifty-seven subjects into one, hiding where a model is strong and where it guesses. A model might excel at formal logic and fail at moral reasoning, or vice versa, and the aggregate obscures both. Per-subject scores exist, but they rarely make it into headlines.
MT-Bench, introduced by Zheng and colleagues, takes a different approach. Instead of multiple choice, it presents multi-turn conversation prompts: write a travel itinerary, debug some code, give advice on a dilemma. Instead of checking answers against a key, it uses another model—LLM-as-judge—to score the responses. The judge assigns ratings, typically on a ten-point scale, based on criteria like helpfulness and coherence.
This solves some problems and creates others. It scales better than hiring humans to read and rate thousands of responses. It handles open-ended tasks that multiple choice cannot capture. But it replaces one measurement problem with another: now you are measuring how well a model satisfies a judge that is itself a model, with its own biases and failure modes. If the judge prefers verbose answers, models learn verbosity. If it rewards confident tone over accurate content, models learn confidence. The judge's preferences become the de facto specification.
Chatbot Arena, also described in the Zheng paper, sidesteps the judge problem by using humans. Two models answer the same prompt anonymously; a human picks the better response. Aggregate enough comparisons and you get an Elo ranking, borrowed from chess. This is closer to what users care about—does this model produce output I prefer?—but it is expensive, slow, and still biased by who the raters are and what prompts they choose. A model optimised for Arena performance is optimised for the preferences of the people who vote there, who are not a random sample of all users.
None of this means benchmarks are useless. They are necessary. A model that scores twenty per cent on MMLU is not ready for deployment, and a model that scores ninety per cent has learned something real. But the number is not the capability. It is a proxy, and every proxy has a gap between what it measures and what you actually care about. The gap is where contamination hides, where format constraints bite, and where a judge's biases propagate. Treating the benchmark as the goal is how you get models that excel at tests and falter in production.
Why it mattered then
MMLU arrived in 2021, as model scale was climbing and labs needed a way to demonstrate that larger models were not just larger but more capable. The fifty-seven subjects were chosen to cover what a well-educated human might know, and the multiple-choice format made it possible to test thousands of questions without human grading. It gave the field a shared reference point. When a new model launched, its MMLU score was expected in the announcement, alongside parameter count and training compute. MT-Bench and Chatbot Arena followed in 2023, as attention shifted from knowledge retrieval to conversational quality. Multiple choice could not capture whether a model was helpful, coherent, or pleasant to use, and human evaluation did not scale to the pace of releases. LLM-as-judge offered a middle path: automated, open-ended, and fast enough to keep up with weekly model drops. The methods emerged because the questions the field was asking had changed.
Why it matters now
Benchmarks now shape training decisions as much as they measure outcomes. Labs tune models to perform well on published tests, sometimes explicitly, sometimes through data choices that implicitly optimise for benchmark coverage. This is not always bad—targeting a benchmark can surface real weaknesses—but it creates a feedback loop where models become good at the things benchmarks measure and neglect the things they do not. Contamination remains unresolved. Some labs filter benchmarks from training data; others do not disclose their methods. A model's score is only meaningful if you trust the lab's contamination controls, and trust is unevenly distributed. LLM-as-judge has become standard practice for evaluating conversational models, but the choice of judge matters enormously. A model judged by GPT-4 is being scored by OpenAI's preferences, which may not match yours. The method is pragmatic, not neutral. Leaderboards proliferate, each with different tasks, judges, and contamination risks, and the result is a measurement landscape that is both more detailed and harder to interpret than the single-number summaries suggest.
The surprising detail
Chatbot Arena rankings are based on pairwise human comparisons, and the humans are self-selected volunteers who visit the Arena website. This introduces a demographic skew: the raters are disproportionately people interested enough in language models to spend time voting on outputs, which likely means they are more technical, more English-speaking, and more tolerant of experimental systems than the general population. A model that ranks highly in the Arena is a model that appeals to this specific group. Whether that generalises to other users is an open question, and one the Elo ranking does not answer. The method is transparent about this limitation, but the limitation often gets lost when the ranking is cited elsewhere.
Remember this
Every benchmark measures a proxy, not the thing itself. The gap between them is where contamination, format bias, and misaligned incentives all live.
Test yourself
A lab reports that its new model scores ninety per cent on MMLU, up from eighty-five per cent for the previous version. Name two distinct reasons this might not mean the new model is meaningfully more capable in production use.
First, the improvement could be due to contamination: if the new model's training data included MMLU questions, it may have memorised answers rather than learned the underlying skills. Second, the gain might be concentrated in a few subjects where the model was already strong, leaving weaknesses in other areas unchanged—the aggregate hides the distribution. A third reason worth knowing: multiple-choice performance does not directly predict open-ended generation quality. A model that is better at recognising correct answers among options may not be better at producing useful answers from scratch, which is what most production tasks require.
Go deeper
- Measuring Massive Multitask Language Understanding · arXiv · Dan Hendrycks et al. · 2020-09-07
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · arXiv · Lianmin Zheng et al. · 2023-06-09
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.