evaluation-with-adversarial-n‑way-forcing

Judging a language model by the text it writes freely is a noisy business. The output can be fluent, on topic, and confident, and still be wrong on the one point that mattered. Automatic scoring tools mostly count how many words an answer shares with a reference answer, so they reward safe, generic phrasing. Human readers catch subtler errors, but they tire, disagree with one another, and cannot read every sample a large evaluation produces. The result is a score that blends real understanding with mere polish, in proportions nobody can state.

Forced choice removes the ambiguity by removing the freedom. The model is not asked to invent a continuation — the next stretch of text that follows a prompt. It is handed a short, fixed list of candidate continuations. One is correct. The rest are not. The model must pick the correct one, or score all the options so that the correct one comes top. Nothing about the prompt changes between models or between runs; only the choice varies. What gets measured is preference — which ending the model rates highest — not invention.

The word adversarial describes how the wrong options are made, and it is where the method earns its keep. They are not random sentences pulled from unrelated writing; random strings are easy to reject, and a test built from them flatters everyone. Instead, each distractor — a wrong option written on purpose to attract a careless answer — is built to be tempting. One might keep the right names and places but reverse the relationship between them. One might flip a negation, so that a line reading identically in every other way now says the opposite. One might answer a neighbouring question in the same subject, or follow a pattern that is common in ordinary text but false here — the mistake people and models most often make on that kind of item. Set beside the true continuation, every option now sounds equally polished, equally on topic, equally finished.

That is the whole trick, stated plainly: if every option reads well, reading well cannot be what the model is scored on. Polish carries no information once all the candidates have it. The only thing left to separate a correct answer from a guess is whether the model genuinely favours the true line over lines engineered to imitate it. A model leaning on shallow cues — shared vocabulary, familiar phrasing, the general shape of a right answer — will be drawn to the traps, because the traps were built from exactly those cues.

The term n-way names the size of the list, and it works as a control knob. A 2-way item offers two endings, so a model guessing blindly is right half the time. A 4-way item drops blind guessing to 1 in 4; a 10-way item, to 1 in 10. The longer the list, the less a good score can be luck — but only if the added options are genuinely hard. Pad the list with obvious rubbish and the score rises without the difficulty rising. This is why the value of n, and the recipe used to build the distractors, belong in the report beside the headline number. They are not implementation details; they are part of what was measured. The same correct continuation can look strong against weak foils and weak against a crowded field of near-misses.

Scoring runs along one of two routes. The first uses the model's own numbers. A language model assigns every possible continuation a probability — its own estimate of how likely that text is to follow the prompt. The evaluation reads off which option the model rated most likely. One correction is usually applied first: a longer option accumulates a lower total probability simply by having more words, each of which costs a little probability on its own. Left uncorrected, the test would quietly favour short endings. So the score is divided by the option's length — an average per word — before the comparison is made.

The second route treats the model as an exam candidate. The options are labelled A, B, C and D, folded into the prompt, and the model is asked to name one letter. This looks like the same question, and usually is. But the two routes can disagree on the very same item, particularly for models tuned to follow instructions: a model may assign its highest probability to the correct continuation yet name the wrong letter, or the reverse. Which route a leaderboard used is another fact the headline number hides.

There are quieter failure modes too. If the material used to build the test leaked into the model's training data, the model may recognise the correct line rather than prefer it on merit, and the score inflates. And there is no single agreed standard for building adversarial distractors, nor for exactly when to normalise by length. Results can shift when those choices shift, which is why any one suite is best treated as a specific instrument, not a universal yardstick.

The method does not claim to measure writing quality, and that narrowness is its strength. A model can rank a fixed menu correctly and still produce weak prose when left alone with an empty page. It can write beautifully and still fall for a well-made trap. What forced choice offers is a controlled comparison: same prompt, same options, one question — does the true continuation win?