The discrete model is easier to inspect, but it writes worse stories.
Two models of the same size, trained on the same children's stories for the same amount of compute. One is a standard transformer. The other passes its middle state through a 512-entry discrete codebook, making that state easier to inspect. In blind comparisons, the judge preferred the standard model in 85.4% of pairs.
The test below uses the same story pairs. Choose the completion you think came from the discrete model and compare your result with the judge's.
Take the test
Which completion came from the discrete model?
Each story opening is the first 40 tokens of a real story the models never saw in training. Both completions were written under identical settings and cut off at 200 new tokens, so each one stops mid-sentence. Their left-right order is randomized when the page loads.
Completion A
Completion B
Which one came from the model with the 512-entry codebook?
The bottom bar is the null comparison. Two seeds of the standard model should be indistinguishable on average, and the judge returned 51.7%. The 85.4% preference in the main comparison is well above that reference point.
All 60 pairs come from experiments/09_story_quality/generations.json, taken
in the order produced by the generator. The examples were not selected by quality.
The result
The discrete model did not meet the preregistered threshold.
Before generating the stories, we committed a rule to the repository: the discrete model would pass if its grammar and consistency scores were within the uncertainty range of the standard model, even though its was already worse. The generated samples were then judged and evaluated using that rule. Both criteria missed by a wide margin.
In this experiment, the added inspectability came with a substantial capability cost.
| what was measured | standard | discrete | real stories |
|---|---|---|---|
| 1.6608 | 2.2892 | n/a | |
| 58.8% | 48.9% | n/a | |
| Grammar, 1 to 10 | 7.00 [6.78, 7.22] |
4.54 [4.27, 4.82] |
9.47 |
| Consistency, 1 to 10 | 5.83 [5.58, 6.08] |
3.08 [2.86, 3.30] |
9.47 |
| Head-to-head win rate | 85.4% to the standard model, [79.5, 89.8] over 179 comparisons | ||
Square brackets are 95% over three seeds. Every model saw the same 60 openings, so each opening is its own control and the differences are taken within a prompt before being averaged.
Judge controls
Two controls check whether the judge can separate quality from noise.
It can spot good writing. Real human-written passages from the same story collection were dropped into the same blind format and judged as if they were model output. They scored 9.47 on grammar and beat the standard model 93% of the time. This checks that the judge can recognize a clear quality difference.
It does not invent winners. We also ran the standard model against a second copy of itself trained with a different random seed. Those two use the same architecture and training recipe, and the judge returned 51.7%. This is the reference point for an indistinguishable pair.
Position bias is present and cannot explain it. The judge picks whichever completion it reads first 41.6% of the time. That leaning is the same size in every comparison type, and sides were shuffled, so a constant offset cannot produce an 85 to 15 split in one comparison and a 52 to 48 split in another in this balanced design.
| comparison | n | win rate |
|---|---|---|
| standard against discrete | 179 | 85.4% [79.5, 89.8] |
| standard seed 0 against standard seed 1 | 60 | 51.7% [39.3, 63.8] |
| real human stories against standard | 60 | 93.3% [84.1, 97.4] |
Limitations
- The judge almost never called a tie, once in about 300 comparisons, including zero times in the 60 comparisons where a tie was the honest answer. So the low tie rate in the main test is not separate evidence of a gap, and we do not count it as any.
- One judge model, not calibrated against a second judge or against human raters. The effect is large; a close result would need more than this.
- One size, one budget, one codebook: 29.5M parameters, 87M tokens, 512 symbols at one point in the network. Whether the gap narrows at a different scale or with a bigger codebook was not tested here.
- One decoding setting, matched across both models for fairness but not tuned for either.
Observed failure
The discrete model often loses track of entities.
Many sentences from the discrete model are locally well formed, but the samples often switch the identity of an object. Given an opening about a kite, one completion contains the following sequence in a single paragraph:
It is a star of water. "Wow, look at the balloon!" Ben says. "It is so pretty!" "My kite!" Anna says. "My flag is better with all the other toys." They have fun with the drum. They share it with the other kids. They make new shapes for the trumpet.
The object changes names six times. The standard model stays on the kite but becomes repetitive. One possible explanation is that local grammar requires less persistent state than entity identity. This is an interpretation of the samples, not a measured mechanism.
It shows in the scores too. Consistency falls further than grammar, 2.72 points against 2.44, and the standard model's own consistency score of 5.83 is also low.
The full results and evaluation data are available in the repository.
The write-up, the 420 generations, the 720 judge calls and the exact prompt the judge was given are all committed. The total spend on judging was $0.03.