Frame 01 · The failed model
A readable symbolic bottleneck made a small model worse: the ordinary control won 85.4% of 179 blind comparisons. So I stopped polishing explanations and built a test for whether they actually cause the answer.
Measured failure · three seeds · blind evaluation
Archive print · Raphael Khalid
01 / the failure
Two matched 29.5M-parameter models differed only in their internal state. Forcing one through inspectable symbols damaged story quality and entity tracking.
02 / the harder test
Erase a real reason: the answer should break. Change an unused note: it should not move.
They look identical. Attack them.
Illustration, not a trained-model result. The matching synthetic audit passed across five seeds.
03 / what survived
04 / evidence, not aspiration
05 / the bet
Make the reasoning impossible to hide.
Next: train the ledger and let a matched flat baseline try to beat it.
Scope: This is an executable surrogate and causal-audit testbed—not a trained coding model, not a public benchmark result, and not evidence of parity with a 27B model.
Frame 02 · Latest research run
research runThis panel summarizes the latest checked-in run and the evidence used to choose the next experiment.
This run checks structured state, constrained generation, interventions, and checkpoint recovery. It does not show that the model can reason.
Data contract: add a fresh Autoresearcher export at data/autoresearcher.json and this panel will show the latest run. Credentials and model weights do not belong in the site.
Frame 03 · Researcher Score
The score rewards progress, the final result, useful variety, and a clean record. The formula is fixed before a run starts.
HumanEval+ is the capability check. The deterministic null scored 0/164. That establishes a floor, not T or Q.
Each test instance has 16 or 20 dimensions. Every proposal records a score, an evaluation cost, and an archive niche. The controller also checks the latest 16 proposals.
For each instance, compare the best result with the measured starting point and the exhaustively verified optimum. Clip that value to 0–1 as q(t).
T uses the final q(1). Q compares archive quality with the ideal archive. D uses the whole progress curve. V checks valid and non-duplicate proposals.
If G passes, apply the fixed weights above. If it fails, R is zero and the run is not promoted.
Where T and Q come from: Experiment 16 is not a “16-bit” input to this formula. It is a separate HumanEval+ null. T and Q come from the scored AR1/AR2 landscapes: T is the final normalized result, and Q is the quality of the archive of different solutions.
Frame 04 · Compiler playground
This is the compiler in the browser, built with and running inside this page. The code stays in your browser. Errors update as you type and underline the relevant characters.
The playground includes the lexer, parser, name resolver, type checker, and diagnostic renderer from the command-line compiler. Try removing one item from the opening capacity example.
Frame 05 · Model inspection
At each character, a small experimental model selects one of 1,024 fixed vectors. The linked view shows the full codebook extracted from the checkpoint and the text positions where each entry appears.
See what each symbol fires on, how often, and where it lands in a generated trace.
interpretability.html → The compiler, full sizeLoad a broken example, read the error, edit the line, and watch the message move.
playground/ → The blind testIdentify the 512-symbol model's output. The blind judge preferred the dense control in 85.4% of pairs.
blind-test.html →Frame 06 · What it catches
These systems can store several items by adding their vectors. Beyond a measured limit, retrieval becomes unreliable even though the operations remain valid and the tensor shapes still match. A conventional runtime does not report an error.
Raly tracks the number of superposed items. In this example, the compiler uses a measured capacity of 31 and reports both the source of that bound and the width needed for 40 items.
The block on the right is copied from
compiler/crates/raly/tests/ui/capacity-exceeded.stderr, which is checked
character for character on every build. The playground above
prints the same thing, from the same code: it is the file it opens with.
error[RALY5001]: this bundles 40 items into a space that holds 31 --> capacity-exceeded.raly:9:5 | 9 | bundle( | ^^^^^^^ 40 items superposed here | ...continues to line 15 ::: capacity-exceeded.raly:3:1 | 3 | space Small = MAP[1000] | ----------------------- `Small` holds 31 items | = note: 31 is the capacity of `Small` at dimension 1000, measured at 95% retrieval in experiments/04_capacity = note: past capacity, cleanup returns the wrong atom and accuracy degrades towards chance without anything failing at run time = help: superpose fewer items, or declare `Small` at dimension 1247, or 2048 for a power of two error: 1 error
experiments/04_capacity, where retrieval stays above 95% at thirty-one
items and falls below it at thirty-two. A literature summary put the figure near ten.
Frame 07 · The annotation
When the type lists the roles, a function signature describes part of what the vector represents. That information is available without running the model or training a separate probe.
Compile-time dimension checking is not new; Dex, Futhark, and F# already provide related mechanisms. Interpretable-by-construction modeling is also an established area that includes concept bottleneck models, sparse transformers, and KANs. Raly's narrower experiment is to put VSA-specific structure in the .
Frame 08 · How it works
raly check runs every pass every time. No pass is allowed to stop the next one, so
a typo does not hide an unknown name and an unknown name does not hide a capacity error. You
receive a combined list in source order.
.raly file
width tracked as a unit of measure, so a mismatch prints the leftover ratio instead of the words "unification failed"
family is a short fixed list, so MAP[1024] and FHRR[1024] stop being the same type
how full is a range of whole numbers over measured capacity: bundle adds, bind multiplies, cleanup drops back to one
slots use
,
so taking out a slot the vector never had is a compile error
import torch D = 1000 codebook = torch.sign(torch.randn(1000, D)) scene = torch.zeros(D) for i in item_ids: # 40 of them scene += codebook[i] # add it to the bag # No exception. Shapes correct throughout. # What you read back is no longer what you # put in, and nothing anywhere says so.
space Small = MAP[1000] fn everything(...) -> Vec[Small] { bundle(a, b, c, /* … 40 items … */) } error[RALY5001]: this bundles 40 items into a space that holds 31 // This is a golden test. The exact text // above is asserted on every build.
Frame 09 · Experimental results
Each result is linked to its methods and limitations. Where applicable, the experiments define failure criteria in advance, report a null or baseline, and verify headline numbers with a second calculation. Negative results are included.
Retrieval stays above 95% up to thirty-one items bundled into a width-1000 vector, roughly three times a literature baseline. Four widths were measured; this is the number the compiler prints.
experiments/04 →On BEIR scifact, mean-pooling eight passages drops recall from 0.877 to 0.619. A max-scoring control attributes about 76% of the loss to averaging itself; doubling nominal width does not remove the cost.
experiments/07 →At matched parameters, forcing the middle through 1,024 symbols moved top-1 accuracy from 0.8383 to 0.8065. Larger codebooks improved both accuracy and role prediction in this synthetic experiment.
experiments/06 →We measured how well a symbol predicts whether the character it fired on belongs to a premise, a working step, or the answer. Three ways of guessing, on the same text:
| how you guess the role | gets it right | over the floor |
|---|---|---|
| always say the commonest one | 0.5801 | the floor |
| look at the raw character, no model | 0.6314 | +0.051 |
| look at which of the 1024 symbols fired | 0.6640 | +0.084 |
The symbols beat the raw-character control by +0.033. This is the additional predictive
signal associated with context-dependent code selection. Reproduce the comparison with
experiments/06_discrete_core/leakage.py.
The full findings, including the negative result on this project's own idea, live in the repository. All experiments →
Frame 10 · Overnight research
On generated repair tasks, typed legality and public search raised the learned sketch from 52.1% raw pass to 89.6% full-system pass. The deterministic null also reached 89.6%. A state-only controller changed 50.0% of raw decisions when state was removed. These are synthetic Python results, not evidence from the Raly compiler or a public coding benchmark. The next checks are EvalPlus HumanEval+, MBPP+, BigCodeBench-Hard Complete, and a later LiveCodeBench sample. Read the audit →
Frame 11 · What exists
The items marked as missing have not been implemented.
raly checks programs; it cannot run them.Limitations