Skip to content
tomo-labs

Predictions

The prediction ledger: every measurable claim about a future run, filed before the run, with a floor, a ceiling, and the mechanism that would explain the number. Verdicts are written against the filed band, hit or miss, and an entry is never edited after its run starts.

Every gate run in this lab is bound to a prediction that was filed before the run launched. This section is that ledger: one page per prediction, checked into git, so "was this filed before the run" is answered by commit history rather than by memory.

Why file predictions at all

A benchmark sweep with no prediction is a pile of numbers that can be read to say anything. The failure mode is familiar: run enough arms, then narrate whichever gap looks flattering as if it had been the hypothesis all along. Filing the band first closes that door. If the number lands inside the band, the mechanism story earned its keep. If it lands outside, that miss is the finding, and it gets a written verdict instead of a quiet reframe.

The ledger's aggregate hit rate is itself a health number for the lab. A ledger that always hits is predicting too wide. A ledger that mostly misses means the mechanism stories are decoration.

The entry format

Each entry carries the same fields, fixed here so every page reads the same way:

id:        the prediction's name, stable forever
filed:     the date the band was frozen, before the settling run
suite:     which task set settles it
tier:      which model tier the settling run uses
engine:    the arm or arms under test
band:      floor and ceiling, both sides, so the claim can miss in either direction
mechanism: the one-sentence story that explains the band, the load-bearing part
settled by: the run that grades it, with the reproduce command
result:    filled after the run, a link to the note or report that carries the numbers
verdict:   hit, over, or under, with one sentence

The mechanism sentence is what separates a prediction from a guess. "The oi engine will land 12 to 14 of 14 because the finishguard family closed the three last-mile failure shapes" is a prediction. "The oi engine will do well" is not, because nothing could falsify it.

The band has a floor and a ceiling because a claim that only bounds one side cannot miss in the direction that matters. "At least X" can never miss high, and missing high is evidence too: a mechanism that overperforms its own theory is not understood yet.

The rules

The ledger is append-only. An entry is never edited after its settling run starts; the only writes allowed after that point are the result link and the verdict. A mistaken prediction is answered by its verdict, not by revision. A revised band is a new entry with its own filing date.

Over and under are distinct verdicts on purpose. An under means the mechanism story overpromised. An over means the model or a guard did work the story did not credit. Both feed the queue of what to try next.

A conditional prediction is still a registered one. If the run that would settle an entry never launches in a given window, the entry is carried forward unscored, never quietly dropped.

Reading the ledger

Entries are named by their id. The P-C family tests the cost model, the P-X family tests the context model, the P-M family (bare numbers) tests the model policy, and the P-M0 through P-M4 entries are the milestone gate predictions of the 2105 campaign. The naming is inherited from the spec docs that filed each claim; the ledger keeps one flat list rather than one list per doc, so a reader can audit the whole register in one pass.

P-C1 Landing the effective-cost columns changes no engine behavior but reorders at least one published rival comparison. P-C2 On DeepSeek-class caching, tomo-oi's effective cost per pass on the core suite lands in 0.3x to 0.6x of tomo-cx. P-C3 The spend ceiling never fires on the core 14 at defaults, and fires first on swebench-live invention-shaped tasks. P-M0-1 The M0 engine estate changes no core-14 pass count; its wins are built for the harder suites. P-M0-2 The gitingest cheap-model salvage rate holds at or above 6/8 after the M0 estate lands. P-M1-A The aggregated table shows at least one previously-published single-run gap shrinking inside noise under n >= 10. P-M1 Cheap-first loses no grade on substitution-tagged tasks and cuts cost per pass by at least 2x. P-M2-1 The M2 pairing run reproduces tomo-oi at 13 to 14 of 14 at under half of tomo-cx's total tokens. P-M2-2 The false-positive scanner finds at least one latent guard over-trigger during M2 slice two's development. P-M2 The targeted effort purchase converts at least one standing near-miss per sweep, under 2.5x fresh tokens. P-M3-A The spend-ceiling slice changes nothing on any suite at defaults; a pure rail is invisible until a runaway makes it visible. P-M3 New zen-class models onboard without parser changes at least half the time. P-M4-1 At the first full gate, tomo-oi on the free tier is level or better on grade and under half the best rival's effective cost. P-M4-2 On the frontier tier, the grade clause is met on the core suites, and the cost clause verdict is OPEN. P-M4-3 The infra column shows tier-correlated structure, and no arm's grade verdict changes when capped fails are excluded versus included. P-X1 The agent-engine head+tail truncation fix changes no pass rate, and improves rounds on at least one debugging-shaped task. P-X2 The context overflow valve fires on zero of the core 14, and fires only on long dynaconf-class runs, if anywhere. P-X3 If the working-set header A/B runs, the header cuts rounds on multi-file tasks, or it does not land.