Skip to content
tomo-labs

P-M4-3: the infra column shows tier structure, and grades ignore it

The infra column shows tier-correlated structure, and no arm's grade verdict changes when capped fails are excluded versus included.

id:        P-M4-3
filed:     2026-07-17, from spec 2105 doc 10 section 8.5, before the M4 full gate
suite:     every suite the gate runs
tier:      all tiers
engine:    every arm on the sheet
band:      floor: rate-limit caps concentrate on the free tier and stream behavior
           differs by provider (the infra column shows real structure, not noise).
           ceiling: no arm's grade verdict changes in a sensitivity check that excludes
           versus includes capped fails.
mechanism: caps are properties of the provider weather, not of the agents, so they
           should correlate with tier and provider; and because the report already
           keeps capped fails out of every rate, re-including them should only be able
           to move a verdict if the separation machinery is broken.
settled by: the M4 full gate report's CAPPED, 429S, and DROPS columns, plus a
           sensitivity recomputation over the same result set.
           go run ./cmd/lab report && go run ./cmd/lab report --suite swebench-live
result:    (open)
verdict:   (open)

This prediction is a canary for the infra-separation machinery itself. If excluding capped fails flips a verdict, the gate has an infra problem to fix before its verdicts mean anything, and better to have said so in advance.