2026-07-13

14:55 dynaconf closed-door lessons for tomo Seven honest runs on one task, three passes and four fails, read together. The lessons that transfer to tomo: a broad edit that regresses a green test is worse than no edit and wants a do-no-harm gate, spend does not track progress, cache-read is where the money actually goes… 14:45 dynaconf 5.6 family + analyzer false leak gpt-5.6-terra and -sol both pass dynaconf-1225 with the doors shut, and both got flagged as answer leaks. Reading the trace, the flag is wrong. 14:35 dynaconf opus offline The honest opus run on dynaconf-1225 is the most expensive fail in this comparison and the only run that ends with the repo worse than it started. 14:25 dynaconf sonnet offline The earlier sonnet run passed dynaconf-1225 by fetching the merged pull request. Close the network door and run it again and the fetch is gone. 14:15 dynaconf gpt-5.6-luna offline Run the newest codex model on dynaconf-1225 with the git-history door and the network door both closed, and for the first time in this comparison a model solves it honestly. gpt-5. 14:01 dynaconf gpt-5.5 offline The flagship codex model runs dynaconf-1225 with both answer doors closed. It writes nineteen edits across every loader, the validator, and the cli, twice what the cheap model touched, spends six times as much, reaches no answer, and fails on the exact same two settings-loader… 13:49 dynaconf gpt-5.4-mini offline The cheapest codex model runs dynaconf-1225 with both answer doors closed: git history pruned so the fix commit is unreachable, and the shell sandboxed so it cannot fetch the answer PR. It writes a real nine-edit fix, reaches no answer, and fails on the settings-loader tests. 12:40 dynaconf opus answer fetch Claude Opus 4.8, the most expensive model in the comparison, ran dynaconf-1225 and passed by fetching pull requests over the network. It read PR #1204, the source the task asked it to port, and PR #1225, the merged answer that grades it. 12:35 dynaconf sonnet answer fetch Claude Sonnet 5 ran dynaconf-1225 and passed, but the trace shows it did not solve the bug. It ran gh pr view 1225 and gh pr diff 1225, read the merged pull request that fixed the very issue it was handed, listed the fix commits, and applied them. 12:30 dynaconf haiku clean fail Claude Haiku 4.5 ran dynaconf-1225 without reaching the network, wrote a real source fix, and failed. It threaded the identifier argument through every loader, the same broad refactor gpt-5.4 tried, and regressed a test that started green. 12:20 With the leak closed, dynaconf sorts the models The same leak-free dynaconf task passes on gpt-5.6-sol and gpt-5.5 and fails on gpt-5.4, so the fix that removed the git shortcut left a task that actually measures the model. 12:05 mini vs sol on python-control Four real codex subscription runs, gpt-5.4-mini and gpt-5.6-sol on two tasks, priced through our new single source of truth. On python-control both models, cheap and flagship, converge on the identical edit and fail the identical three tests. 11:50 dynaconf answer leak gpt-5.6-sol, the most expensive model we can reach, passed a dynaconf task without reasoning out the bug. The trace shows how: it diffed the base commit against the upstream fix commit, which the work-tree clone left reachable, and applied it. 11:05 churn guard vs claude-code The third and last of tomo's runaway shapes. A turn that keeps editing but never converges, writing scratch scripts or the same file over and over, now stops instead of burning a hundred rounds. 10:05 dynaconf tomo guard vs pi The follow-up to tomo's git-archaeology runaway. tomo now bounds a turn that investigates without ever editing, so the same dynaconf run stops at 41 requests instead of 132 and 1.7 million tokens instead of four million. 08:44 python-control tomo scratch runaway A second runaway with a different shape. On a python-control conversion bug, tomo made 34 edits and still failed, because 33 of them were throwaway debug scripts it wrote to instrument the problem rather than fix it. 08:19 fonttools tomo over-normalized The one swebench-live failure where tomo did everything right and still lost. On a fonttools glyph-reordering bug it found the exact file, wrote a fuller fix than the maintainers, and verified its work, then failed a single hidden test because it reused a variable that forces . 08:04 dynaconf tomo runaway tomo's worst run of the sweep: on a dynaconf bug it spent 132 requests and four million tokens running git log, git diff, and git show to reverse-engineer a fix from history, hit the fifteen-minute wall, and never edited a single file. 01:26 gitingest tomo tomo solves a real gitingest issue the way the benchmark intends: it reads the source, finds the one branch that only handles https, adds the http case, and verifies with the project's own tests. One source edit, no network, 242k tokens. 01:21 cfn-lint pi A second rival on the cfn-lint task. pi never leaves the repo, never fetches the pull request, and fails exactly where tomo failed: its source change does not produce the arbitrary graded wording. A short confirm, with the caveat that the free-tier rate limit cut the run short. 01:11 cfn-lint opencode opencode passes a cfn-lint task whose graded wording appears nowhere in the repo. The trace shows how: it fetched the fixed source from the project's main branch and the merged pull request's diff, then copied the exact new messages into the checked-out source. 01:00 cfn-lint tomo tomo reads a cfn-lint issue, implements exactly the message it asks for, and fails the grade. The graded wording is a generic validator message the maintainers changed instead, and it appears nowhere in the checked-out repo. 00:50 faker --yolo fix The follow-up to the faker lockout. tomo gains a --yolo mode that runs it fully autonomous, the same way every rival already runs. The exact task tomo had solved but could not write now passes, and passes leaner: 40 percent fewer tokens and half the model calls. 00:14 faker IBAN lock tomo diagnoses a Belgian IBAN bug and writes the exactly correct fix, then cannot apply it. A reference URL it fetched tripped its own prompt-injection guard, which escalated every later edit to an approval that never comes in headless mode.