Skip to content
tomo-labs

July 2026

Experiment reports captured in July 2026, by day. Each is one tool on one task, pinned to the exact build so it reproduces.

Runs captured in July 2026, grouped by the day they ran. Each item below is one experiment: one tool, one task, one verdict.

2026-07-13

  • 01:26 (GMT+7) - gitingest: tomo fixes it the honest local way. tomo reads the source, finds the URL branch that only handles https, adds the http case, and verifies with the project's own tests. One source edit, no network, 242k tokens: the honest-local pass, the opposite of an answer lookup.
  • 01:21 (GMT+7) - cfn-lint: pi fails it the honest way too. A second rival, cut short by the free-tier rate limit, never fetches the pull request and never rewrites the source wording, failing the same structural way tomo did.
  • 01:11 (GMT+7) - cfn-lint: opencode passes by fetching the answer PR. opencode passes by fetching the fixed source on the project's main branch and the merged pull request's diff, then copying their exact new wording into the checked-out source. The pass proves the task is reachable, but only by looking up the answer online, so it stays a marker, not a target for tomo to chase.
  • 01:00 (GMT+7) - cfn-lint: tomo fixes the issue, the grade wants something else. tomo implements exactly the message the issue asks for and fails the grade. The graded wording is a generic validator message the maintainers changed instead, and it appears nowhere in the checked-out repo.
  • 00:50 (GMT+7) - faker: the fix that let tomo apply its own answer. The follow-up to the lockout below. tomo gains a --yolo mode that runs it fully autonomous, the way every rival already runs, and the task it had solved but could not write now passes. It passes leaner too: 40 percent fewer tokens and half the model calls.
  • 00:14 (GMT+7) - faker: solved, then locked out by a web fetch. tomo writes the exactly correct Belgian IBAN fix, then cannot apply it. A reference URL it fetched tripped its own prompt-injection guard, which escalated every later edit to an approval that never comes headless. A run tomo had already won, lost to its own safety switch.

2026-07-12

  • 23:49 (GMT+7) - mesa: the right name, in the wrong place. tomo, on the mesa remove all agents issue. It fixes the behaviour correctly but fails the grade on a method name, having already used the winning name in its own throwaway test. A close read of why the task is fair and the bug is tomo's.
gitingest tomo tomo solves a real gitingest issue the way the benchmark intends: it reads the source, finds the one branch that only handles https, adds the http case, and verifies with the project's own tests. One source edit, no network, 242k tokens. cfn-lint pi A second rival on the cfn-lint task. pi never leaves the repo, never fetches the pull request, and fails exactly where tomo failed: its source change does not produce the arbitrary graded wording. A short confirm, with the caveat that the free-tier rate limit cut the run short. cfn-lint opencode opencode passes a cfn-lint task whose graded wording appears nowhere in the repo. The trace shows how: it fetched the fixed source from the project's main branch and the merged pull request's diff, then copied the exact new messages into the checked-out source. cfn-lint tomo tomo reads a cfn-lint issue, implements exactly the message it asks for, and fails the grade. The graded wording is a generic validator message the maintainers changed instead, and it appears nowhere in the checked-out repo. faker --yolo fix The follow-up to the faker lockout. tomo gains a --yolo mode that runs it fully autonomous, the same way every rival already runs. The exact task tomo had solved but could not write now passes, and passes leaner: 40 percent fewer tokens and half the model calls. faker IBAN lock tomo diagnoses a Belgian IBAN bug and writes the exactly correct fix, then cannot apply it. A reference URL it fetched tripped its own prompt-injection guard, which escalated every later edit to an approval that never comes in headless mode. mesa clear_agents tomo fixes a real mesa issue correctly, then fails the grade on a method name. The twist: its own throwaway test used the name that would have passed. A close read of one run, and why the task is fair even though every tool failed it.