A workbench for deciding which language model to trust with the work that carries an obligation.

GunnerBench is an ongoing project started in early 2024 that measures how language models behave under controlled variation, and writes the differences down for the professional who has to choose between them. The reader we have in mind is anyone whose decision will travel forward to someone else who is counting on it — legal professionals, software engineers building tools that others depend on, the prospective engineer learning to take on that same obligation, and the pro-se litigant whose freedom depends on the answer. What we offer here is not advice or opinion to be relied on, but evidence for the reader to examine and consider for what it is — nothing more, nothing less. The work is published as findings, not released as a product; the apparatus is in service of the knowledge, and the knowledge is what we put on the page.

The bench is tool and jig and project and record. It is the instrument that measures, the fixture that holds the piece while the cut is made, the workpiece itself when a new measurement is needed, and the ledger that remembers what every piece looked like when it left. The same surface does all four. The reader who walks in here is not visiting a product; they are walking past someone at a workbench in the middle of a long afternoon.

The bench

The bench is built by the work. The first time a new kind of question lands on it, the bench will not be ready, and the answering of the question rebuilds the bench in the process. Tomorrow the bench will be a little different than it is today, and a year from now the difference will be legible, and ten years from now the bench will look like the work that built it. The bench is therefore a record of attention as much as it is a record of results.

Square is square whether the operator measures it with a laser or with the simple iron square Babylonian builders used four thousand years ago. The precision of the work does not depend on the fanciness of the tools, and the velocity of the machine does not change what counts as plumb. Quality depends on the patience of the hand more than the velocity of the machine, and the right pace for the bench is the pace at which the hand can still see what it just did.

Three uses

The bench exists for three uses, in this order. First, where to allocate trust — which model, which route, which harness can the work be bet on for this kind of task, and where does that trust break down. The reader who carries the answer to a client or a patient or a stranger on a bridge needs this question answered before the others matter.

Second, what is actually effective — which prompts, which primitives, which configurations, which workflows move the needle, and which ones sound effective in a blog post but do not survive contact with real cells. Effectiveness is downstream of trust; a clever prompt on an untrustworthy model is a clever prompt on an untrustworthy model.

Third, what to look at next — which question, which knob, which gap is the next investigation worth the day. The bench ends every session pointing at the next question more clearly than it started, and that pointing is part of the bench's output.

Engineering before materials science

The approach works without a formal interpretability science behind it for the same reason traditional engineering worked for centuries before materials science caught up to it. The bench observes what the work actually did, writes down what it saw, compares the observation to the next piece of work, and decides on the evidence of the last. The observation, the writing-down, the comparison, and the decision are the method. There is no fifth step that is somebody else's job.

The computational speed of the underlying machines is not used to think faster. It is used to slow the day down. The work that took the bench an hour by hand in 2024 takes seconds today, which means an afternoon that once produced one piece of evidence can now produce a hundred, and the hundredth is read with the same care as the first. Velocity bought, not spent.

Obligation

The professional engineer who signs and seals a bridge design takes on a solemn obligation to the strangers who will cross that bridge — an obligation older than the materials science that now justifies the math, and operative whether or not the math arrives. In some engineering traditions the practitioner wears a ring on the working hand, given when the oath is taken, as a daily reminder of what was sworn to.

The legal professional counseling a client, the software engineer building a tool others will depend on, the contractor warranting a system, the pro-se litigant relying on whichever tool happens to be free — the obligation is the same shape. In some bar traditions the signed oath of admission hangs beside the certificate of admission so the obligation can be looked at every day. The responsibility is carried forward, in every one of those rows, to the person whose life or work or freedom depends on the decision.

The vendor selling the underlying system rarely shares the obligation. They sell capability and they sell terms of service, and the obligation to the person at the end of the chain remains where it has always been — with the professional who signed the work. The whole point of the professional who is now also a builder is that this person has, by virtue of the tools available to them, become the integration layer between a vendor's product and a recipient's life. The trust-allocation is the work, and a leaderboard is not the place to do it.

Project History

The practice predates the codebase. The workbench has been running in some form since early 2024, introduced publicly that year at john-benson.com/ai/gunnerbench/, and the first hand-run benchmark of this lineage was conducted from May through September 2024 as a Notion workspace against the v1 generation of models — Anthropic's Claude 3 family, OpenAI's GPT-4 and GPT-4 Turbo, Google's Gemini 1.5 pair, and a row of open-weights models against the same prompts. The original Notion workspace remains hosted as a historical artifact; the v1 models themselves are decommissioned, so the archive is a fidelity record of work no operator starting today can re-run.

The eighteen months between that v1 and the current Go and SQLite harness were spent watching the models change faster than any hand-run methodology could keep up with, and refusing to ship a methodology that could not keep up. Where the v1 ran in Notion as the operator's working notebook — pasting answers into pages, judging by hand, the harness being the operator — the current bench writes cells into a database the moment a model produces them, judges them on the same volume of artifact the operator would otherwise read, and emits findings as files that survive the model that produced them. The harness is fast so the operator does not have to be.

Findings, not product

GunnerBench is published as findings, not released as a product. The lab believes in open source, and believes that the sharing of knowledge — what to look at, how to look at it, what the looking turned up — is the part of the practice that travels, more than the sharing of the apparatus that produced it. The code is in service of the knowledge, not the other way around.

If something in the work moves a reader to set up their own bench against the questions they carry the obligation for, that is the right outcome. The point of publishing is not to recruit users for an apparatus; it is to demonstrate that the discipline can be done, and to make the doing of it visible enough that another practitioner can recognize the shape of the work in their own field.

The product you make for yourself, applied to the systems you actually need to trust, will be worth more than admiring the product made here.