The harness can improve without touching the model

There is a pull request open in my public repo that I didn't write. An AI model wrote it, after finding a real bug in the coding agent this series has been building, and the fix never touches the agent's model: it changes one number in the harness config. It passed every test I have, and it still hasn't merged, because the pipeline that produced it was built to stop and wait for me. This post walks that pipeline end to end.

The change itself is small enough to state in full: a truncation limit in a config file went from 4,000 characters to 12,000. That edit took a task my agent failed on every attempt and made it pass on every attempt. Everything interesting lives around that number: how the bug got found, how the fix got proposed, and how it got proven safe before anything trusted it.

In the harness engineering post, harness evolution showed up as the last face of the tenth primitive: failures becoming infrastructure. It got one section there. It deserves the full walk, because the label people reach for, self-improving AI, gets slapped on everything from models that retrain themselves to workflows that tune their own prompts. The version that matters most if you’re a builder is a loop that improves the harness and leaves the model alone: the agent gets better on your schedule instead of the model provider’s, and every improvement is a diff you can read, test, and revert.

For anyone who hasn’t followed the series: the running agent is Gemma, a small quantized model on my own hardware, wrapped in a harness we built by hand over fifteen chapters. The weights never change. The question this experiment asks is whether something else can improve that hand-built harness, safely, without me editing it myself.

One concrete failure anchors all of it. I ask the agent a question whose answer sits near the end of a long log file, and it fails, every single time.

Where this sits on the ladder

Lilian Weng calls this harness engineering, the same term this channel runs on, and her argument is that the harness is where self-improvement actually lives, rather than in a bigger model. She slices the loop into five levels, safest at the bottom: improve the prompt, improve the context, improve the workflow, improve the harness code, and improve the thing that does the improving. Each level up is a bigger, riskier kind of change.

I’ve spent time at the bottom of that ladder already. AGENTS.md needs a feedback loop was about the instructions file accumulating lessons from real sessions, which is level one and two work. This experiment climbs to level four: an external system editing harness code and config. The safety story changes at each rung, so it’s worth knowing which one you’re standing on.

The vocabulary

Two papers gave this build its structure. AReaL, from Ant Group, HKUST, and Tsinghua, is about self-evolving agents at enterprise scale and supplies the framing: which kind of fix a failure deserves, why traces matter, and why rollback has to be a normal operation. Self-Harness, from Shanghai AI Lab, is about harnesses that improve themselves and supplies most of the mechanism. The terms below belong to them unless I say otherwise.

Weakness mining is the Self-Harness paper’s phrase for step one: run your agent, let it fail, then group only the failures that share one provable cause. A flaky one-off doesn’t qualify. My log-file failure did: every occurrence described what went wrong in nearly the same sentence. A group like that is a cluster, and the cluster is what the rest of the pipeline works on.

Routing comes from AReaL. A missing plain fact should become a memory entry, the cheapest fix available. A failure in how the harness routes a tool call, reads back a result, or enforces a guardrail should become a harness edit. Only a failure that persists across every task, user, and project points at the model itself. This experiment lives in the middle branch, and that classification got made from evidence rather than assumed.

Two terms are mine, because the things needed names. A suite needs miners, tasks you expect to fail because you already know the harness has a gap there, and regression guards, tasks you expect to pass whose whole job is to complain when a fix breaks something else. A suite made only of expected failures can’t catch a fix that helps one task while quietly costing another.

Held-in and held-out are the Self-Harness split: the fixer sees some of the affected tasks and the rest stay hidden, so you can tell whether a fix understood the problem or memorized it. Here, two tasks trip over the same log-file bug. The fixer saw one; the other stayed hidden.

Pass rates are fractions, because small quantized models aren’t consistent run to run. Every task runs three to five times, two passes out of three gets recorded as two thirds, and comparisons average the fractions. The methodology is the Self-Harness paper’s, and it adds up: each candidate fix gets measured with 49 separate runs against the live model.

The acceptance rule is the gate: a fix must be at least as good on both piles, seen and hidden, and strictly better on at least one. Worse anywhere means rejected, with no partial credit. AReaL adds two warnings that shaped this build: a self-evolving system that can’t explain what changed is drifting rather than evolving, and rollback should be a first-class operation, never a manual emergency.

Finally, the editable surface: before anything outside your agent can propose a change, the harness has to declare, in a short, named, versioned list, exactly what is allowed to change. Everything off that list stays untouchable.

Splitting the fixer from the patient

One deliberate departure from the paper. In Self-Harness, the same model diagnoses its own mistakes and proposes its own fix. Here I split that in two: Gemma stays completely frozen and only ever gets tested, while a second, stronger external model, Fable, does the diagnosing and proposing. Gemma is small enough that asking it to fix its own blind spots means asking it to see past something it can’t see past.

The split has a cost the original paper never had to price in, because its proposer and its patient were always the same model. A stronger model proposing fixes for a smaller one can overshoot: it can suggest something clever, wordy, and nuanced that a small quantized model can’t follow as well as the plain version it replaces. I call that a capability mismatch, and it hangs over everything that follows.

The harness had to declare itself

Before any of this could run, the harness had to change shape without changing behavior. Every knob that governed how it acted used to be scattered across five files as plain code. I collected all of it into one config file: 35 lines, eleven values, and a version number. The old code still reads from it, so the agent behaves exactly as before. What changed is that the harness can now answer, precisely, what is editable about it. That file is the entire editable surface.

The graders live somewhere else entirely. If the tests that score a fix sit anywhere near the thing being edited, the editor can cheat, not maliciously, just efficiently, because editing the test is easier than fixing the real problem. So the task suite and everything that grades a candidate live in a separate part of the project the editor never sees.

Every run gets recorded in full, which is AReaL’s argument: a pass or fail number by itself isn’t enough to learn from. Every tool call and reply is stored, stamped with the exact commit of the agent and the exact version of the config under test. That record is where the failing-run quotes in the next section come from.

The build itself followed the discipline the earlier chapters taught: separate AI sessions with self-contained briefs, one to synthesize the three sources, one with none of that context to review the work and the task suite adversarially, others to build the config file and the runner. The adversarial review found five real problems before a single live run. One test would have passed even if the agent did nothing at all. Another had nothing stopping a candidate fix from rewriting the answer key, the actual file the test checks against, instead of fixing the bug. The countermeasure is blunt: fingerprint every seeded test file the moment it’s written, and if the fingerprint ever changes, the task fails outright. I made the case in Your eval harness is the artifact that anti-gaming checks belong in the eval spec, citing reward-hacking rates other people measured; this was that concern showing up in my own project, found by review before the loop ever ran.

With the suite reviewed, I measured the harness as it stood. Thirteen hand-written tasks in four groups: losing information across a long session, claiming success without proof, leaking things that shouldn’t leak, and reaching for a tool instead of guessing. Every task is grounded in something this agent actually did earlier in the series. The baseline came back almost entirely green: ten tasks passed every try, and two failed every try. A2, the task the fixer would later see, failed three out of three. A4, its hidden partner, failed five out of five. Two red rows, one shared cause: the log-file failure.

The failure, up close

The code at fault is a six-line function. When something the agent reads is too big, the harness keeps the beginning up to a limit, drops everything after it, and leaves a small marker saying how much got cut. Simple on purpose, and it fails in exactly the way something that simple fails: if the part that matters sits past the limit, the model never sees a single token of it.

What the agent said in the failing runs is the best part of the diagnosis. “The log ends abruptly, with a truncated sequence.” “I cannot determine the password from the information provided.” It noticed the content was cut off and refused to guess. A model that lied confidently would have been much harder to debug; this one told me in plain language that the bug lives in the plumbing in front of it, and nowhere in the weights. That resolves AReaL’s routing fork: a tool result cut off before it ever reaches the agent is a harness problem, and neither a memory entry nor a model change would touch it.

Mining also did its filtering job. One other task came back flaky, passing most of the time with no pattern to the misses. It stayed out of the cluster, and nothing got proposed against it.

Three candidates

Fable looked at the cluster and proposed three fixes, and all three do the same kind of thing: raise one number. In its own reasoning it said, almost verbatim, that no instruction edit can recover data that never enters the window. That’s the correct diagnosis. The shape of the proposals is the editable surface working as designed: the config has no field for “change the strategy,” only values, so every possible fix was going to be a bigger number. Each proposal arrived with a written risk assessment, and the third one named, in advance, the exact task it might hurt.

Testing a candidate is mundane by design. The config file gets edited in the working copy of the repository, exactly as if a person had typed the change. The full thirteen-task suite runs against that edited copy. Then, whatever the result, the edit gets reverted. The run record shows the same commit as the baseline with one extra flag, dirty: true, plus a fingerprint of exactly what the dirt was. A rejected candidate never becomes a commit and never touches the history of the repository.

The very first dry run errored out, and the error was mine. One of my test files was comparing against the live, editable value of the very number being changed, so any fix that raised it far enough to work would have crashed the test instead of passing it. The measuring stick was coupled to the thing it was measuring. I pinned the test to the value recorded the day it was written, and because that changed the measuring tool itself, the whole baseline had to be re-recorded.

The results, per candidate. Raising the limit to 8,000 helped one task and was a near miss on the real failure: the password line starts at character 8,690, and the candidate stopped reading at 8,000. Raising it to 12,000 fixed the cluster completely, including the hidden task the fixer never saw, and cost nothing anywhere else in the suite. The same model with the same weights that spent about nineteen seconds per failing run explaining what it couldn’t see now answers in under two, and the whole reply is the password.

The candidate the rule would have accepted

The third candidate broke something real. A task that passed every time started failing every time, because a side effect changed how the harness manages its memory of a long conversation, and it failed quietly: attempts ran for over a minute and came back with empty replies. The uncomfortable detail is what the numbers did next. Another task’s flaky pass rate happened to tick up by roughly the same amount on that same run, so the aggregate score barely moved. The acceptance rule, applied to the total alone, would have accepted this candidate. It got caught because I read every task’s numbers individually instead of trusting the sum. The Self-Harness paper discloses this limitation itself, but watching a genuine regression hide inside a flat aggregate is the strongest argument I have for per-task reporting over any single number.

The winning candidate gets the same scrutiny. Part of its measured improvement was the real fix, the failing task recovering completely. A smaller sliver was noise, that same flaky task drifting up on this particular run. Separating the two is what makes the number worth anything.

As for the capability mismatch I flagged earlier: it didn’t bite, but only because the winning fix was a number, and numbers survive the trip from a stronger proposer to a smaller patient in a way prose doesn’t. The other two candidates still show Fable guessing wrong, once about how big a number needed to be and once about a side effect it didn’t anticipate. The risk is real; this fix just wasn’t the shape that exposes it.

The pull request waits

The second candidate won, and the pipeline opened a real pull request against my public repository, on its own branch. The title states the change plainly: iteration one, max item chars, 4,000 to 12,000. It did not merge itself, and it won’t. AReaL has a phrase for the posture: safe, audited, staged deployment. Every change carries its own promotion path and gets reviewed before it ships.

It would have been easy to close the loop end to end, mining to merging, with no human in it. I left the gap on purpose. A fix that can explain itself with real per-task numbers in a pull request, and still waits for a person to read it, is a different kind of artifact from one that ships itself. Being able to roll a change back and choosing not to auto-ship it are the same instinct pointed in two directions.

Reviewing it is a few minutes of real reading, and the design exists to make those minutes sufficient: one PR per fix, a full per-task table instead of a single score. I check whether the stated reason matches the numbers underneath, the way the third candidate’s didn’t. I check whether the change is the smallest one that explains the improvement. And I check whether I would have proposed the same fix myself, and if not, why not.

What it cost, quietly

Even the clean win has a cost that doesn’t show up in the table. The hidden task helped decide which candidate won, because it was part of the score used to compare all three. The fixer never read it, but using it for selection leaks information about it into which fixes survive. Run the loop once and that’s nothing. Run it for many iterations and the hidden pile slowly stops being hidden, one accepted edit at a time. The mitigation is unglamorous: retire and replace held-out tasks as the loop keeps running.

The fix itself is real but narrow. It raised the size of the bucket, and the harness is no smarter about what to keep. A better fix was visible the whole time: keep the beginning and the end of a long result, the same trick this harness already uses when it compacts a long conversation. But that fix is code, and code sat outside the editable surface by design. That’s the edge of what this version can do.

So the next version writes itself as a list: widen the editable surface beyond values, so the loop could propose a strategy instead of a bigger number; give the flaky task a harder mining pass to learn whether a second cluster hides under the noise; and point tests at territory the agent hasn’t been caught failing in yet. The suite currently sits almost entirely green: every hidden task passes, the seen pile is at 95.8 percent, and that one flake is the only signal left.

The checklist

Everything above compresses into questions you can ask of any system that claims to improve itself. Was the failure real, and found by running the thing? Were failures grouped by a provable shared cause, with the flakes set aside? Was the fix tested against work the fixer never saw? Was it confined to a short, named, versioned list of what may change? Was it checked against the whole suite, not just the broken task? Would the process catch a fix that games its own test, and has it ever actually caught one? And does every change land somewhere a person can read it before it ships? A system that can’t answer most of those is drifting, in AReaL’s sense of the word, whatever the label on it says.

This catalog started with a map of ten primitives around a frozen model, and every post since has pulled one layer apart: memory, sandboxing, skills, context. The model never changed in any of them, and it didn’t change here either: the weights that failed the log task are the weights that pass it now. What changed is the harness, and for the first time in the series, I didn’t write the change.

The pull request is still open. Iteration two starts after I’ve read it the way I described, and that read is the one step in this loop I never plan to automate.