Rendered at 06:49:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
seizethecheese 14 hours ago [-]
The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
seizethecheese 14 hours ago [-]
Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
Edward40 14 hours ago [-]
Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task frontier-harness-eval@v1.1.0 dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, reducing repetitive friction by fixing skills and tools, that is also important.
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
seizethecheese 13 hours ago [-]
This doesn't answer the question at all.
numeri 7 hours ago [-]
unless they edited it in later, the link at the bottom lists which tasks they ran it on. It definitely doesn't answer why those tasks or why specifically 62 of them
Edward40 7 hours ago [-]
We just want to keep the category unified so we have tasks for every domain like ML, science and others, and also have a chance to see the divergence from tb2.1 to tb4.
dotancohen 13 hours ago [-]
It's still the most informative thing to come out of that account or the linked article.
jzemeocala 14 hours ago [-]
For real, i remember setting up advanced Linux configurations back before the 3.0 kernel and I'm pretty sure I had a higher failure rate.
The biggest difference is that I, as a human, get mentally exhausted. but modern AI (when given unlimited tokens and tools) is like if Stephen hawking had a meth habit and the tenacity of Sisyphus with a bad case of syphilis after being told he can be cured if he can get that boulder over the hill
And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"
dotancohen 13 hours ago [-]
That quote is outdated. My phone nags me now twice daily to update, and the update removes features I use so I'm careful to tread around it.
My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.
Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.
We've definitely passed the point where the derivative has gone negative.
Edward40 13 hours ago [-]
I'm also building a leaderboard in the meantime, which will release in a few days with more detailed harness version info and methodology.
seizethecheese 14 hours ago [-]
Confirmed. This is 17 TB4 tasks and 45 TB2.1 tasks. Why?
Edward40 14 hours ago [-]
We want to see how performance diverges between TB2.1 and TB4 while keeping costs manageable, since running the entire dataset is really expensive.
We kept the category unified so we have tasks for every domain like ML , science and others.
12 hours ago [-]
Edward40 11 hours ago [-]
[flagged]
onlyrealcuzzo 11 hours ago [-]
There's thousands of benchmarks, and nobody can agree on what the best models are if you run all of the benchmarks which costs tens of thousands of dollars for frontier models.
Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.
How are we supposed to derive value from this?
Edward40 10 hours ago [-]
Although people disagree with the best models, the worst model is obvious.
Edward40 13 hours ago [-]
Good finding! Model preference is interesting.
Edward40 14 hours ago [-]
Because these tasks are genuinely hard and require long-horizon reasoning, running full evaluation experiments gets expensive fast.
For example, on bun-sourcemap-leak (https://hub.harborframework.com/tasks/terminal-bench/bun-sou...), all 440 trials finished cleanly with zero infra errors or timeouts, yet 0 out of 440 passed all hidden-variant tests (the best was 35/36 across three DeepSeek V4.1 Flash runs).
Some of these tasks take up to 8 hours per run, so if you are not careful, you end up burning massive amounts of compute and API spend just to get an accurate evaluation signal.
warkdarrior 12 hours ago [-]
I wouldn't call it "burning of tokens and compute" if you get a useful signal out of it.
plufz 14 hours ago [-]
Yeah exactly, compared to what baseline? People here will probably be stunned when I say this, but my pre-agent code was usually not bug free on my first attempts either.
themgt 14 hours ago [-]
Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015). For the other three factors, it was −0.03 to +0.02 (p ≥ 0.88). Within each model, tool calls tracked cost as well as steps. Table 4 below and Table 5 in the appendix give the values.
For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.
Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!
Edward40 14 hours ago [-]
Cutting the system prompt and tools is easy. Cutting steps requires understanding the model's RL strategy and how it was trained to explore.
bisonbear 14 hours ago [-]
I wonder if Claude Code would do better given they just introduced code mode, which should significantly cut the number of steps required
adrithmetiqa 13 hours ago [-]
Agreed. Does this help anyone decide what model or harness to use.
Nope.
Edward40 12 hours ago [-]
[flagged]
Zigurd 15 hours ago [-]
I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.
nolok 14 hours ago [-]
I don't understand why that idea was adopted that IA had to be that "give one command have a perfect run and correct answer" thing when it's clearly not how it works and an eternal "the last % is worth 99%" issue.
I know why the US frontier labs push it of course, you can't push a trillion dollar plus idea if you also say it should be a cheap commodity and the best and latest are now the answer, a bazillion mid levels are.
But most work of value I do is with a swarm of "not the latest nor the brightest" models, with a smart one controlling the dance. Most decisions or review is better done through consensus. The myth that "if we keep going it will reach a point where it's always correct" seems so obvious to me, it doesn't work like that, for that matter we don't work like that.
This is not a jab at it. I work with it. I love it. But I think the story imperative are a mismatch for the actual needs of most companies, and it's hiding between the "frontier needs" of special research & co, or the governemental needs.
On the same line of thought: it's pretty obvious as great as these things are, no they're not going to "replace all employee", and no employee without them won't be able to compete either. They're a supercharger, a self multiplying smart assistant with capabilities in any field, and it feels so wrong to me that they've been presented in a complelty different way.
krrrh 14 hours ago [-]
> it doesn't work like that, for that matter we don't work like that.
The parallels with self-driving cars are interesting. We demand that agents don’t just produce better code than most humans (which the frontier models do), but that they perform much much much better.
Edward40 14 hours ago [-]
[flagged]
wat10000 14 hours ago [-]
I recently tried some personal evals to see how some different models performed. I was trying Opus, Sonnet, Gemini 3.8 (this was pre-Argon) and a couple of others. I picked a pull request I'd recently made with heavy Claude assistance, one that seemed simple enough to be quick to solve but with enough subtleties that I could differentiate the results. Set up the agents to run clean without knowledge of the solution, and set them loose.
To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.
My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.
Edward40 14 hours ago [-]
[dead]
contravariant 14 hours ago [-]
I think Conway's law applies. In an ideal situation your code mimics your organisational structure.
The places where you let an agent do its thing is bound by the (implicit) interface your prompt creates. In an ideal situation this interface is a simple isolated thing in your application.
If you just let it do its thing, expect a monolith. You shouldn't rely on it coming up with interfaces that mesh well with what you had in mind, and each of those interfaces represents a decision that it took, restricting future iterations of the code. So a monolith is not only what you can expect, but also the best you can hope for.
If you want to let it cook, create more places for code that does a thing and you don't care how.
dcss_gardener 12 hours ago [-]
You can steer it a lot with implicit context too, like ants following pheromones. If you put an empty docs/ dir in your repo agents will put documentation there without being told. If you put one inline test, they'll write every test inline. If you seed a decision record they'll record their decisions. If you have a lowercase sql query they'll use lowercase sql. If you write a test for wcag AA contrast they'll make ui with wcag AA contrast. And so on.
With some up front care you can create an environment where they mostly do what you want when you let them rip without oversight. As the codebase grows it becomes less likely they'll see any one example, and more rarely expressed ones can get missed. And then codebase predating LLMs or designed without this consideration you just have to accept what you get or wrangle them manually.
Ericson2314 14 hours ago [-]
BTW the usual point of Conway's structure is to try to adjust the org / who is working on the software, so you don't get a bad software architecture.
The "objective" nature of code is revealing a flaw in the human division of labor.
contravariant 10 hours ago [-]
I think the law is more powerful if you view it as a force of nature. You don't have to make your organisation conform to the shape of the software, you can often do it the other way around.
The trouble starts when the requirements for your software are at odds with the requirements of your organisational structure.
pjc50 14 hours ago [-]
Oh that's a good insight. The agent counts as part of the "organization" for this purpose.
Edward40 12 hours ago [-]
[flagged]
sl0wbrainRot 11 hours ago [-]
Keeping them on a short leash is leetcoding. Handing them one recursive function at a time
Devs are the PMs structuring the sprints now cause they know when/where/how to pivot
See discourse wrt who Nvy should pick as ceo (if JH doesn't come back anyway); at least from what I seen the theme is if they are not an engineer who can keep up with tech and know why Nvidia needs to change course Nvidia is toast
Same with software; everywhere I worked PMs that were not engineers sucked to be at as the PMs Duning-Kruger called the shots not the demands of technical problems before the devs
I appeal to fellow engineers too; consider in this transition into poly-job expertise phase to explore and apply them edge case solving skills to the domain of law.
4+ million devs in the US. 1.3 million lawyers. Just 1/3 of devs pivoting would act like a massive forcing function
And local/hosted AI assistants can flatten the learning curve.
Been looking into pre-law courses. Scoring a 71 of 100 on a mock bar exam without studying made me realize that stuff is high school level logic. So imma keep going for now. See where it leads.
Zigurd 11 hours ago [-]
> Devs are the PMs structuring the sprints now cause they know when/where/how to pivot
You need not just the PMs but the management structure on top of them to get with the idea that product development should be iterative and experimental, not spec driven. But yeah that's how things are going.
sl0wbrainRot 10 hours ago [-]
Look at all the small businesses where a sole proprietor and 1-2 others handle it after directly interfacing with their customers which number in the dozens
All those layers appeared in response to ZIRP to fuel job creation. They did not appear out of real necessity and iteration as you call out. You're blithely rattling off a startup spec sheet.
You need that perhaps, or believe you do from extensive prior experience.
2010s are gone; public opinion on such an economy has effectively turned the corner for good.
Same way devs complain about others getting into their flow, computer users at large are tired of a dev riding shotgun, acting like a helicopter parent. They don't want a subscription to you.
A forced obligation of your having reach into their lives while they have no such leverage feels invasive af. A few customers might diggit but overall the <50 crowd grew up with this stuff. The demand to help grama add a pop3 account to Outlook Express is as bygone as the cloud SaaS era
rrr_oh_man 15 hours ago [-]
> objectively nicer than I would've gotten from the effort I would've put into manually coding
I find that only works with topics I know very little about.
datsci_est_2015 14 hours ago [-]
This comment could be interpreted at least two ways, and I’m not sure which it is:
- tongue in cheek: in topics you know very little about, you don’t have the ability to tell whether the LLM output is trash or not
- very serious: LLMs allow you to create things you could not have created before due to not having the knowledge
Sort of like a Dunning-Kruger or Gell-Mann amnesia.
devmor 14 hours ago [-]
I don't see why both can't be true.
LLMs are allowing people to create things with very little effort compared to before, and those creations are of questionable quality - indeterminable by the LLM-user due to a lack of experience.
ygjb 14 hours ago [-]
So, basically, the same as before, but LLMs provide the creations of questionable quality at a faster rate :P
I think the quality of the results vary with the rigor of the process. If you one-shot prompt a new service (even with Fable5) "build me an airtable competitor, no bugs", you will get some real code camp quality results, but a well defined set of prompts and a workflow that leverages multiple coding agents reviewing each others work, and implementing test cases at a relatively granular level yields radically different results.
Couple that with a rigorous design/spec session, and multiple checkpoints (say at release milestone) to monitor drift from the original design makes the process a bit more intentional and measurable, while still being far faster than a typical engineer can build.
yugoslavia4ever 14 hours ago [-]
Yes, but before you learned as you went, researching and finding out what good (or at least better) looked like; what learning is done when you wave your hand?
ygjb 3 hours ago [-]
Honestly? I have more time to learn now than I did before. I can also do something that was really challenging before LLMs as well, which is turn my lack of comprehension into an demand, guided tour through a Socratic exploration of the unknown topic, including generating labs specific to the skills I want to develop and with the resources I have available.
It's like anything else with LLMs, if you treat it like a magic wand, you haven't really learned or built anything, but using it as a true assistant, it can be a powerful and flexible tool.
embedding-shape 15 hours ago [-]
Not every problem should (or even could sometimes) be solved the same way. Are you planning to maintain the produced code for more than a year? No, then wave away. But if you're expecting others to depend on it, and you want to be able to continue evolving what it is, then a tight leash and full understanding of how it actually works, is very much needed, even if you don't actually type characters into an editor anymore.
Otherwise you still end up with balls of spaghetti where each change takes longer and longer to work into the codebase. With a bit of planning, you can keep tackling changes and they keep taking the same amount of time more or less.
smallnamespace 14 hours ago [-]
The cost of AI is rapidly falling while its engineering capabilities keep rising. This implies that code is becoming more fungible, and you should expect any code you write today to mostly be replaced in the near future.
The requirements will be more stable than the code. Focus on them instead.
datsci_est_2015 14 hours ago [-]
Robust systems experience very little turnover in nearly every non-software discipline that involves systems. Why the hell would we insert risk into our system-building processes by needlessly turning over code?
smallnamespace 5 hours ago [-]
Because the code is an artifact that represents one particular way to accomplish a goal. There are always better ways to accomplish goals.
It’s important to separate the requirements and the metrics and tests that show your requirements are met from the implementation itself. But if your metrics and tests are robust, then it is safe to pour compute into searching for better and better implementations, subject to budget constraints.
Zigurd 14 hours ago [-]
> The requirements will be more stable than the code.
These kinds of changes to the way teams are formed and managed, and how products are developed is what really fascinates me about the impact of coding agents.
Even if it works, for some definition of works, to dump a huge spec into a coding agent context window, I suspect this is where a lot of horror stories about agentic coding come from.
Teams, and the management structure above them, are going to have to get used to more collaborative and iterative product definition and development in order to extract the most value from coding agents.
catdog 14 hours ago [-]
> The cost of AI is rapidly falling while its engineering capabilities keep rising.
Wouldn't count on it. There likely is a ceiling, we might or might not be near it.
reddit_clone 13 hours ago [-]
Definitely with the cost. I strongly believe the price now is heavily subsidized and will shoot up in the future. (Not new. First hit is free..)
wtch98 14 hours ago [-]
> The cost of AI is rapidly falling while its engineering capabilities keep rising. This implies that code is becoming more fungible, and you should expect any code you write today to mostly be replaced in the near future.
I have code I wrote in 2006 which is still in daily use adding business value at my company. Why would it need replacing?
smallnamespace 5 hours ago [-]
One reason is because a lot of older code is insecure. Another reason is that if you can find a better way to accomplish the goal, then you should eventually switch to it.
Technically, the FAA sees positive business value from using floppy disks over magnetic tape or punch cards.
But that’s not the right metric, since it ignores the fact that floppy disks are worse than a modern alternative.
If your code from 2006 is already perfect and there is no room for improvement, then there never be a need to replace it. But this doesn’t describe most software.
There is one more thing to mention, which is that any code from 2006 that you’re running has survived the quality filter of time. Most code from 2006 isn’t in use today. I think even a smaller proportion of code written today will be in use 20 years later.
tempfile 14 hours ago [-]
> This implies that code is becoming more fungible
Code has always been fungible, in the sense that it could be refactored.
> you should expect any code you write today to mostly be replaced in the near future
Why on earth do you think this? It does not follow from your other points. Just because you could turn your business into a rube goldberg code juggling machine, why would you?
skydhash 13 hours ago [-]
it's amazing that you could read a book like "The Design and Implementation of the 4.4 BSD Operating Systems" (1996) and still find it relevant to browse the code of OpenBSD and NetBSD, three decades later. While good software gets plenty of updates and additions, the core ideas barely bulge over time. A lot of churn is people playing around with no reason instead of solving the problem properly the first time.
alala-lala-lon 13 hours ago [-]
[dead]
krrrh 14 hours ago [-]
A shift is realizing that refactors go from something that was costly and infrequent to something that can happen inexpensively and frequently. It takes some rebalancing.
I recently used agentic processes to bring a young codebase to a functioning state very rapidly. This forced discussions about design principles that were folded back into a short document in the repo which then led to a major refactoring plan, and a massive pruning of code that was written the week prior. There are solid engineering approaches in the codebase now that wouldn’t have been achieved without this discovery process.
jwilliams 9 hours ago [-]
[flagged]
Edward40 14 hours ago [-]
[flagged]
xyzzy_plugh 13 hours ago [-]
I don't like this framing.
For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.
Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.
This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.
But I can't build a mobile app with this harness, or make a movie.
These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.
Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.
Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
botanrice 13 hours ago [-]
hey! would you be so kind as to share your method or point me in the direction towards resources on how to setup a harness as you describe?
I've been tinkering with harnesses recently but have mostly used github copilot on vscode mostly thus far as my ai tool and feel like the harness setup on there is lacking (or really confusing...). many thanks if so :)
just trying to learn without googling "how to setup a harness" and get bombarded by a trillion yt vids or competing harness providers lol
NichoPaolucci 8 hours ago [-]
I would also consider just grabbing pi and tailoring it to your needs. It's super direct, no batteries, and I added a handful of smaller tools that blow everything else out of the water.
I've got some people on my team using 80-100 skills / plugins and their setup takes FOREVER to run even a simple modification.
Mine is just as correct and the feedback loop is lightyears faster, which is what I'm personally after.
xyzzy_plugh 11 hours ago [-]
Just ask one to write the other.
"Write me a harness in $LANGUAGE that uses the $MODEL API and has tools for X Y and Z" etc etc
I added remote management and web console over tailscale in about 20 minutes last week.
vpanghal 9 hours ago [-]
[flagged]
Edward40 13 hours ago [-]
I am thinking about how to measure the efficiency of ultilzing existing context in one specific domain.
lhk931122 2 hours ago [-]
But I feel like later on the model itself will end up learning even the harness, so I wonder if there's any point in setting up a harness well on your own. I don't know if OpenAI or Anthropic built their harness well or upgraded their model well, but I ended up throwing away almost all of the harness I set up personally, and the time I spent on it became a waste of time. It seems like wasted effort, same as prompt engineering. From the point of view of each individual user. Since an already optimized system prompt is in place.
cpeth 10 hours ago [-]
Harnesses are all fairly simple loops from the ones I have looked at; there is not much to them.
As someone who regularly uses Claude Opus 5.5 from Claude Code and from Copilot, I would say harness barely matters in terms overall performance and behavior. Opus is nearly identical in both harness. The opposite is also true, if I switch Copilot to something like Gemini 4 Argon it's completely different even though I'm in the same harness.
Just some andecdata.
shepherdjerred 5 hours ago [-]
The harness is incredibly important when you're working in parallel with multiple agents. In this regard Claude Code is the best I've found.
drivebyhooting 9 hours ago [-]
Doesn’t the harness include a bunch of prompts for when to create subagents, their communication patterns, and how to verify their outputs?
That can make a huge difference, as evidenced by the agent swarms success at cracking maths.
chickensong 8 hours ago [-]
The harness is what you make it, or not. The OP claiming all harnesses are simple loops is looking at it from a simple perspective. As soon as you start customizing or building a bespoke harness, YMMV.
Edward40 6 hours ago [-]
I don't mean the harness, just the loop. Memory could be a part of the harness.
chickensong 6 hours ago [-]
Sorry, I misused OP there, was referring to @cpeth in this comment thread.
Thanks for your eval work, it seems like a tough area to measure!
0xcoffee 52 minutes ago [-]
Please also benchmark Sol 6.2 medium. It's one of the most used combinations, most people are not running Astra.
bisonbear 14 hours ago [-]
These are cool results and I'm glad that tokens are being invested to eval harness capabilities.
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
"Cost per pass" is distinct from "cost per successful task."
Cost per pass is simply the average cost per task divided by the pass rate.
If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.
bisonbear 14 hours ago [-]
In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.
To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode
Edward40 12 hours ago [-]
We use cost per pass, not average cost per task, because we want to see the harness keep a high pass rate and low cost at the same time.
krawczstef 3 hours ago [-]
There's no IP in a harness, and they don't matter that much.
Harnesses simply make different choices around certain context engineering things, what they name tools, how they compact, their system prompt, etc.
So it's not surprising that they work "better" for some models than others, but I'd bet that if you made each harness have same system prompt and tool names, the variance would shrink further...
So I would say know what you're optimizing for and construct the harness you need for it instead (it's just a while loop with an llm call and function calling after all) ...
ford 14 hours ago [-]
To be clear - "Failed" here means "Did not meet all pass criteria on a given benchmark task", not "Experienced a runtime failure"
scubazealous 14 hours ago [-]
While this could be better, I wonder how much time the average developer spends on runs that fail? How much time do students spend on runs that fail?
fennecfoxy 14 hours ago [-]
Yeah, developers sometimes go down the wrong path as well. It's a non deterministic system exploring a huge problem space...
bityard 14 hours ago [-]
This is a good point. I'm (so far) only a hobbyist developer and _most_ of my non-trivial projects took at least 2 prototypes before getting to something that was fully usable with few caveats or technical debt.
I think this is not too different from physical products, either. You often hear of companies spending multiple years and iterating through dozens of prototypes in order to bring something new to the market.
I don't think there is a shorter path: you never truly understand a problem until you take up the engineering work to solve it.
hi_hi 8 hours ago [-]
I see cost and speed being measured, but not intelligence. Is there a measure of how these harnesses impact the effectiveness of the LLMs?
scosman 11 hours ago [-]
We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
reversengineer 9 hours ago [-]
Is it me, or do the results seem to show a lack of significant difference between both models? And, because of the potential non-determinism in Generative AI to create a huge degree of variance in outcomes, I am dubious about the repeatability of these results.
Edward40 9 hours ago [-]
We ran 4 attempts per task, that's just like when you roll dice 4 times.
evalstate 14 hours ago [-]
Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/
GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
zodwick 10 hours ago [-]
I wonder if there is corelative improvement between the same lab's own harness and model.
For example I remember Codex having a different line number format than CC and every other harness had to have a two format tool to maximise performance.
I wonder if sucha thing exists still or do they RL on all possible combinations now
franzb 10 hours ago [-]
Original title: "In 27,280 coding agent runs, 64% of inference spend went to runs that failed"
cronin101 14 hours ago [-]
I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
bisonbear 14 hours ago [-]
In MiMo's v2.6 public RL training run, they explicitly trained on 23 different harnesses in order to minimize overfitting on any single harness. I imagine that this technique has to be common in RL training runs given that model providers have little control over where their models are used.
It's typically easier to tune the harness around the models' existing behavior. In my experience the primary failures in models > 5B params on coding tasks are just that they aren't able to translate instructions into a correct description or implementation of the problem (will write wrong code, wrong tests, and wrong documentation). The basic tool calling and syntax of generated code is fine.
ford 14 hours ago [-]
I don't think this is common, though it would be interesting if ThinkingMachines/HuggingFace/OpenRouter/BaseTen offered fine-tuned variants of the most popular open models that worked with specific harnesses
However, the hardware requirements are still quite enormous.
I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.
jrmg 14 hours ago [-]
What does "last-mile fine-tuning on [a] harness" mean?
cronin101 14 hours ago [-]
Taking the open weight model and altering those weights in order to overfit for success on the tool shape / quirks of the harness you are running the model within.
As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.
Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.
I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.
elAhmo 12 hours ago [-]
So, on my daily coding tasks choice of provider, model family, model version, effort/thinking level and harness all have an effect. Some of these are multiplicative (like harness), so this is all extremely confusing for the end user about what the hell should I use do do my tasks.
alok-g 10 hours ago [-]
Genuinely asking: Why is Kilo code (https://kilo.ai) not present in this and most such analysis? It does not also get talked about much on HN in general.
Edward40 8 hours ago [-]
Kilo does not seem that popular. Point it out if I am wrong.
etherlink 7 hours ago [-]
[dead]
Majromax 15 hours ago [-]
Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps.
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
Edward40 14 hours ago [-]
[flagged]
augment_me 14 hours ago [-]
This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context.
If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
Edward40 14 hours ago [-]
[dead]
SAI_Peregrinus 13 hours ago [-]
When a task passes all tests, it's done. More tokens should be spent on failing runs, because most runs where tokens are used should fail. For the same reason that your lost keys are always in the last place you look for them.
samueloph 13 hours ago [-]
Am I'm missing something or at this point the distinction between "harness" and "agent" doesn't exist anymore because too many people use them interchangeably?
Is this post mixing them up or is my understanding of these concepts off?
layer8 13 hours ago [-]
harness + model = agent (plus sub-agents)
Do you have a different understanding?
Edward40 12 hours ago [-]
[dead]
jimbob45 2 hours ago [-]
I’m struggling to understand the difference between “harness” and LangChain, which seems to have already been a failure.
ogmutlu 13 hours ago [-]
They mean the same thing?
bob1029 14 hours ago [-]
The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.
pitabas106 13 hours ago [-]
[flagged]
colinmarc 14 hours ago [-]
How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.
Edward40 14 hours ago [-]
[dead]
bensyverson 12 hours ago [-]
This is interesting and valuable, but I'd love to see a follow-up that tests a more parallel or subagent-heavy workflow.
anigbrowl 1 hours ago [-]
We ran the Harbor evaluation with each harness’s default settings, because we think this is closest to how people use a harness every day.
Programmers, famous for not obsessively configuring/customizing their tools
jrmg 14 hours ago [-]
"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title.
The page's title is: "Introducing FrontierHarness Eval v1.1"
Edward40 14 hours ago [-]
[dead]
andai 14 hours ago [-]
>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
d-sanoj 11 hours ago [-]
I think the harness matters less when we are measuring the work done. There are all kinds of benchmarks in differ cases showing one model is better than the other. The existing ones like Claude Code or codex work well with their models.
Edward40 8 hours ago [-]
I think Claude Code does not care about token efficiency, the more users use, the more users pay.
You do need to optimise prompts too, eg I found that openai models just gave up in Pi, but adding a line to the prompt matching Codex eg "Once implementation is authorized, the agent should persist through the bounded change, tests, review and final validation. Complexity, expected duration and context compaction alone are not reasons to stop." fixes that. Otherwise it would look at tasks and say they were too hard.
winddude 12 hours ago [-]
Reading a lot of the comments everyone is talking about state actors, etc.
Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.
BugsJustFindMe 13 hours ago [-]
This is nice and all, but I really want to see the best harness for the actual frontier models ChatGPT and Claude, not just open weight models.
maerF0x0 14 hours ago [-]
So? I'm sure exceedingly high amount of EC2 cpus are idling, or lost to discarded branch predictions etc.
We waste whatever is cheap to gain what is dear.
(Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)
worik 12 hours ago [-]
Given the variation between results for different, but similar capability models it is hard to take this seriously
0xbadcafebee 13 hours ago [-]
The choice of agent/harness is, as people frequently point out, just one of many variables that changes agent performance. Skills, MCPs, Extensions, system prompt additions, often have drastic effects, making things better and/or worse. It's interesting to compare them side by side, but probably won't reflect your own unique experience.
Second thing to consider: ZCode was sending everyone's code to Alibaba... not sure I would use their harness just to save a few cents on coding.
jedisct1 14 hours ago [-]
It would have been nice to see swival.dev in that list.
But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.
shevy-java 14 hours ago [-]
Well, it is a milking system. You have to pay for those tokens anyway.
Edward40 14 hours ago [-]
[flagged]
brookritz 12 hours ago [-]
The TLDR saya:
Basesten, Harneas and Leaderboard and still have no clue.
underdeserver 14 hours ago [-]
Wait till you hear what the human success rate used to be.
aleqs 12 hours ago [-]
Another bullshit result obviously meant to promote a specific company/product. Codex is garbage btw.
guanlan 16 hours ago [-]
Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found:
The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3.
Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost.
The model still sets most of the bill (93% of cost variation). The harness is the second lever.
Starting prompt size doesn’t predict cost. Output tokens do.
64% of inference spend went to runs that didn’t pass.
radlad 15 hours ago [-]
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
This is why for open-weights models at least harness aware post-training is becoming a thing. Also closed source harnesses that aren't fully editable are basically an anti-pattern. The labs should just hurry up and productionize self-evolving harnesses, and have the models internalize harness engineering, so they can optimise the harness for the given task. These are solvable problems.
xiaoliangliang 3 hours ago [-]
[flagged]
apishare_api 7 hours ago [-]
[flagged]
johnrc 12 hours ago [-]
[flagged]
Viacol 5 hours ago [-]
[dead]
11 hours ago [-]
amarinhodev 10 hours ago [-]
[dead]
shiqimei 16 hours ago [-]
[dead]
NamlchakKhandro 6 hours ago [-]
[dead]
Olscore 12 hours ago [-]
[flagged]
markasoftware 11 hours ago [-]
From the Hacker News Guidelines [1]:
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
Your post history is almost entirely promotion of your product. Please stop. I'm flagging your post.
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
https://hub.harborframework.com/datasets/frontier-harness-ev...
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
The biggest difference is that I, as a human, get mentally exhausted. but modern AI (when given unlimited tokens and tools) is like if Stephen hawking had a meth habit and the tenacity of Sisyphus with a bad case of syphilis after being told he can be cured if he can get that boulder over the hill
And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"
My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.
Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.
We've definitely passed the point where the derivative has gone negative.
We kept the category unified so we have tasks for every domain like ML , science and others.
Then, somebody runs a small subset of benchmarks - with not enough data to have any statistical significance - and makes further claims.
How are we supposed to derive value from this?
For example, on bun-sourcemap-leak (https://hub.harborframework.com/tasks/terminal-bench/bun-sou...), all 440 trials finished cleanly with zero infra errors or timeouts, yet 0 out of 440 passed all hidden-variant tests (the best was 35/36 across three DeepSeek V4.1 Flash runs).
Some of these tasks take up to 8 hours per run, so if you are not careful, you end up burning massive amounts of compute and API spend just to get an accurate evaluation signal.
For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.
Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!
I know why the US frontier labs push it of course, you can't push a trillion dollar plus idea if you also say it should be a cheap commodity and the best and latest are now the answer, a bazillion mid levels are.
But most work of value I do is with a swarm of "not the latest nor the brightest" models, with a smart one controlling the dance. Most decisions or review is better done through consensus. The myth that "if we keep going it will reach a point where it's always correct" seems so obvious to me, it doesn't work like that, for that matter we don't work like that.
This is not a jab at it. I work with it. I love it. But I think the story imperative are a mismatch for the actual needs of most companies, and it's hiding between the "frontier needs" of special research & co, or the governemental needs.
On the same line of thought: it's pretty obvious as great as these things are, no they're not going to "replace all employee", and no employee without them won't be able to compete either. They're a supercharger, a self multiplying smart assistant with capabilities in any field, and it feels so wrong to me that they've been presented in a complelty different way.
The parallels with self-driving cars are interesting. We demand that agents don’t just produce better code than most humans (which the frontier models do), but that they perform much much much better.
To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.
My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.
The places where you let an agent do its thing is bound by the (implicit) interface your prompt creates. In an ideal situation this interface is a simple isolated thing in your application.
If you just let it do its thing, expect a monolith. You shouldn't rely on it coming up with interfaces that mesh well with what you had in mind, and each of those interfaces represents a decision that it took, restricting future iterations of the code. So a monolith is not only what you can expect, but also the best you can hope for.
If you want to let it cook, create more places for code that does a thing and you don't care how.
With some up front care you can create an environment where they mostly do what you want when you let them rip without oversight. As the codebase grows it becomes less likely they'll see any one example, and more rarely expressed ones can get missed. And then codebase predating LLMs or designed without this consideration you just have to accept what you get or wrangle them manually.
The "objective" nature of code is revealing a flaw in the human division of labor.
The trouble starts when the requirements for your software are at odds with the requirements of your organisational structure.
Devs are the PMs structuring the sprints now cause they know when/where/how to pivot
See discourse wrt who Nvy should pick as ceo (if JH doesn't come back anyway); at least from what I seen the theme is if they are not an engineer who can keep up with tech and know why Nvidia needs to change course Nvidia is toast
Same with software; everywhere I worked PMs that were not engineers sucked to be at as the PMs Duning-Kruger called the shots not the demands of technical problems before the devs
I appeal to fellow engineers too; consider in this transition into poly-job expertise phase to explore and apply them edge case solving skills to the domain of law.
4+ million devs in the US. 1.3 million lawyers. Just 1/3 of devs pivoting would act like a massive forcing function
And local/hosted AI assistants can flatten the learning curve.
Been looking into pre-law courses. Scoring a 71 of 100 on a mock bar exam without studying made me realize that stuff is high school level logic. So imma keep going for now. See where it leads.
You need not just the PMs but the management structure on top of them to get with the idea that product development should be iterative and experimental, not spec driven. But yeah that's how things are going.
All those layers appeared in response to ZIRP to fuel job creation. They did not appear out of real necessity and iteration as you call out. You're blithely rattling off a startup spec sheet.
You need that perhaps, or believe you do from extensive prior experience.
2010s are gone; public opinion on such an economy has effectively turned the corner for good.
Same way devs complain about others getting into their flow, computer users at large are tired of a dev riding shotgun, acting like a helicopter parent. They don't want a subscription to you.
A forced obligation of your having reach into their lives while they have no such leverage feels invasive af. A few customers might diggit but overall the <50 crowd grew up with this stuff. The demand to help grama add a pop3 account to Outlook Express is as bygone as the cloud SaaS era
I find that only works with topics I know very little about.
LLMs are allowing people to create things with very little effort compared to before, and those creations are of questionable quality - indeterminable by the LLM-user due to a lack of experience.
I think the quality of the results vary with the rigor of the process. If you one-shot prompt a new service (even with Fable5) "build me an airtable competitor, no bugs", you will get some real code camp quality results, but a well defined set of prompts and a workflow that leverages multiple coding agents reviewing each others work, and implementing test cases at a relatively granular level yields radically different results.
Couple that with a rigorous design/spec session, and multiple checkpoints (say at release milestone) to monitor drift from the original design makes the process a bit more intentional and measurable, while still being far faster than a typical engineer can build.
It's like anything else with LLMs, if you treat it like a magic wand, you haven't really learned or built anything, but using it as a true assistant, it can be a powerful and flexible tool.
Otherwise you still end up with balls of spaghetti where each change takes longer and longer to work into the codebase. With a bit of planning, you can keep tackling changes and they keep taking the same amount of time more or less.
The requirements will be more stable than the code. Focus on them instead.
It’s important to separate the requirements and the metrics and tests that show your requirements are met from the implementation itself. But if your metrics and tests are robust, then it is safe to pour compute into searching for better and better implementations, subject to budget constraints.
These kinds of changes to the way teams are formed and managed, and how products are developed is what really fascinates me about the impact of coding agents.
Even if it works, for some definition of works, to dump a huge spec into a coding agent context window, I suspect this is where a lot of horror stories about agentic coding come from.
Teams, and the management structure above them, are going to have to get used to more collaborative and iterative product definition and development in order to extract the most value from coding agents.
Wouldn't count on it. There likely is a ceiling, we might or might not be near it.
I have code I wrote in 2006 which is still in daily use adding business value at my company. Why would it need replacing?
Technically, the FAA sees positive business value from using floppy disks over magnetic tape or punch cards.
But that’s not the right metric, since it ignores the fact that floppy disks are worse than a modern alternative.
If your code from 2006 is already perfect and there is no room for improvement, then there never be a need to replace it. But this doesn’t describe most software.
There is one more thing to mention, which is that any code from 2006 that you’re running has survived the quality filter of time. Most code from 2006 isn’t in use today. I think even a smaller proportion of code written today will be in use 20 years later.
Code has always been fungible, in the sense that it could be refactored.
> you should expect any code you write today to mostly be replaced in the near future
Why on earth do you think this? It does not follow from your other points. Just because you could turn your business into a rube goldberg code juggling machine, why would you?
I recently used agentic processes to bring a young codebase to a functioning state very rapidly. This forced discussions about design principles that were folded back into a short document in the repo which then led to a major refactoring plan, and a massive pruning of code that was written the week prior. There are solid engineering approaches in the codebase now that wouldn’t have been achieved without this discovery process.
For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.
Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.
This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.
But I can't build a mobile app with this harness, or make a movie.
These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.
Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.
Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
I've been tinkering with harnesses recently but have mostly used github copilot on vscode mostly thus far as my ai tool and feel like the harness setup on there is lacking (or really confusing...). many thanks if so :)
just trying to learn without googling "how to setup a harness" and get bombarded by a trillion yt vids or competing harness providers lol
I've got some people on my team using 80-100 skills / plugins and their setup takes FOREVER to run even a simple modification.
Mine is just as correct and the feedback loop is lightyears faster, which is what I'm personally after.
"Write me a harness in $LANGUAGE that uses the $MODEL API and has tools for X Y and Z" etc etc
I added remote management and web console over tailscale in about 20 minutes last week.
As someone who regularly uses Claude Opus 5.5 from Claude Code and from Copilot, I would say harness barely matters in terms overall performance and behavior. Opus is nearly identical in both harness. The opposite is also true, if I switch Copilot to something like Gemini 4 Argon it's completely different even though I'm in the same harness.
Just some andecdata.
That can make a huge difference, as evidenced by the agent swarms success at cracking maths.
Thanks for your eval work, it seems like a tough area to measure!
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
https://www.stet.sh/blog/cost-per-successful-task
Cost per pass is simply the average cost per task divided by the pass rate.
If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.
To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode
Harnesses simply make different choices around certain context engineering things, what they name tools, how they compact, their system prompt, etc.
So it's not surprising that they work "better" for some models than others, but I'd bet that if you made each harness have same system prompt and tool names, the variance would shrink further...
So I would say know what you're optimizing for and construct the harness you need for it instead (it's just a while loop with an llm call and function calling after all) ...
I think this is not too different from physical products, either. You often hear of companies spending multiple years and iterating through dozens of prototypes in order to bring something new to the market.
I don't think there is a shorter path: you never truly understand a problem until you take up the engineering work to solve it.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
For example I remember Codex having a different line number format than CC and every other harness had to have a two format tool to maximise performance.
I wonder if sucha thing exists still or do they RL on all possible combinations now
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
https://mimo.xiaomi.com/rl/#overview
However, the hardware requirements are still quite enormous.
I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.
As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.
Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.
https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)
I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
Is this post mixing them up or is my understanding of these concepts off?
Do you have a different understanding?
Programmers, famous for not obsessively configuring/customizing their tools
The page's title is: "Introducing FrontierHarness Eval v1.1"
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.
We waste whatever is cheap to gain what is dear.
(Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)
Second thing to consider: ZCode was sending everyone's code to Alibaba... not sure I would use their harness just to save a few cents on coding.
But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
https://news.ycombinator.com/newsguidelines.html
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
Your post history is almost entirely promotion of your product. Please stop. I'm flagging your post.
[1]: https://news.ycombinator.com/newsguidelines.html