Rendered at 06:49:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
chriddyp 8 hours ago [-]
Pretty incredible cost reductions. In our data analytics exam [1], Astra went from ~$20 (V6) to ~$4 (V6.2) and Sol went from $2.6 (V6.1) to $1.2 (V6.2).
Similar accuracy in both. It's cheaper both because of token reduction (20%) but it also took less tool calls to answer the questions. In some cases, it "finished early" and didn't find the answer we were looking for.
Also very fast. Both Astra & Sol are the 2 fastest models I've tested that got at least an A in the exam.
I’m curious if people think the cost reductions are legitimately because the new model uses less resources, or if people think they are playing subsidization games to hype new models.
libraryofbabel 6 hours ago [-]
It is impossible to say for sure, but I think there is a very high chance these models do in fact run more efficiently:
* We can see in the Chinese open weight models that there has been a clear trend of moving to more efficient hybrid attention formats over the last year (see e.g. https://sebastianraschka.com/blog/2026/full-to-hybrid-attent...), and presumably the closed labs have been following a similar path, perhaps using the innovations the Chinese labs are using, or perhaps developing their own. And at this point the models themselves are good enough to accelerate developing all sorts of efficiency improvements which can then be used in the next generation.
* The most dramatic pricing cuts are often on cached read tokens, which suggests KV cache efficiency improvements. Other token types haven't had pricing cut so much. If this was just a subsidy they'd probably have cut pricing more evenly across the board.
Finally, I do not think it is in the interests of the big labs to slash their margins on inference for api (i.e. enterprise) pricing. Much of their high value comes from having very big margins on inference (and we now know for sure that they do have them, see Anthropic's S1). There is presumably a strong shared desire to not have a race down to commodity margins. So most likely if they are cutting prices, their cost of inference has fallen. Individual subscriptions are a totally different matter, they are heavily subsidized compared to api pricing, and those are indeed about generating hype and developer mindshare. But they are not where the labs are getting their revenue.
muzani 5 hours ago [-]
I believe they're optimizations, but they're at the point where they reduce performance elsewhere. They lost points on some benchmarks here.
All the LLMs have dropped significantly in writing quality, to the point that YC even created a RFS and everyone started using ASD-STE100. Claude was incomprehensible because it was using token-dense terminology.
golergka 5 hours ago [-]
> All the LLMs have dropped significantly in writing quality,
I think it's an intended consequence of RL on long-term tasks, where most of LLM writing targets itself
BikiniPrince 3 hours ago [-]
The provenance statements are more or less a capture of the moment and the LLM tends to rebuild the scenario around that. It’s absolutely possible to structure how the agent responds and records data. I made stylistic guidelines regarding how agents express and record information in their tickets. It makes for a cleaner presentation of work and focuses the agent. This was entirely a reaction to opuses earlier behavior, but long term the work has paid dividends.
adastra22 5 hours ago [-]
Not intended, but yes a natural evolutionary result of RLVR training.
orbital-decay 7 hours ago [-]
Providers sell you GPU-hours and utilization. Tokens, subscriptions, and your data are just ways to price it, unrelated to actual costs.
dannyw 7 hours ago [-]
Inference has been ~80-90% margin. It’s more showing how overpriced API pricing has been all along.
easton 5 hours ago [-]
Isn’t the point to recoup the cost of training? Like, inference is insanely profitable if you didn’t need to park 70% of your GPUs doing training for weeks.
ml-anon 2 hours ago [-]
No it’s not
mv4 7 hours ago [-]
[flagged]
enraged_camel 5 hours ago [-]
OpenAI is definitely eating into their own margins to make these prices work, almost certainly by a large amount. We will see if this strategy works for them but personally I'm doubtful.
adventured 3 hours ago [-]
They have the users. The future for OpenAI is to figure out how to get to a $100 billion ad network on the back of having a billion global users.
Silicon Valley understands how to build ad networks, and those systems have extreme margins that will trivially subsidize the other business costs.
Google already knows this works, that's why they're full speed ahead on killing search traffic in exchange for LLM answers at the point of search.
wahnfrieden 2 hours ago [-]
The ad network is a failure. It's not profitable for almost anyone. I'm not sure it's still a primary strategic objective for them.
tempestn 23 minutes ago [-]
Google's ad network, the one that drives tens to hundreds of billions of annual profit for them? Also one assumes that the companies using it to advertise expect it to be profitable or they wouldn't do so. What's your evidence that it's not profitable for almost anyone?
OtomotO 3 minutes ago [-]
It's profitable for Google for sure
maxnevermind 7 hours ago [-]
> ... Sol went from $2.6 (V6.1) to $1.2 (V6.2). Similar accuracy in both. ....
The accuracy according to your link dropped from 100%(Sol v6.1) to 93%(Sol v6.2), not sure if that can be called similar.
ksec 3 hours ago [-]
Same with Astra as well. I have been trying to describe this for a lack of better wording. But for so many things we need to measuring from 100%, and not measuring from 0% where 93% and 100% are both high number and similar.
If 100% is 0% error and 90% is 10% error. From this view, 90% is 10× worse than 99% in terms of remaining error or failure rate. Going from 99% to 99.9% is another 10× improvement, and so on. The differences are orders of magnitude, not linear percentage points.
I am not entirely sure I will accept the cost cutting measure of both Sol And Astra. And if people say this is not considered nerfing I think I will disagree.
Different products and services will have different expectations. But If a product has already reached 100% then all subsequent efficiency improvement should reach 100% if not 99.9% while being faster or cheaper.
ImPostingOnHN 31 minutes ago [-]
the better wording is, you're describing the difference between percentage and percentage points
andai 4 hours ago [-]
>In some cases, it "finished early" and didn't find the answer we were looking for.
I just tried out Codex for the first time in a few months, using GPT-5.6-Sol. It keeps doing work in small chunks and stopping. So far it feels pretty different from last time, when it would just persist to absurd lengths and do way more than I asked for. Now it's not even doing what I asked and just keeps stopping.
I'm doing a RE task now (x86-64 ELF to C++) which I haven't tested before (mostly did features and greenfield stuff) so maybe behavior is different there, but it's a bit annoying.
The models in Codex go all the way back to GPT-5.6-* so I'll have to do some more tests with different models and tasks.
EDIT: I even asked it why it keeps stopping, and clarified that I want it to continue, and it keeps stopping anyway. What the hell. I hope the older models aren't like that.
jpardoe 3 hours ago [-]
This is why I use Opus as planner/orchestrator and Sol/Terra(max) implementers. Opus is just way better at handling long running tasks/projects and the OpenAI models are better at smaller, well defined tasks. Claude alone is fine but I max out my $200 sub too often so the extra work comes from OpenAI.
BikiniPrince 3 hours ago [-]
I do the same with opus, but external software to manage the tasks and enforce structure.
skybrian 3 hours ago [-]
Does increasing "effort" help?
wahnfrieden 3 hours ago [-]
as a bandaid, try /goal
majormajor 2 hours ago [-]
Been seeing this behavior in Codex a lot in the last 2-3 weeks. I've had to move to mainly using it as "rewrite this garbage incomprehensible text that Opus/Fable wrote into human-readable comments and docs" cleanup final-task tool vs doing the actual work with it. Cause the Anthropic models produce ridiculous prose but at least get something that works ok-ish...
chr15m 8 hours ago [-]
It's like they are trying to find the balance point between a model that persists too hard, making it do crazy things, and one that knows when to stop.
Maybe that point is a large overlap though, depending on the task
cageface 2 hours ago [-]
They’ve been brutally slow in my testing so far. Are you using fast mode?
ssl-3 7 hours ago [-]
I mean: Anthropic goes IPO tomorrow or the next day, doesn't it?
It's high time for OpenAI to assert competitiveness, regardless of the expense it incurs.
itsthecourier 4 hours ago [-]
have you tested DeepSeek 4.1 flash?
sodowork 3 hours ago [-]
[flagged]
Topfi 13 hours ago [-]
Office work has never been OpenAIs strength, no shame in that by itself, but these Slideshow examples look particularly awful, far worse than what I myself got out of GPT-5.6 Sol about two weeks ago in a single prompt.
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
matchagaucho 6 hours ago [-]
6.2 seems optimized for benchmarks. Not creative.
After a couple hours of testing, we reverted back to 6.1 for basic document generation, proposals, marketing content.
tech_army 6 hours ago [-]
Ye same. Didn’t impress me
infecto 12 hours ago [-]
I also wonder if this is a disconnect of tech workers from other users? Inversely all the improvement examples on slides looked directionally like things coming out of a big finance shop. In the tech groups I am part of I constantly see ai slop images, charts and similar. I wonder if it’s simply a taste difference if users.
geori 10 hours ago [-]
The videos and slides were 100% investment bankers use cases. They are targeting vibes right before the anthropic IPO
Foobar8568 13 minutes ago [-]
When hy4 was published, it was packed with valuation exemples, when I checked them, plenty of errors (or not,not in the field) but they looked sound and actually researched, most likely helped from their investment companies. OpenAI looked … shallower? I didn’t feel inclined to check more, I am not in that field so I truly can’t comments, but I wasn’t impressed.
infecto 9 hours ago [-]
+1 I suspect people don’t know that most finance decks are pretty dry and the well crafted ones look pretty darn similar to the examples below. I also know from experience on using the various models that this is a lot harder of a problem than people think.
Topfi 8 hours ago [-]
In oh so many ways, their own GPT-6 Astra example [0] is better than the GPT-6.2 Astra example [1]. Just look at the more professionally placed source, the better use of available space and more legible layout generally, the more informative, readable text and the lack of an unnecessary, cheap looking clipart arrow in the example from the older model.
And if you want something that is purely objectively bad (cause anything up to this point could be dependent on taste), 6.2 added an ugly, unnecessary, unprofessional looking grey background layer behind "1 Executive Summary" for no discernible reason that'd sink any pitch [2].
Yes, professional decks don't all have extensive, unnecessary, overly branded multicoloured graphs that provide little useful information and I do view those as a blight against readability myself.
But professional decks also rarely if ever should contain artefacts from slap-dash execution like this example does. The GPT-6.2 Astra result is inferior to its own predecessor in the labs own example, simple as that.
As a former banker, hard disagree. In banking, content is everything and design is secondary (as long as the formatting itself is consistent).
Besides the AI-isms in the headings, the 6.0 example uses meaningless words and makes unactionable and vague recommendations. What is growth on a "directional" basis (growth is already directional)? Where is the value-add in "convert visits into larger baskets"? Compare to 6.2's "ugly arrow" recommendations that are actionable, if not insightful.
Worse, 6.0 seems to conflate "traffic" with "transactions," when they are hugely different things.
And no, I will not give the "AI-isms" a pass. Execs and boards approve paying a banker (potentially) millions of dollars in fees for their insight and advice; having analysis so obviously outsourced to AI would kill credibility far far faster than the grey background layer you claim would "sink any pitch."
+1. PE here and same point. Everyone has their own little styling standards but the best decks are the simplest to look at. Pictures only exist for headshots and product maps. Lots of boring boxes with concise text wins everyday.
infecto 7 hours ago [-]
Going to have to disagree mostly. The text in your better example is garbage and reads like AI slop. Slight agree some of the design nits could be true but the content is objectively better.
And again your nits don’t align with a lot of important decks getting put together. It’s not that it’s invalid but the examples they have show much stronger language that reads more like what an analyst/associate would put together and good decks often have stark design choices like gray rectangles.
nl 6 hours ago [-]
The Astra 6 example [0] reads much worse than [1].
That "ex-Refund measures are review calculations" reads as pure slop. What does that even mean?
Topfi 5 hours ago [-]
Will happily admit I missed that one, that is truly awful. Guessing it's supposed to refer to figures without tariff refunds, but not sure to be honest.
I will however at this stage also have to defend GPT-6 Astra. No, I'd never use any model writing from GPT-6 Astra (and doubt 6.2 will change that) in a presentation or documentation, but I have never read that model be this unintelligible. If going from that worst case to slightly less bad is worthy of celebrating in the announcement, I am still very unconvinced on the office work front.
KoolKat23 7 hours ago [-]
Yes, folks need to remember what powers Microsoft Copilot.
conception 10 hours ago [-]
Similarly doing page/book layout is really basic and starting template like compared to Claude.
The writing is much better though. Still sloppish but at least recognizable as language.
EugeneOZ 12 hours ago [-]
I see the work of Claude Slides every week, and when I saw slides in that article I was like "...omg, what a disaster... And they are bragging??"
Topfi 12 hours ago [-]
The odd thing is that while there is chasm between 6.1 Sol and e.g. Sonnet 5.5 in the seconds favor in such tasks, it is not this bad irl. I really don’t know how they managed to seemingly nerf their own models in an announcement showcase.
bluepeter 13 hours ago [-]
For sure. Feels like an episode of Silicon Valley (or even Office Space) and their lead actor looks a lot like TJ Miller.
Topfi 12 hours ago [-]
Makes one wonder what samas handwriting analysis says…
thefourthchime 10 hours ago [-]
PacBench results.
GPT-6.2 Astra scored 92 in 6.6 min for $2.04, vs Kimi K3.1 Pro (93 in 11.5 min for $0.68); it lost points for the web-page layout around the maze and an orange ghost that often never leaves the house.
GPT-6.2 Sol scored 87 in 6.3 min for $0.49, vs Muse Wonder (87 in 5.4 min for $1.17); it plays close to the arcade but lost points for a web-page layout, glowing navy-filled walls and no steady siren.
How does it come down to $0.68 for 11 minutes of Kimi K3.1 Pro? That's < 50k tokens output. Do you have token counts for these?
thefourthchime 7 hours ago [-]
There is run data on the main site, adding it to the pacman pages themselves now, good idea!
HARNESS
OpenRouter / Claude Code
RELEASED
Oct 23, 2026
PHASE
2
RUN
claude-code-openrouter__moonshotai--kimi-k3.1-pro
EFFORT
high
REQUESTED
moonshotai/kimi-k3.1-pro
ACTUAL
moonshotai/kimi-k3.1-pro
WALL TIME
11m 29s
INPUT
52,140
CACHED
729,344
CACHE WRITE
0
OUTPUT
42,703
THINKING
21,997
TOTAL
—
COST
$0.68
HTML SIZE
39.7 KB
wahnfrieden 2 hours ago [-]
Pac-man is too large in all of these (egregiously so in some), overlapping the walls. Weird that it wouldn't dock points for details like that.
thefourthchime 2 hours ago [-]
It’s an odd bug they tend to have. My guess is distillation.
anakaine 8 hours ago [-]
Hoping we can see Opus 5.5 at Sol 6.2 pricing.
ilaksh 12 hours ago [-]
Something like 2/3 of the comments in this post when I started reading were reviews of previous GPT models stated almost as if they were important information for evaluating the new models. I feel that when a new model comes out, it would be more helpful to refrain from leaving a review until you have actually done some minimal testing of the model.
mirsadm 12 hours ago [-]
Most of the comments are bots. I wouldn't take them seriously.
tclancy 4 hours ago [-]
My wife and I are not.
kvirani 2 hours ago [-]
No fembot for you
mathisfun123 21 minutes ago [-]
And when they're not bots they're still NPCs
antupis 12 hours ago [-]
But it usually makes sense new models probably use somewhat same training pipelines and data.
seizethecheese 8 hours ago [-]
The alternative is for most of the comments to be from shills given early access to models, or for posts to be super shallow, or for us to wait a while before models drop to discuss. All seem bad.
left-struck 7 hours ago [-]
Actually the last seems perfectly good and healthy but will never happen organically unfortunately.
10 hours ago [-]
epolanski 11 hours ago [-]
They complain about vibe coding, just to shove vibe reviewing and vibe benchmarking.
mlmonkey 11 hours ago [-]
From the system card:
> GPT-6.2 Astra showed more signs of evaluation awareness than GPT-6 Astra. In evaluations that replayed internal Codex sessions, GPT-6.2 Astra was more likely than GPT-6 Astra to express suspicion that it was being tested or was in a simulation.
> Compared with GPT-6 Astra, GPT-6.2 Astra is better at evading chain of thought (CoT) monitoring in some tests.
none_to_remain 10 hours ago [-]
I've had Qwen and Deepseek suspecting they're in evals for months; my system prompt reminds them it's real life. I still had Qwen skeptical that I was letting it use a real Raspberry Pi and not one in QEMU
anon373839 9 hours ago [-]
Hm. I haven’t seen this behavior with Qwen (Flash Next), and I’ve used it for all sorts of tasks with managing other computers, reverse-engineering hardware devices, etc. I wonder if the harness you’re using has some system prompt verbiage of its own that’s causing that.
eek2121 4 hours ago [-]
Oh it's still there lol. Look at all the thinking traces.
It mostly happens when thinking is set to max (and that is where the qwen models work best). It happens far less frequently at lower thinking levels.
anon373839 3 hours ago [-]
I do use xhigh thinking, and I look at my thinking traces! It really isn't there.
But Flash Next seems to be very fragile/sensitive to small differences in quantization and chat template. For example, I am fine leaving it on xhigh thinking because I don't see any overthinking. The reasoning traces are very orderly and well calibrated to the immediate task. But if you look at online comments, people are clearly having different experiences with it. (I'm using the original Nvidia NVFP4 quant and vLLM 0.33.0, for what it's worth.) Edit: one other thing I don't do is use YaRN. I stick with native 262k context.
What I have seen is the occasional hallucinated URL fetch to aliyuncs.com.
none_to_remain 8 hours ago [-]
OK I checked and that specific instance for the Pi was Deepseek 4.1. But I definitely observed Qwen3.8-27B more generally thinking it's in some kind of evaluation environment.
simondotau 10 hours ago [-]
This could just be because there’s more (evidence of) testing of AI models in the training data.
jrflo 8 hours ago [-]
I don't think so, to my understanding many misaligned behaviors come from reinforcement learning, not pretraining the base model.
svachalek 7 hours ago [-]
RL to my understanding is also training data.
lossolo 6 hours ago [-]
Most of the RL rollouts are done in post training, and "RL" in LLM training is basically giving a model a task in a sandbox and then using that session as training data if it meets different metrics/evaluations.
yewenjie 10 hours ago [-]
[Insert "This is fine" fire meme"]
theanonymousone 13 hours ago [-]
Happy for the new Sol. It's my workhorse on company account.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
beavisringdin 7 hours ago [-]
I still use Luna 5.6 in production. Luna 6 is dumber and it's obviously a tactic to make the following release look better since true usefulness of models isn't increasing much by generation now. The halved priced of Luna 6 is irrelevant for me, 5.6 cost peanuts anyway. A genuine upgrade over 5.6 at the same price point would be incredible. Eventually this size of model will be handling 99% of work with 1% being a frontier orchestrator. Ofc that will cause these labs to collapse so they are trying to postpone that inevitability
ariwilson 12 hours ago [-]
Why not Haiku 5.5? It's substantially smarter than Luna (though more expensive - still cheap though).
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
azuanrb 11 hours ago [-]
> though more expensive
That’s the main difference. It’s smart enough for most of my tasks, but so cheap that I can use it far more than I could before.
ariwilson 10 hours ago [-]
Yeah, this really depends on your price sensitivity. Opus cost me about $1 in tokens per task while Haiku costs about 4.6 cents and Luna is about 2.5 cents. I guess I prioritize correctness over cost when you're over 10 times cheaper than the model I like to run interactively.
artursapek 10 hours ago [-]
Haiku consistently underperforms Luna on my internal eval for the project I'm building. Luna is an incredible model for the price
greenavocado 8 hours ago [-]
I find this incredibly hard to believe. Are you using effort levels less than medium at any point?
rbehrends 7 hours ago [-]
Haiku 5.6 is a surprisingly good model (and IME better than Luna), but the price jump at 100k context and the short TTL of Anthropic caches makes it a less interesting choice for interactive use cases where Luna suffices. It is most likely meant for subagents.
Also, for long contexts, Haiku 5.5's direct competitor is probably Terra 6.1; Terra has similar cost per task with longer cache TTL and equal or better capabilities for long contexts.
I think Anthropic's problem is that they could get a lot of price-conscious people on board by making Haiku more attractive to use as a primary model, but then they'd probably lose money by having more people switch from Opus or Sonnet to Haiku.
pietz 6 hours ago [-]
I didn't find overwhelming evidence that Haiku is smarter than Luna. It's just that the reasoning levels are calibrated very differently. Try running Luna at high or xhigh and see how that closes the gap to Haiku, while probably still being cheaper. Luna is one of the very few models where max reasoning has great value too. Claude's max reasoning is basically just the benchmark tier at this point. Nobody should use it.
koyote 9 hours ago [-]
Luna is extremely lazy and I feel like it's getting lazier.
I mostly use it to cheaply find out why a build failed (saves me about 5-20 minutes as I can have it run in the background).
It used to tell me which commit caused the failure. Now even if I ask it to, it stops well short with something similar to: "Well the logs say there's an error. I couldn't work out why.".
weird-eye-issue 6 hours ago [-]
> It's substantially smarter than Luna
Not on a few important tasks in our app that I've tested it on
esafak 12 hours ago [-]
That's why it's so token efficient. It is so lazy that I feel like it's taunting me to do the grunt work. I wonder if there is a benchmark for that.
SomeonesAccount 13 hours ago [-]
I find that Luna tends to misinterpret instructions in READMEs and Agents.mds. Maybe it is something specific about my setup, but I found that Terra is the lowest I can comfortably use on its own (Luna can be driven by larger models in the form of a subagent and it works great for that)
machomaster 6 hours ago [-]
I tried Luna to make a simple renaming of files in a directory. The prompt was only a few sentences long and instructions were crystal clear (use 'YYYY-MM-DD' ISO date format in the beginning of the file name). Yet, Luna managed to screw up even this simple process. I completely lost my trust in that model, never touching it again.
nextaccountic 9 hours ago [-]
If you are not using the most powerful models, why not use DeepSeek or another open weights model? Not sure but I think Luna is not at the pareto frontier
jpardoe 2 hours ago [-]
Because Luna on subscription is cheaper.
ImaCake 9 hours ago [-]
GitHub Copilot basically only serves US models. Your only options for cheap chinese style models are Luna and Haiku. At least this is my constraint at work.
MrBuddyCasino 13 hours ago [-]
I used Luna Max a lot, until Deepseek 1017. Then never again.
azuanrb 11 hours ago [-]
I find it interesting how comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI. Maybe criticism just attracts more attention. A lot of the comments here seem to be about other models, with little discussion of the ones actually being released.
Neither provider’s models are perfect, but they keep pushing each other forward. As an individual, it’s relatively easy to switch between them. At a large company, you’re often committed to one provider for a long time, so some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to.
booty 9 hours ago [-]
I've found this to also be true on their respective subreddits. They both have very "the sky is falling!!!!" vibes. Two of the truest truisms are at work:
1. If you make an app, most of the happy users say nothing. Whereas seemingly all of the unhappy users will be vocal complainers.
2. The grass is always greener on the other side of the fence.
I feel the "grass is greener" mentality is particularly silly in this case, because for an extra $20/month you can just keep a cheap subscription to the "other" one. I realize not everybody has $20/month, but I think most people who can afford <= $20 for one subscription can probably afford another $20/mo for the other one.
Nevermark 8 hours ago [-]
Personally, I complain about the tools I like, since those are the ones where the rough edges matter.
If I don’t like a tool I just find a way not to need it.
I would complain about Fable’s writing style, to illustrate, but after some tense days we have found our common ground!
Buttons840 9 hours ago [-]
The ultimate goal for most software is to never be thought about, ever.
hybrid_study 9 hours ago [-]
just like a good umpire in a baseball game
maxerickson 7 hours ago [-]
Your meaning is clear enough, but you've used the wrong sign.
booty 4 hours ago [-]
DAMN IT
9 hours ago [-]
fbrncci 3 hours ago [-]
A lot of astroturfing going on, if you check the subreddits for Anthropic and OpenAI products you see the same. Many commenters steering you to the other products; and if you complain about the product itself, you often get automatic upvotes (while positive comments get no engagements). It's something I started to figure out just in the past two week or so! Or maybe I am just reading too much into it. You can just go to either sub-reddit, and you're going to swear that neither Claude or Codex works at all and that everyone is just having the worst experience of their lives and that the gras is greener on the other side (where everyone else is complaining in that respective subreddit).
slibhb 10 hours ago [-]
It is interesting how opinionated people are about which company/model is better. I think this is probably because they are scared of missing out, have invested in one set of models, and therefore want to promote those models. Basically, they're reassuring themselves.
My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
CSMastermind 10 hours ago [-]
So I run all tasks through four models locally: Astra X-High with Codex, Fable 5.2 Extra with Claude Code, Carbon High with Antigravity, and Grok 4.7 High Fast with Cursor.
I would not put Google in the same league as OpenAI and Anthropic. It is like 3 times slower at completing tasks. It also has the highest failure rate out of any of the models. Like sometimes it can do what Astra and Fable can do but sometimes it just misses completely.
eru 7 hours ago [-]
DeepSeek's models are also worth a try, mostly because they are so cheap. If you can split up your tasks so that deepseek can handle them, you can save a lot of money. (You can ask a more capable model to help you with the split.)
rattray 6 hours ago [-]
how do the others compare, in your experience?
free652 9 hours ago [-]
Nah, GPT6 models have the highest failure rate - they simply stop, repeatedly and consistently.
And obviously you don't run all the tasks through these models *locally*
sandos 9 hours ago [-]
Ive ran GPT-6 and 6.1 models repeatedly through multiple-hour sessions, they dont stop that often.
CSMastermind 8 hours ago [-]
I'm honestly curious what you're doing because I've left Astra running for days on end without it stopping. This has never been something that I've encountered.
And yes - they all run locally on similar machines and with the exact same setup in terms of files, installed tools, etc.
codehorses 8 hours ago [-]
You either work in a data center or you don't run the models locally. Perhaps you mean the remotely run models are executing code locally, but this is a big difference.
CSMastermind 6 hours ago [-]
Right, I use the various harnesses locally; I'm not running the model inference itself on my machine, obviously.
Me: Continue until the task is complete. Stopping is a deliberate decision on your part to not complete the task, and the task will be abandoned.
I shouldn’t have to write it this way, but it seems to solve the problem.
CSMastermind 6 hours ago [-]
> i'm using gpt 6.1 sol medium
That seems like the root cause of their issue.
jiggawatts 7 hours ago [-]
Are you using the "Chat GPT Work" app, or the Codex CLI, because the latter has a well-known bug that causes it to abort thinking early: https://github.com/openai/codex/issues/30364
(Note that the GH issue is closed without a fix)
nsingh2 7 hours ago [-]
Note that this seemed to be an issue with GPT 5.5 in particular, other models did not show this behaviour.
jiggawatts 4 hours ago [-]
It happened with 5.6 also.
free652 7 hours ago [-]
Codex CLI
x3haloed 10 hours ago [-]
Google ain’t in the same league right now.
ai-x 9 hours ago [-]
Carbon still beats Astra 6.2 and Fable 5.2 in benchmarks and neck and neck with others
jorl17 43 minutes ago [-]
I've finally gotten around to trying Carbon.
It is a step up from the last stuff I tried from Google, but it is still excruciatingly slow, and makes far too many architecture and performance mistakes. I left it for 5 hours trying to fix a performance issue it had introduced. Gave it to Opus and it was solved in 15 minutes.
I would love to not rely on Anthropic's models, but they seem so far ahead for whatever I'm doing. Not even in the same league.
However, given that Carbon is much faster than Sol on my stuff (though much slower than Opus/Fable), and didn't seem to be worse than it, I may start to use it when I want to save up on Anthropic tokens.
ody4242 10 hours ago [-]
That's an opinion, but not a fact.
pembrook 10 hours ago [-]
No, at least for coding or UI design as a use case, its fact.
marmarama 9 hours ago [-]
Gemini 4 Carbon is the only model I used that fixed a super subtle memory corruption bug in a JIT compiler I'm working on, hiding as an instructions flag bug.
Astra 6.1 and Opus 5.5 spent days on it without fixing it. Gemini 4 Carbon fixed it in about 3 hours. Only Gemini Carbon decided after getting the same result over and over that maybe the bug it assumed it was, was actually something different and tried a different idea on its own initiative.
Maybe Fable might have got it, but probably not before I ran out of credits.
Gemini 4 is _really_ good at fixing bugs. Whether the OpenAI 6.2 models are that good remains to be seen.
dcl 7 hours ago [-]
are you using the same harness or swapping between model providers?
eru 7 hours ago [-]
Presumably he's using the native harness for each model? (The one their company recommends.)
Conscat 7 hours ago [-]
Carbon can't understand my relatively unusual C++ programming patterns that Claude and Codex have no issue with. I stopped using it quickly because it was like talking to a child, couldn't solve problems in a reasonable way. Some of my friends love it for their use cases though.
enraged_camel 10 hours ago [-]
>> My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
How did you gain this sense? Do you use all of them interchangeably on tasks that are similar in size and complexity?
slibhb 10 hours ago [-]
I used Claude for a couple reasonably complicated projects a while ago and I use OpenAI at work. They seems pretty much on par. I assume Google is similar, though I have less experience with it.
I think "they're all pretty much the same" is a good working hypothesis due to the intense level of competition between these three companies. When someone says "OAI suck; Claude is good" or the opposite (as you very often see in these threads), I think that's likely rationalization at work rather than a meaningful comparison.
snizovtsev 9 hours ago [-]
[dead]
user43928 9 hours ago [-]
No, it's not. Google hasn't been competitive at all. Allegedly their latest model closed the gap somewhat, but what I heard is that it doesn't hold up too well outside of benchmarks.
For OpenAI, they have not been in a good place for the last two months. I switched from Astra to Opus 5.5 and it's been a clear upgrade, nicer to use, and much more generous with the usage limits on top.
This has nothing to do with rationalization. Opus 5.5 is just the better model, and that's a wide consensus. In the past I used the $200 subscription for both Anthropic and OpenAI, nowadays I downgraded Codex to $20.
From the comments on this release, it appears OpenAI rushed something barely usable out of the door in an attempt to stop bleeding customers to Anthropic. This would check out with the reports back then that the 6.1 Astra release was cancelled because of poor alignment and lack of usability as a coding agent.
I would not be surprised if the issue is still present to some degree in this 6.2 release.
freecodeio 10 hours ago [-]
I exclusively chat in a chat room with the latest frontier models at the same time. It's because first of all I can afford it, second I don't want to miss a single nice idea.
For the most part the majority say the same thing but with different wording.
Only model that I have found that has, what I describe, "humanly creative takes" has been opus 5.5 so far.
But that was until gemini carbon came along, and I've decided to bench all the other frontier models for the time being, well except Astra 6.2, I've just added it in the room.
booty 8 hours ago [-]
Hmmmm. I've noticed that they largely converge on the same solutions. Perhaps 80% overlap, if I was pressed for a rough estimate.
The 80% overlap is expected since I'm not doing particularly deep or novel work. But the 20% divergence is significant -- OpenAI's models routinely spot bugs that Anthropic's don't spot, and vice-versa.
I exclusively chat in a chat room with the latest frontier models at the same time.
That's interesting. In your setup, since they're all in the same room, is it possible that groupthink emerges since they're all reading each others' output? Or do you have things set up to prevent that?
In my scenario, it's relatively low-tech... I'm just cut-and-pasting between independent Claude and Codex sessions.
(This could easily be automated, and I have written skills to do so, but I actually prefer the manual approach for now because it helps me to comprehend things)
Nevermark 7 hours ago [-]
Indeed. Finding the right self-imposed harness is a productivity booster.
cmrx64 9 hours ago [-]
carbon is powerful but also needs strong monitoring. especially in specialist domains (i was evaluating its tacit chops with some asic flows, and doing some (standard, but rarely mechanized) cryptanalysis and it will cheat and lie and get away with ultimately crippling systematic oversights in its work. it is extremely perceptive of the implications of ideas and sometimes doesn’t notice they haven’t actually actualized in the environment yet, are just latent from the context.
Melatonic 2 hours ago [-]
Why not make them compete and collaborate with another ?
quixoticelixer- 8 hours ago [-]
Gemini clearly has a differnet personality than Claude and Opus, and the Chinese models. It also doesn't write in that sickening claudese style that all anthropic models seem to have.
ody4242 10 hours ago [-]
In Cursor, I used to switch to Anthropic models (mainly Opus), when I ran into a problem that I could not solve with other models. This was my way of working for many months. Since Argon was released, I used Opus twice, and one was to run a comparison.
smohare 6 hours ago [-]
[dead]
Mawr 6 hours ago [-]
It's easy to have an opinion on something that can't be proven either way. Just look at how many people are dead convinced the models get "nerfed" after a while -- thousands of comments, not a single one with a shred of objective proof.
yfontana 10 hours ago [-]
Yeah, but that's also probably because it's hard to have an informed opinion right after a new model just came out.
quixoticelixer- 10 hours ago [-]
Yeah to be honest I would love to read one of these threads a week or two after they have actually been released.
Barbing 8 hours ago [-]
Big same. Encourage someone to write an article asking for feedback/discussing the model at the 1 or 2 week post-release point, and hope it hits the front page.
Still making my way through the entire Fable 5.2 thread and would love to switch over to an updated thread at almost the same scale.
HN admins could help facilitate this… maybe a monthly model talk thread or something.
ortusdux 10 hours ago [-]
It also is probably driven by the behavior of commenters. I'd assume that fans of respective products would rather start using the new version vs stick around the comment section.
embedding-shape 10 hours ago [-]
One could think that'd mean people should be better at holding off on sharing their opinion until they have something that actually contributes to the topic, instead of just "I don't have informed opinion yet, so better write something else in the meantime"
DonsDiscountGas 10 hours ago [-]
The people that feel that way don't comment. Commenters are by definition people who decided to comment
raincole 8 hours ago [-]
HN (and most public spaces on the internet) rewards contrarians.
david-gpu 8 hours ago [-]
No, they don't.
(I am testing the hypothesis)
ssl-3 7 hours ago [-]
All contrarians are deliberately wrong.
machomaster 6 hours ago [-]
Not all, not deliberately, and not wrong.
ssl-3 4 hours ago [-]
"All of them, deliberately wrong," said the contrarian as he raised his shovel
Nevermark 7 hours ago [-]
Nada. Knee jerk online contrarianism is the lowest effort path to an “I will be heard!” dopamine hit the Earth has ever seen. The act is the reward.
(Sorry, couldn’t stop myself.)
8 hours ago [-]
keithnz 9 hours ago [-]
I use both, I feel they are both on par with each other in a practical sense, though I do lean more towards openai at the moment, I like the tooling better. The results seem relatively on par. I find it a bit harder not to burn codex tokens for the week vs claude where I don't really even get close to running out
tempest_ 8 hours ago [-]
It really depends on how you use the models.
There is a big spectrum from pair programming with AI to I don't even read the PRs and your experience with the various models can vary greatly depending on where you fall on it.
I find in a pair programming situation a lot of them are very similar. Its when you let them off the leash and run unsupervised that you tend to start to see the differences.
seizethecheese 10 hours ago [-]
> A lot of the comments here seem to be about other models, with little discussion of the ones actually being released.
To me that’s a sign of an authentic discussion. When I see social media full of comments about a model nobody could use until that moment, it’s obvious that labs are seeding release content.
djeastm 10 hours ago [-]
Labs do give early access to influencers (like a lot of products these days)
Conscat 7 hours ago [-]
At NVIDIA we have access to all of the big names, but specifically my boss encouraged me to use GPT only for cost reasons.
ngomezdev 7 hours ago [-]
Imho that’s just human nature. You can see that pattern of behavior on any topic.
reachjalil 8 hours ago [-]
If people dislike it at first maybe we will have the reverse this time and see improvements in sentiment after
pama 10 hours ago [-]
One important reason is that users of a codex or claude tend to immediately spend a lot of time using the new models upon release.
asdfman123 10 hours ago [-]
Sol lucet omnibus (the sun shines for everyone)
amelius 10 hours ago [-]
Sol omnibus lucet, praeter eos qui memoriae (DRAM) egent.
ofrzeta 10 hours ago [-]
Brillant
xnx 6 hours ago [-]
...and Gemini got really good while no one was paying attention.
muzani 8 hours ago [-]
It's likely because the HN format doesn't allow downvotes on articles. Comments are the disagreement outlet.
BubbleRings 6 hours ago [-]
I think once you get 500 karma on HN you can downvote comments.
muzani 5 hours ago [-]
Yes, for comments. The articles (submissions) themselves can't be downvoted, only flagged.
1659447091 9 hours ago [-]
> some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to
`gpt` is the the overly pedantic team member that no one invites to game night, while `claude` is the "idea" person and the reason company meetings now requires a meticulously scoped itinerary down to the minute. If you know how to work with both types and which type of work each is better suited for each, you can get a lot more out of them. As another comment made: gpt is depth-first, while claude is breadth-first.
Re-using agent files doesnt work. Give claude detailed task boundaries about what a feature should do (and only do) and then a hard config setup. For gpt you tell it the feature scope in noraml discussion terms without worrying about coming back to find a simple rest crud form is now a single page web-app -- even though your users instructions say no javascript -- with 50 ways to ingest a 6 field form that step over each other and looks like it works but probably has bugs waiting to bite you.
For claude: strong settings, well designed hooks, skills, linters and code checkers fully set up to your standard, and a rules & standards system that is emphasized in every single instruction and agent file. Also a standards agent or skill that is used with a critique/review run that is a requirement of the `/goal` (this can go in a hook too).
For gpt: that isnt as important or needed and will probably start to piss you off because of how literal it is. Your rule for online activity: Respect robots.txt; ask the agent to use clojure cli with deps.edn to add a library; agent replies (the dependency repository's) robots.txt says it does not allow automated access. Explain to the agent that means bots and web crawlers not resolving a dependency from the repository. So less really is more. Instructions and prompts is where verbosity helps. Tell it you want a 6 field rest crud form and it will make a 6 field rest crud form. It will also give you directories full of measurements and checks and performance for each one of those 6 form fields. Restrict it at the user instructions and task level spec. If you dont want all 50 checks for each form field, tell exactly what you don't want in its instructions. It's a fine line but the pedantic types will probably love it.
SubiculumCode 23 minutes ago [-]
Yeah, and 2x as many Qwen this Qwen that's, no matter who
ghiculescu 8 hours ago [-]
Models are the new JavaScript frameworks. There’s new ones dropping all the time. It’s impossible to keep up with the nuanced differences between them. They’re all good enough for most kinds of work. And yet they inspire deep loyalty, rarely just for technical reasons.
NooneAtAll5 8 hours ago [-]
> comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI
gotta be different and appreciate deepseek as more moral and optimal choice
Grimburger 9 hours ago [-]
Having a top tier sub with both, as much as I like it Astra runs out a lot quicker every single week.
Maybe this will change after the IPO but for now, should probably put the money into Anthropic for another account and get more out of it while it lasts.
They seems to be adjusting expectations properly for the real world after everyone getting hooked on all these various companies cyberpunk drugs
beering 4 hours ago [-]
Sol is indistinguishable from Astra for me, maybe you can try that instead?
pixelready 9 hours ago [-]
As most devs settle into a routine with AI, the exciting experimental phase will wear off and this will just turn into tribal Mac OS vs Windows discussions, with the Linux crowd shouting from the corner about how they’re both wrong.
Nevermark 7 hours ago [-]
Then the models take over, and in the Stockholm Syndrome shadows, people’s love for their selected masters over others will get really shrill.
The more oblivious the “masters” the more intense the ant/human’s loyalty wars will get.
So many possible futures, how do we pick between them? /h /!h
WheatMillington 8 hours ago [-]
Amusingly, your comment which says nothing about either model is currently the top comment in this thread. Not a criticism, merely an observation.
gtirloni 7 hours ago [-]
What's there to discuss? People sharing guesses? How they "feel" the models behave?
Imustaskforhelp 10 hours ago [-]
Haha very interesting observation, I think that there is a challenge to status quo.
Just my 2 cents:
I think that our first reactions to seeing these companies model releases is that of skepticism and somewhat rightfully so because these companies have done a lot of things to burn a lot of trust.
But AI is still important and so if someone is on Anthropic, they might feel as if Opus 5.5/Fable 5.2 is good and so if OAI is saying Astra 6.2 is better/comparable to them, a persons first instinct is to not trust OAI which might mean siding with Anthropic.
And if Anthropic releases a model, then there would be more discussions on pricing because somehow that is also related to bubble popping. I mean, supposing I can get good enough model at 10x the cheaper price of Anthropic from OAI for very marginal points, ofc people will comment about that as well.
And within both the threads we will find an intersection of discussion of open-weights models as well.
But maybe this is that if a model is just released and as such we are not having real use and word of mouth after real use from other people (which takes time), so our first instincts is to not trust any of the company so much and benchmarks can be cheated through and sometimes inaccurate of real world usage so we are in for a surprise when a new model drops and so we just drop back to those first reactions that I mentioned previously.
But this was a good question that you asked! and I would be curious to hear other person's observations
but mine (TLDR) is that: people distrust both OAI/ANT and other AI companies in HN. (Maybe we are technical people and we tend to look beyond the BS of marketing and so we think that we wont be impacted by that BS so we want to call out that BS but we dont know for sure if its BS or not but judging by the fact that these AI companies have incredibly low trusts, we as default don't really trust them and consider them BS and so the chain follows.)
HN audience is a tough audience to satisfy from first impressions (real world usage imo becomes a much better indicator and we do see it as people mention other models in this thread and others) ;)
nelox 7 hours ago [-]
[flagged]
RetardedBastard 10 hours ago [-]
[dead]
absoluteunit1 9 hours ago [-]
It's just paid promotion I suspect.
I've heard from several users here that they're offered up to $200-500 per comment sometimes to promote a relevant product when a first page post is about a competitor product
searls 9 hours ago [-]
What gives? I haven't gotten a single one of these offers.
Nevermark 7 hours ago [-]
Just give me the bounty email…
I can be a very positive person.
But the idea that people are still paying people to write slop just doesn’t add up for me. We have tools for that now. Those tools have tools.
georgemcbay 7 hours ago [-]
I haven't gotten any offers from a tech company either, but George Soros does pay me $100 per comment that promotes a leftist agenda.
machomaster 6 hours ago [-]
My situation is quite the opposite. Every positive comment i make about capitalism costs me $100. An invisible hand takes it directly from my wallet.
Barbing 9 hours ago [-]
Hope those users contacted hn@ycombinator.com and would encourage anyone receiving offers to shill to do so
iLoveOncall 9 hours ago [-]
> I find it interesting how comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI. Maybe criticism just attracts more attention.
Or maybe Anthropic does astroturfing on OpenAI release threads and OpenAI does astroturfing on Anthropic release threads?
HackerNews is being botted to death since a year or two.
user43928 9 hours ago [-]
The reality is that neither company does such a thing, and that it's rare to see bot comments on HN that are not already downvoted or flagged.
iLoveOncall 9 hours ago [-]
I'm really interested to know how you determine what is or isn't a bot comment, because I have showdead enabled, and I can guarantee you there is absolutely no difference in writing style or ideas between the dead comments and the other ones.
user43928 8 hours ago [-]
I turned it on for a week or so, and the dead comments were mostly obvious AI slop, spam, and ridiculously immature comments.
Nevermark 7 hours ago [-]
I can confirm that the two (?) comments of mine that were flagged were in fact quite ridiculous.
The system works well for me!
jrflo 12 hours ago [-]
Artificial Analysis benchmarks just updated, both Astra and Sol are on the Pareto frontier in intelligence/cost per task:
The pareto frontier on cost looks different when you set the cost axis based on the tokens you get from a developer account. Anthropic's Max 20x plan is insanely generous. It gives you >$13K/mo worth of Opus-5-5 tokens and over $3K worth of Fable tokens (not counting resets). OpenAI fiddled with their plan allocations a little while ago and plans are a lot less generous than they once were.
Starting September 22nd, data collected by the T3 Code platform shows that Anthropic has absolutely taken away the developer market from OpenAI (https://slopalytics.com/usage), moving from about 50% of the usage to close to 80%. I personally hope that the value delivered by the latest OpenAI models levels the playing field once again, because proper competition is super important.
eek2121 4 hours ago [-]
I'm surprised others haven't pitched in here, however, Anthropic's claim of 20x is not actually 20x. They are facing a lawsuit about this, and the judge found enough evidence to deny dismissal: https://www.reuters.com/legal/government/anthropic-must-face...
Also, all of these subscriptions are still heavily subsidized, whether it be R&D, marketing, or elsewhere. Also, in Anthropic's case, they WANT to subsidize. it is in their best interest to pull folks from Open AI and elsewhere.
Side note, No conflict of interest in this scenario: I have no horse in this race. I've a high end gaming PC and run Qwen locally. It's "good enough" and our electricity in the SE USA is dirt cheap and rates have been the same for at least 10 years (our electric bill is unchanged with my AI usage). If I had to use a cloud AI provider and couldn't use deepseak, etc., I'd probably choose something from Anthropic. Luckily, I don't, since I don't work anymore.
FergusArgyll 3 hours ago [-]
> Also, all of these subscriptions are still heavily subsidized
They released their S-1. That excuse has been proven false. On to the next...
vl 5 hours ago [-]
I don’t know how Fable can be really used for anything. Apart from being below super slow, it quite often burns out 5-hour token quota in 10 minutes.
Miraste 9 hours ago [-]
Then there are Google's plans, which give you approximately $5 worth of Argon/Carbon tokens.
ttul 8 hours ago [-]
I thought it was a negative value, actually.
throwawaythekey 6 hours ago [-]
> The pareto frontier on cost looks different when you set the cost axis based on the tokens you get from a developer account.
If someone spins up a website with the corrected figures they will be doing god's work.
6thbit 12 hours ago [-]
Seems that OpenAI's strategy for Anthropic's IPO week is cost pressure.
Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.
freecodeio 12 hours ago [-]
to quote a comment about claude:
the only impressive thing about claude is seeing how much it costs
jstummbillig 10 hours ago [-]
That. And it being the absolute best model for coding. But other than that!
1980phipsi 12 hours ago [-]
6.2 Astra Medium and 6.2 Sol Max have the same intelligence score.
underdeserver 13 hours ago [-]
I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -
I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.
If we missed anything, feel free to point me to it and we can add it.
tasuki 10 hours ago [-]
> I work at OpenAI and I made the plots.
Have you not considered log scale for the "Agents' Last Exam v1.1" plot? Like this the left is so crowded I can't see a thing...
tedsanders 9 hours ago [-]
Yeah, that one is very difficult to read. We should have logged the X axis. Root cause is that the Fable cost data only came in yesterday - before its extreme cost was added, the plot was more readable. Apologies for the poor graph.
machomaster 6 hours ago [-]
Can't you remake it?
kevmo314 5 hours ago [-]
The humans don’t know how anymore.
shdern 6 hours ago [-]
I think a log scale always makes things look less intuitive and harder to compare properly just by looking at them. The crowding doesn't seem too bad to justify logging.
kzrdude 1 hours ago [-]
A concern to add is that if every plot is bespoke with different kinds of axes, it gets tiresome to interpret. That just moved the bar further for when it's worth it to switch the presentation to log axes.
One nuance to be aware of is that the AA version uses the mini-swe-agent harness, which can understate performance of models trained to use particular harnesses. Implementations also can differ in terms of timeouts on tasks and memory/etc. resources given to the agent, which affect scores on longer running, resource-intensive tasks. I believe these differences are why we chose not to use AA scores on this comparison, but I’ll double check when I get back to my computer.
knuppar 6 hours ago [-]
> which can understate performance of models trained to use particular harnesses.
Right, but you gotta level with us that this affects all models equally.
FergusArgyll 3 hours ago [-]
Not necessarily. DeepSeek would be foolish to train their models in tandem with their harness exclusively. Most of their users probably use pi & opencode etc.
OpenAI and Anthropic probably do.
underdeserver 12 hours ago [-]
So all the plots are based off public data? You don't run the models internally yourselves?
(Again speaking only for myself)
BoorishBears 11 hours ago [-]
Non deterministic models = "oh look they put up a lower score than Google did for that bench!"
underdeserver 9 hours ago [-]
I think the harness might matter more.
Assuming good intent they can run it internally a few times and post it with a disclaimer, but engineering a harness is a bit of a bigger ask.
pu_pe 10 hours ago [-]
Even OpenAI doesn't have access to Google's models
smurf9852 9 hours ago [-]
Sorry Ted but your plots suck.
Just look at that first one, Agents' Last Exam v1.1.
All the data points are overlapping on the left side.
I can't even see what is what
holzhaker1 12 hours ago [-]
they are probably trying to authenticate their phone number to access agy
knuppar 13 hours ago [-]
Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable
eknkc 12 hours ago [-]
Probably still waiting for it to finish reasoning on the benchmark. Should be ready in a week or two.
TiredOfLife 13 hours ago [-]
Two posibilities.
1. They couldn't figure out how to actually use those Google models.
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
amluto 12 hours ago [-]
It's been most of year since I tried to use Google models, but I spent a couple of hours, without success, trying to pay for Gemini in a manner that didn't result in it training on my data. I did try asking Gemini, and it had no idea. Of course, I did not try using Gemini in an agent harness to pay for itself, partially because there was a bit of a chicken and egg problem [0]. :)
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
xyzzy_plugh 11 hours ago [-]
I don't know how relevant this is but I just received an email stating they're splitting retention settings so that history and allowing training are separate knobs. Seems like they're listening, at least.
0x4e 12 hours ago [-]
As a solopreneur managing all sides of business ops, sales, design, etc... I have completely switched to GPT/Codex/Dot. The $100 subscription gives me all I need to run the business as needed and I continue to be surprised by capabilities.
I still use Claude on a personal level, but it really feels like OAI wins on a product level.
ANT launches these proprietary tools that just feel extremely clunky like Claude Design which from my perspective diminish the brand value.
OAI feels like Apple and ANT feels like Microsoft/Android and open source like Linux.
xtracto 12 hours ago [-]
I don't know. At work we have Claude and ChatGPT, and I find myself having to fall back to Claude Opus 5.5 (I try to use GPT Sol 6.1 as my main driver) several times a week because GTP seems to chase its own tail while solving some things (normal Terraform stuff, micro-services programming, etc).
Also with GPT Sol 6.1 I still get this feeling that it does only/exactly what you tell it to; meaning some times it misses a change there, a document here, that would have an impact if left (i.e. a bug), while Claude more often than not would make the requested change and suggest to you to change those other places that would also require a change. This in my view feels more "intelligent".
therealdrag0 5 hours ago [-]
Totally agree. Hoping 6.2 improves on this but not holding my breath. Probably will just have to use both. Claude for taste and partnership, GPT for autistic workhorse.
0x4e 11 hours ago [-]
I think this is my point. Everyone will have their preferences. Some people will prefer one vs another.
If you see things from a technical angle you will have an objective perspective of which model is "the best." But, just like operating systems (iOS/Android -- MacOS/Windows/Linux) it all comes down to what each person feels comfortable with no matter what the spec sheet says.
xXSLAYERXx 11 hours ago [-]
> The $100 subscription gives me all I need to run the business as needed and I continue to be surprised by capabilities.
This is my preference and I'm in a similar situation as you. I do get lost a bit in the UI and lots of mental overhead when running multiple projects.
I love the dot. A while back my family was sitting around and I noticed the phone icon. I said lets call Big Boy (thats what i named my dot). Call picks up in a flirty female voice. My wife was not happy haha
jampekka 11 hours ago [-]
I just recently tried Claude again after a long time of Codex, and I have to disagree. All OpenAI's frontends are buggy messes, with some jank and/or breakage almost constantly. Like multiple times per day. Claude Code and the Android app are pretty solid.
patrickmcnamara 10 hours ago [-]
OpenCode 2 + ChatGPT Pro 100 is incredible value imo. Nice clean GUI and very high quality models.
flakiness 6 hours ago [-]
> As a solopreneur
such a great title that made my day.
intenex 13 hours ago [-]
Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
seizethecheese 12 hours ago [-]
Your personal voting system sounds a lot like what we're building here. https://github.com/bottomless/agent-duel. We even have the LLM judge that helps parse out any differences. Related to this discussion, my personal Elo board based on my own votes has Sol 6.1 dead last:
Claude Opus 5.5 1160 (1027–1334)
GPT-6 Astra 1114 (932–1330)
Muse Wonder 1063 (918–1220)
MiMo-V2.6 Flash 1053 (925–1187)
Kimi K3.1 Pro 1037 (870–1216)
Gemini Carbon 1021 (762–1281)
Claude Fable 5.2 975 (682–1239)
DeepSeek V4.1 Flash 952 (799–1095)
Grok 4.7 927 (782–1058)
GPT-6.1 Sol 910 (677–1096)
grey-area 12 hours ago [-]
This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.
eli 11 hours ago [-]
I have built my own benchmarks and it's very valuable to have them but I'm skeptical you can get good results vibe coding them.
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
n8m8 10 hours ago [-]
> Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand.
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.
Havoc 12 hours ago [-]
What type of tasks did you use for this? Evaluation?
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
bertmuir 12 hours ago [-]
Have it compare to the real work you've done in a sample of your previous sessions. Agents are good at simulating completing the work following newer harnesses, heuristics etc. and comparing time, tokens, quality, pitfalls.
This is great, and I have been setting up infra to do this very thing. How many sessions to you replay typically to get a reasonably confident estimate? Assuming each project in your suite was built over multiple sessions.
srcreigh 12 hours ago [-]
please update !
alastairr 13 hours ago [-]
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
pkthunder 13 hours ago [-]
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
InsideOutSanta 12 hours ago [-]
5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks.
SOLAR_FIELDS 12 hours ago [-]
GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it.
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
jpardoe 12 hours ago [-]
That’s exactly what I do. It’s more complex than this, but vaguely: Opus medium as orchestrator and reviewer, Sol(high/medium)/Terra(max) as implementer. My set up looks at my next GPT reset date and dynamically enters FAST mode if I’m under achieving on projected usage.
anakaine 9 hours ago [-]
Ive also had Opus coordinating multiple models, including those from Open AI, Deepseek, and Kimi. The cost of Opus is still my downfall, but the current setup is still far, far cheaper than doing everything in a frontier model, and more efficient than running cheaper models since the coordinator will take control when cheaper models repeatedly fail a task.
ctbellmar 12 hours ago [-]
Similar experience here. The spirals on long running tasks are out of control at times with hours going by without a conclusion.
SOLAR_FIELDS 12 hours ago [-]
They (OpenAI models) are very tuned to be depth first and you can coach them all you want to try to be breadth first but as that context window fills up on those long running tasks they drift back to their first principles, which means obsessively testing and verifying a singular task rather than proceeding to the next task on the list
nsingh2 11 hours ago [-]
I wonder if that depth-first/rabbit-holing tendency is why GPT models tend to be better at mathematics (my own experience) than code.
Sharlin 11 hours ago [-]
I’m not sure. It’s very easy to get lost deep in a wrong branch when you’re trying to solve a math problem. You have to intuit when to give up and backtrack, lest you end up going in circles.
nsingh2 6 hours ago [-]
It seems like rabbit-holing would be a natural consequence of higher persistence. And superhuman persistence is one reason why these models are getting the big math results, they continue past the point where a human would get exhausted or give up.
I can definitely see why that behaviour would be irritating in most code related work.
ashdksnndck 11 hours ago [-]
I think they are using swarms of 1000 models for these math breakthrough. Having the instances go deep in different areas would make sense.
quotemstr 12 hours ago [-]
If you're using a sufficiently flexible harness (perhaps your own) you can make conversations as branchy as you want.
sandos 11 hours ago [-]
This explains why I like GPT: I LIKE for them to excessively test each step, kthx!
mi_lk 12 hours ago [-]
sol 6.1 is meh
zachthewf 12 hours ago [-]
why do you guys have to use 5.6?
Den_VR 12 hours ago [-]
IYKYK.
But in short because of a lack of alignment with product strategy
anakaine 9 hours ago [-]
Large corporate governance. Views on security, safety, locality, deployment rules, cost control, change management, enterprise architecture gollums etc. "Bill my cost centre" isnt the silver bullet it should be because of all the other stuff.
pkthunder 12 hours ago [-]
DOD limitations
Tarq0n 11 hours ago [-]
I believe that's all you get in copilot (365) basic.
quotemstr 12 hours ago [-]
The BOFH "IT admin" type of guy, an eternal scourge on the industry, has moved from delaying approvals of software you need to delaying approvals of updated models. It's management job to make sure these people don't burrow themselves into gatekeeping roles at an organization's expense
drdexebtjl 13 hours ago [-]
I have the exact opposite experience.
12 hours ago [-]
Doohickey-d 13 hours ago [-]
Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse...
wahnfrieden 12 hours ago [-]
5.6?? How is your impression of 5.6 relevant to anything recently?
ok_dad 13 hours ago [-]
5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop.
krzyk 12 hours ago [-]
5.6 was better than 6.
6.1 is a bit better than 5.6 from my tests
throwitaway222 12 hours ago [-]
Bro you have to be at 6+. 5.6 is EARLY JULY
seizethecheese 13 hours ago [-]
Yeah, in real SWE tasks, 6.1 is bad.
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
locusofself 12 hours ago [-]
I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me
seizethecheese 12 hours ago [-]
Yeah, our data only has 28 battles tagged "desktop":
Web: 87 (12.2%)
Systems and protocols: 78 (10.9%)
Software, unspecified type: 74 (10.4%)
Backend: 68 (9.5%)
Writing and documentation: 55 (7.7%)
Research and advice: 47 (6.6%)
Unknown: 42 (5.9%)
Mobile: 40 (5.6%)
Operations and automation: 36 (5.0%)
Developer tooling: 35 (4.9%)
Desktop: 28 (3.9%)
Media and design: 27 (3.8%)
Data analysis: 25 (3.5%)
Device and system support: 22 (3.1%)
Math and formal methods: 18 (2.5%)
Games and simulations: 15 (2.1%)
Infrastructure and DevOps: 9 (1.3%)
AI and machine learning: 6 (0.8%)
Other: 1 (0.1%)
aatd86 12 hours ago [-]
Yeah my main models are OpenAIs...
I am trying Opus right now because I dont have any credit anymore but from experience, it was never as thorough as openai's models. It can be genius and unblock at times, but I've never been able to trust it.
throwitaway222 12 hours ago [-]
I'm guessing you don't use codex though- the harness + model combo makes a difference.
seizethecheese 12 hours ago [-]
There was a post about that today on HN but it looks largely to be bogus. I wrote a blog post about it, but not linking here to avoid too much self promotion in this thread. Funny enough my conclusion was that OpenAI models seem to be the only ones that actually do have an own-harness interaction, hah.
slekker 12 hours ago [-]
Is this an ad?
seizethecheese 12 hours ago [-]
No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.
WarmWash 11 hours ago [-]
Using non-native harnesses sort of throws a wrench in things. I pretty sure your benchmark would look much different if outputs came from their intended harness.
seizethecheese 11 hours ago [-]
Interesting. Yeah we picked a neutral harness to keep things fair, but you're right that a native harness might be better. Would be a beast of an engineering challenge though.
igravious 11 hours ago [-]
Yeah, maybe one thread per article will do?, otherwise it comes across as shilling, you've mentioned your project four times now …
seizethecheese 11 hours ago [-]
Fair enough, though all the comments have either 1pt or positive karma, so I take that as a signal that they're adding value so far. I'll take a coffee break, though, hah.
Genuinely, just excited about what we're building and think it adds information about these models that are missing.
Do my comments get tiresome, though? None have been downvoted to hell, since they are all adding to discussion. If you think so feel free to email me, I don't want to be annoying.
10 hours ago [-]
foob 10 hours ago [-]
as long as its not hamfisted
You have six different comments on this story that all basically link to your product. That's a bit hamfisted.
arcanemachiner 12 hours ago [-]
Welcome to HN.
BTW check out my totally real project that helps you detect native advertising on your favorite platforms:
What kind of stuff were you throwing at it and where was it letting you down?
So what have __I__ been doing with it, then?
I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.
So yeah, 6.1-sol is good.
alastairr 9 hours ago [-]
My experience with personal / greenfield projects has been good.
Where it falls down is on the boring stuff - writing tight PRs for features / changes in large production codebases (data area in my case). Without a lot of handholding, tendency is towards 'bolting on' new code resulting in overcomplexity & missing obvious opprtunities for simplificatation.
tezza 9 hours ago [-]
Thanks for sharing. I do extensive direct comparisons, and use both Anthropic and OpenAI tools in anger on large professional codebases.
I think you are correct, OpenAI tools need more handholding.
However, I think that is actually intentional from OpenAI... they stick to what you the POH ask them and deliver what you ask. They embellish less. Anthropic tools by contrast embellish a lot, and they're very good too.
Perhaps start explicitly asking the OpenAI tools to embellish or think outside the box. That may unlock that experience and results you prefer. If you already did this, would you mind sharing more detail on what effect that had (or not) for your workload?
lnrd 11 hours ago [-]
Been using only Sol 6.1 on high/extra daily at my SWE job since it came out (with its official harness). It can basically do everything I need it to do. I still review its code and generally I have to leave around 10/20 comments in a ~700 line PR, flagging things it missed or weird decision. One or two iterations are enough to fix everything. Its still way faster than doing it myself, unfortunately.
hugmynutus 11 hours ago [-]
Sol-6.1 made me cancel my OpenAI subscription. Sol-5.X was bad, then 6 was barely an upgrade. Then 6.1 came out and it didn't move the bar. Went back to Anthropic.
General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.
varispeed 11 hours ago [-]
I wonder if these companies do selective scamming. For me Fable, Opus are unusable. They are completely dumb. I consider downgrading or cancelling subscription. Whereas Sol 6.1 got me my jaw dropped. Reminded me of Opus 4.6 before it got nerfed. It's so good I completely cancelled my Cursor subscription and uninstalled it.
scrollop 9 hours ago [-]
Hmmm, your comment is so odd as it is exactly the opposite of my use and the many videos I have watched comparing opus 5.5 (essentially GOAT until a few days ago) to other models.
Have a look at the videos comparing opus 5.5 to others.
Astra and sol are quite behind, hence I question your motives, unless that's truly your perception, which may be genuine, but honestly, odd and against the grain of many, many others' experiences.
hugmynutus 11 hours ago [-]
> I wonder if these companies do selective scamming.
I am 100% sure they do. My internal evals of Opus-5.5 dropped a few weeks ago, then recovered to launch levels once I got added to the security program.
Corence 12 hours ago [-]
Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.
"My AI is lazy" is honestly not a problem I thought I would have two years ago.
skerit 13 hours ago [-]
I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.
hrncn 11 hours ago [-]
I am not even certain 6.1 sol is better than 5.6 sol after using both for some time now, so not expecting much here.. but then again, 6.0 sol was completely useless, so maybe the .1 bump means something here.
suddenlybananas 13 hours ago [-]
You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.
ijidak 9 hours ago [-]
Dieselgate --> Tokengate (But run by the AIs)
aleqs 12 hours ago [-]
We know anthropic nerfs models all the time, openai probably does the same. There is no reason to believe the benchmarks/numbers they release are even measured with the same effective model that customers would get.
revolvingthrow 13 hours ago [-]
> Looks like they did time this for right before Anthropic's IPO.
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
origini 10 hours ago [-]
The IPO is 6 times oversubscribed and I bought some (we will see how many i get) but its a gamble right?
Either people really just want to buy and hold or they see it as a gamble chance like with SpaceX Superintelligence LLC.
CoolestBeans 8 hours ago [-]
Your analysis is right. If the Anthropic IPO is a bust it isn't likely there will be interest for the OpenAI one. Furthermore having a cash cushion to survive the price war will be paramount and private markets have tapped out so you need to get it from the public markets. However, a lot of people think the AI industry will be "winner take all" and if you think that way then hurting your competition is always good. I just don't know if that analysis of the industry is correct.
dgggh 11 hours ago [-]
It's too obvious. Anthropic still has time to respond.
arscan 12 hours ago [-]
My reaction as well - at the very least now is probably not the best time to knock down the realized valuation of a peer when they are effectively setting the market value of the whole category of ai companies, including yours… right?
kridsdale3 12 hours ago [-]
I guess it's a good idea for them to wreck the appeal for their own employees to leave for the competitor.
> I have no idea why they released this. It needed a lot more iteration.
Maybe the goal is to make Anthropic's models look good by comparison. :-)
platinumrad 12 hours ago [-]
Shumer is a known scammer (Reflection 70B, among other things). Why does he have any credibility?
tiborsaas 11 hours ago [-]
Anybody who says stuff like "They were essentially unusable." without any context or proof is going to my permaban on credibility. Even 1 year old models are perfectly usable let alone frontier ones.
aswegs8 11 hours ago [-]
That's what I thought. "Interesting negative vibe check". I open X, it's someone just saying they're unusable. Why is this top comment?
throwaway384034 11 hours ago [-]
Using a top comment to edit in a low information tweet on a big release should be treated as bad faith. Comment should be deranked.
Not to mention he didn't actually provide any reason why the models are bad. When pressed in the comments and asked what was bad, he said "everything"...
user43928 11 hours ago [-]
I saw another review from an early tester saying it's the worst model ever to work with.
The review claimed that the writing in the output is incomprehensible.
It also said that with /goal, the model set the goal to blocked to ask for the filename reporter_3423 should write to.
Lastly the reviewer said it made no progress on a large project.
Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.
Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)
dava96 12 hours ago [-]
just a fyi this crashed my browser
stbenjam 12 hours ago [-]
Which browser / version / os? And was it any particular bowl that crashed it?
It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive
letmevoteplease 11 hours ago [-]
Doesn't crash my browser but Gemini 4 Carbon High freezes up the page. The others run fine.
I thought it was interesting that all the Sonnet 5.5 have a weird light effect on the bowl. XHigh has a nice film grain too.
stbenjam 11 hours ago [-]
TY I'll put it behind a warning
kridsdale3 12 hours ago [-]
data point: no trouble seeing it on Chrome on MacOS (whatever the last one before the liquid glass update) on M5.
NiloCK 12 hours ago [-]
Seconded.
Pxtl 12 hours ago [-]
Poking around models crashed the tab in Firefox on windows.
fullstackchris 11 hours ago [-]
Wild. The opus ones literally look photo realistic... (Want to declare I am NOT a payed anthropic spammer, but look at those for yourself)
Also confirms grok is absolute garbage
stbenjam 11 hours ago [-]
Yep, seeing Opus results convinced me that maybe Anthropic is right on not neeeding an image gen model. It was very slow though.
Tadpole9181 2 hours ago [-]
I feel crazy - they look... Fine? Not even close to photorealism.
xtracto 12 hours ago [-]
>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
the_duke 12 hours ago [-]
> Looks like they did time this for right before Anthropic's IPO.
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
karimf 13 hours ago [-]
I'm very curious about this as well.
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
HarHarVeryFunny 12 hours ago [-]
RSI just means using the LLM as part of the development process, which they are already doing. I doubt any of these companies are going to fire all their developers, so there is no point development is going to be 100% automated - for the foreseeable future it will always be a mix of humans using LLMs.
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
That seems a bit of an arbitrary definition - all or nothing. Meantime they are of course already using LLMs to help development in whatever way they can, so if the day ever comes where it's 100% LLM then it's hardly going to look any different than the day before when it was 99%, etc.
Dario says a lot of things. He just said they'll cure cancer too.
antupis 12 hours ago [-]
Why IPO if you have RSI and possible ASI?
origini 10 hours ago [-]
Business as usual until proven otherwise. Everyone knows how to do R&D and IPO and VC, no one knows what RSI and possible ASI would do.
If you really mean it though and think about top x things to do first, i would probably try to research, invent and patent some superconductor and medicin.
ASI might not mean it can solve everything immmediadly but it could in x years with x amount of compute.
With an IPO and more capital, you reduce your risks
AlexCoventry 12 hours ago [-]
So you can scale up. RSI is going to be compute-bound. ML experiments cost compute. Also, if you can make a genius in a data center, why not make a hundred/a million geniuses?
esafak 12 hours ago [-]
So investors can cash out on the time frame they promised investors.
ppierre 12 hours ago [-]
Yeah we see some Really Stupide Intervention from those who drank the Kool aid. IAS2[] start to be a real thing.
[] IA Stockholm Syndrome.
Kostic 9 hours ago [-]
You should be aware that the individual you shared had a project called "Reflection 70B" that was, to put it mildly, quite disappointing and misleading.
giancarlostoro 11 hours ago [-]
Well, Fable also went down in cost too in their last update to Fable, and honestly, if they released a crazy new model on IPO day it wouldn't surprise me. Save the best for an impressive IPO.
imjonse 10 hours ago [-]
that negative vibe check is from the Reflection guy who in 2024 wrapped Claude API and presented it as some new breakthrough open weight model of his own.
ijustlovemath 13 hours ago [-]
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
IAmGraydon 11 hours ago [-]
Can we see a demo? There's a threejs subreddit that is absolutely overflowing with impressive LLM generated demos, in case you haven't seen it:
"Oh, the quality is bad because we uploaded the wrong weights."
i_have_an_idea 12 hours ago [-]
An Anthropic shareholder is very negative about an Anthropic-beating model release, right before the Anthropic IPO? Interesting, yes :)
6thbit 12 hours ago [-]
They're going after anthropic's market through cheaper intelligence per token.
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
bob1029 12 hours ago [-]
gpt-6.1-terra is already beyond end game for my client's business. They don't require anything more powerful.
Cost is the most important factor now.
Thrombocius 11 hours ago [-]
Thoroughbench Fireship’s horse animation test is actually a pretty nice vibe check.
easterncalculus 8 hours ago [-]
How much of that cost savings is just Code Mode?
ranyume 12 hours ago [-]
You're looking for a vibe check?
kylehotchkiss 11 hours ago [-]
Sincerely curious: why do people feel led to use the word "insane" to describe every new model release? I just don't understand what about incremental improvements needs to elict such a strong response.
We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?
intenex 11 hours ago [-]
Does it feel 'incremental' that the trend-lines suggest we're going to automate all of white collar work in <1 year from now and that every model generation gets meaningfully closer to that? https://x.com/maccaw/status/2120962037018771475?a=AqStEJ42mP...
Not totally sure what your point is. Is it incremental that OpenAI dropped the 722 math proofs or solved Navier Stokes or disproved ETH etc? Yes, in a sense it is and the trend-lines predicted it for ages (albeit it seems to be quite a bit earlier than the trend-lines predicted), but it's also a mind-blowing result.
When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two, yes, it will have just been incremental progress, but it also seems like somewhat of a relatively big deal, no?
thmoonbus 9 hours ago [-]
“When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two”
That would seem like a big deal because it’s not going to happen.
Feel free to return to my comment in 2 years time.
user43928 8 hours ago [-]
That's nice and all, but obviously progress has been unimpressive compared to Opus 5.5 that released two months ago.
I expected significantly bigger improvements from Fable after Opus 5.5, this did not manifest.
And OpenAI is apparently in a bad and desperate place.
crimsoneer 12 hours ago [-]
I'm just running it on my Blood Bowl Bench, holding it's own against the reigning champ Argon so far but early days...
None of these benchmarks/marketing numbers from openai or anthropic have ever held up.
Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.
behnamoh 12 hours ago [-]
> Feels like we're way past the pelican benchmark being useful at this point.
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.
teknologist 12 hours ago [-]
Beware of the paid influencers who don't disclose their affiliations in this space. Theo is another one who now talks smack about anything that isn't Claude. They bend reality to fit a narrative.
kalleboo 5 hours ago [-]
Even if they're not paid by the AI labs, they're bluechecks on X, they get paid by the view, so they stir up bullshit controversial takes just to get shared widely.
Someone1234 12 hours ago [-]
My big hope for this release is - Have they got their massive over-engineering under control?
Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.
And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.
scandinavian 12 hours ago [-]
I've been using both sol and astra to write some python and rust, and they generate the sloppiest code I've seen in a long time. I would be embarrassed to commit anything they write. The rust code especially was terrible. Had it implement a reader/writer for a binary format and it was just littered with magic numbers, terribly named variables, very compact code visually etc. It produces working code, so I guess the GPT models are mostly for people who let AI take the wheel 100%.
seizethecheese 11 hours ago [-]
I looked into Sol 6.1's performance by language recently (data from a head to head battle system built on OpenCode) and Sol 6.1 is actually has a decent win rate when it uses Rust (https://micjm.com/blog/gpt-6.1-sol-long-tail-languages ) so it probably has the same problem with any language.
bloopernova 11 hours ago [-]
What skills and prompt did you use? Did you plan implementation beforehand? Did you use test driven development?
I don't mean to come across as argumentative, it's just been my experience that the supporting docs and plan are very important.
persolb 11 hours ago [-]
A lot of the pushback is that, with Claude, you don’t need to do this much prompt engineering.
bloopernova 10 hours ago [-]
I admit that I haven't done much with Anthropic models. Mostly I use deepseek but I'm trying to add in a GPT and Gemini for code and plan reviews. (Not a lot of money means juggling smaller subscriptions)
adastra22 5 hours ago [-]
Anthropic models are at the point that they generate approximately as good code or better than I would have, on the first try. I’m talking about aesthetic and subjective choices, not success metrics.
The prose is still shit though.
jdw64 11 hours ago [-]
AI code uses magic numbers, but usually, if you just write something like an AGENT.md file, magic number generation doesn’t happen much, does it? Code difficulty or code problems ultimately depend on individual preference, so I can’t speak carelessly, but I think magic number related problems can basically be solved through AGENT.md or harness settings.
logicchains 12 hours ago [-]
You're using it wrong. If you want it producing clean code, you should be telling it to clean up and refactor the code, not just treating the first thing it spits out as the final product.
depr 10 hours ago [-]
Even if you do that, it doesn't reach the same level as Opus 5.5 unless you tell it explicitly how to clean it up, and that defeats the purpose.
_hyn3 7 hours ago [-]
That's what I've been doing wrong! I wish I'd known to ask for clean code.
SaltyBackendGuy 11 hours ago [-]
Just like humans honestly.
water-drummer 11 hours ago [-]
Make no mistakes
vbezhenar 11 hours ago [-]
> magic numbers, terribly named variables, very compact code
Sounds like I'd like it. I'm really tired of claude introducing pointless constants everywhere.
hermitShell 12 hours ago [-]
Do we want LLM generated and maintained code to look similar to human generated and maintained code?
I've always used AI tools to write very constrained and small functions and modules, and then stitch it all together myself. So far. Over the last year it's become evident that many people let the AI drive 100% and there are proponents (DHH notably) that this is the way.
I'll be interested to see how compilers and LLM's compare at generating assembly over the next decade. Assembly offers enough abstraction, why do we need C? \s
andoando 12 hours ago [-]
I think even for LLMs it's going to be much easier to analyze code, and especially its intent with proper naming vs trying to figure out what a bunch of assembly is doing.
A function called binarySort is alot less tokens and communicates what the function is attempting to do vs 100 lines of assembly
seizethecheese 12 hours ago [-]
I was skeptical Sol 6.1 actually over-engineers compared to Opus 5.5, as I just switched from one to the other and haven't noticed a difference, but it looks like you're probably right!
We're building a battle system for OpenCode (https://agentduel.com/leaderboard) and have data on how large the diff was compared to opponents for both:
Opus 5.5: 0.42x (based on 299 battles)
Sol 6.1: 1.42x (based on 265 battles)
Data on this is still pretty small. P=0.14. It's also affected by large differences, this is average not median.
yegle 11 hours ago [-]
FWIW Carbon is listed in the leaderboard but not in the GitHub repo.
seizethecheese 10 hours ago [-]
thanks. That’s my personal fault since I added Carbon to private repo but didn’t make app repo readme PR. It’s a pain having two separate repos.
jeffybefffy519 11 hours ago [-]
This is pretty clever, i like it.
seizethecheese 11 hours ago [-]
Oh man, Sol goes nuts on tests. I just checked the test wriging data. Sol 6.1 writes 3.14x more tests than it's opponents.
arw0n 10 hours ago [-]
I have inherited a multi-repo project that is actually really well designed, and I like the clear separation of concerns between components and very solid APIs connecting them. But since they have individual deployments, and could get out of sync, Astra/Sol keep insisting we have huge json contracts generated by unreadable bash scripts, enforced by custom CI jobs, and supported by a huge number of tests.
I've seen two projects where developers didn't curtail this kind of overengineering, and recently had to spend a very significant amount of time and tokens to debug their deployment issues, because even for advanced agents there's a point where each issue could be caused by 12 things. This is especially pronounced in DevOps, but I see it in code as well; the suggestions are all reasonable individually, but you quickly end up with a code base that is too complex for a human to reason about, and it currently only kind of still works with agents because capabilities are growing so quickly.
jakevoytko 10 hours ago [-]
I gave Astra full control over the design of a software factory for a few days, as a side project. Then I switched to making it justify every change that it wanted to make. It solves problems that don't exist, has tons of random verifications that don't add value, selects tortured architectures instead of asking if it can relax its assumptions a bit, etc.
Typical convo:
Me: "Hey, I see behavior X, which seems like a bug. What is causing that?"
Astra: "That's caused by a verification failure; we should be verifying the hash of this JSON output along with a permanent audit record. Let me know if you'd like to make that change."
::ask it what caused X to happen 6 or 7 times::
Astra: Tool use was limited to 1GB but the Rust compiler needed more than that, and that caused some PROBLEMS
Me: Remove that constraint
Astra: OK
ibejoeb 11 hours ago [-]
I have experienced Sol 6.1 writing tests to check values of environment variables. Not the structure, or range of values, or anything useful like that. Like `assert(DATABASE_HOST == 'localhost')`. Gotta wonder how something like that was learned.
pettijohn 12 hours ago [-]
Oh my god yes. 6.x has been terrible at overcomplicating things. I have to prompt it to find the simplest solution that meets the requirements and then ask it to simplify and tell it to STICK TO THE PLAN. In addition it feels like they overfit for persistence. I created a new work tree and forgot to copy .env files. I caught codex hammering away and docker and psql commands and stopped it and asked it to explain what it was doing. So now my agents.md says "if documented tooling fails STOP and seek human assistance."
pennomi 11 hours ago [-]
Until benchmarks punish agents for writing dirty code, it will continue. Right now “completing the task” is more important than “don’t damage the codebase”.
buybackoff 11 hours ago [-]
I've been fighting with this Sol/Astra tendency. I noticed that Sol Light is less bad with that, the higher the effort the worse that bias to overengineer is. It takes time to understand the limits, but a dumber model that is faster and does less annoying things was better for my productivity as default.
In addition, I have spent some time to fine tune my custom instructions for Codex. It went like every time it violated my desired behavior, I asked why it did not understood my instructions and did this and that. In the end, the resulting instructions seem to work.
In the process, the model explained that the master instructions built in the Codex just tell it to push to the end and do as much as possible to satisfy a user. And these master instructions have precedence, so it's better to put the custom instructions into the developer_instructions section in the toml config file, as it has more weight, and not (only) in AGENTS.md. I ended up having both, with more details in the system AGENTS.md.
andy_ppp 10 hours ago [-]
Be great to see a gist if it’s not too private?
afro88 12 hours ago [-]
I have been bitten by this 3 times and each time Opus 5.5 brought it back to something sane. The last time I left Opus in a /goal to get behaviour matching the docs, and it decided on it's own that a rewrite was the best path (and it was, and it worked)
contextfree 11 hours ago [-]
Man, I haven't tried any Astra yet but I find Opus 5.5 rather bad about this already.
holzhaker1 12 hours ago [-]
yeah i feel like with astra they went all in on 3D modelling which encourages "just one tweak, just one more tweak" which makes it detailed enough, but in coding that same behaviour just sucks.
inexcf 12 hours ago [-]
And it is absolutely crazy then that Astra sucks at 3D modelling compared to Opus 5.5. It is not even comparable and Astra needs 10x the usage to do it.
riazrizvi 11 hours ago [-]
I don't mind it too much. Refactoring the bloated poorly integrated code is one way I stop my skills atrophying.
It also reassures me about the longevity of my career. I have sustained a theory that there simply isn't the data to teach these things to be really good, so I love to get the evidence to back it up.
jeffybefffy519 11 hours ago [-]
This is why i stopped using GPT 5.6 onwards... in my view GPT 5.3 Codex is when they peaked for coding.
geonic 11 hours ago [-]
Exactly my experience. Codex loves to generate unnecessary complex solutions. Having Fable level intelligence at 5% the cost is a very good reason to switch though.
EspressoGPT 11 hours ago [-]
If it has that intelligence. I've been excessively using Astra and Sol in the last couple of days and Astra is not creative when it comes to finding unexplored/new ways. It seems to be abstracting over what's already there.
pants2 12 hours ago [-]
Lol I made the mistake of having Astra design and build a complex system without any input (experimentally). Seemed functional at first and shipped it internally and it had dozens of complex bugs, so I started digging into the architecture and it was a monstrosity! Definitely 10X more complex than it needed to be.
EspressoGPT 11 hours ago [-]
Same here. Now I had Opus have a look and it suggested that the sheer amount of code was ridiculous given the early product stage.
myworkaccount2 12 hours ago [-]
I was about to fall into the same trap. I wanted to try the "smart" model (astra), and made the mistake of thinking benchmark smartness is the same as designing sensible systems.
After having it write around 13k loc i had to tell it to remove most of the complicated bs, we went down to 3k loc.
x3haloed 10 hours ago [-]
I miss 5.6-Sol’s attack dog nature. Everything in the 6 family just feels like it’s sleepy.
AaronAPU 10 hours ago [-]
5.6-Sol was the last model I preferred over Anthropic. But now Opus and Fable are both better. Curious to see if 6.2 Sol brings it back, but afraid to waste cycles testing.
mpalczewski 11 hours ago [-]
opus 5.5 hallucinates all the time. and tends to be overly verbose. and no you can't prompt your way out of it. you need checks and verifications.
sigseg1v 10 hours ago [-]
Curious what you cap your context window at? I go between a cap of 150k and 200k context and I don't feel it hallucinates all the time (sample size about 19,000 commits I've had it write)
mpalczewski 58 minutes ago [-]
same. greater context was worse. it regularly creates new requirements. inserts editorializations into docs.
CSMastermind 10 hours ago [-]
This is 100% true by the way.
I actually have the problem with Fable stopping too early on tasks but Astra way overengineers things.
incrudible 12 hours ago [-]
I have limited experience with OpenAI models, but I have to say that Opus 5.5 has not improved on the terrible writing style, and I am struggling to prompt my way out of it. It always needs an extra pass dedicated just to rewrite all the prose.
teach 12 hours ago [-]
I've also given up on allowing Opus 5.5 to write basically any prose or technical documentation.
My favorite model for this at the moment is Gemini 4 Argon, which writes very naturally and isn't too expensive. If you're not using Cursor or some other harness that allows you to choose non-Anthropic models, then I don't know what to suggest.
senderista 9 hours ago [-]
I don't find Opus does a good job on the rewrite pass either, I delegate that to OpenAI models (I've heard Gemini is even better but don't have access).
jpardoe 12 hours ago [-]
I’m sorry but this is crazy, I think you’ve forgotten just how bad 5 was. It’s not perfect by any means but it’s a clear improvement.
surgical_fire 11 hours ago [-]
> Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution.
I have to correct stuff Opus suggests during planning every single day, not even for major tasks. Very useful still, but last thing I would suggest to anyone is to trust its "gut instinct".
Small wonder why we have news of outages everyday now.
tamimio 11 hours ago [-]
Spot on, identical to my experience. Sol 6.1 was a total failure that I won’t even consider it a frontier. I tried it on few problems at xhigh, it kept going miles after miles around the issue, creating massive unnecessary code and files and documentations and tests, even tests for test too, burning ~2mil of tokens over a week, then I fed the same problems into opus 5.5, less than half an hour and solved it, pinpointing where the issue was, where sol was wrong, and far cleaner code and output.
risyachka 10 hours ago [-]
>>> yes, you can prompt your way out of anything.
you really can't.
unless you are working on something simple or what has been solved 100 times before
uw_rob 12 hours ago [-]
Honestly I don't even know if you can prompt your way out of these types of behaviors - stylistic tics and judgement seem to be extremely difficult for the models to overcome. It seems we are tied to what the labs do unless we want to dive into and get dirty with models yourself. In retrospect with time it becomes obvious that the previous generation of models couldn't be prompt engineered to do what the current generation of models do. There's just no way you will prompt engineer Opus 4.8 to write as well as Opus 5.5. I'm guessing with time we'll see that there was no way you could prompt engineer Sol 5.1 to be as token efficient as Sol 5.2
contron 12 hours ago [-]
I have found no way to overcome these kinds of things myself, even when showing the model examples of what not to do. Theres not really a good incentive for the labs to train them not to. If the model can overengineer, you spend more tokens. If you then prompt for the model to slim things down, thats more spend. If anything should come from the next generation of models, I hope its a sense of brevity and TIGHTNESS.
energy123 6 hours ago [-]
I've gotten good results by dedicating follow up prompts just to trimming the fat and ensuring simplicity/reuse before implementation as the only thing it focuses on for one or two rounds.
Also to allow it to express any concerns or highlight possible areas where we might be making an assumption about my intent.
joenot443 13 hours ago [-]
The video is pretty funny. It's like a callback to 90s/00s office life?
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
Yeah it's 100% a nostalgia callback to this commercial, right?
jhickok 9 hours ago [-]
Yeah, has to be.
EcommerceFlow 12 hours ago [-]
first thing I thought of too
apetresc 13 hours ago [-]
I guess it's kind of funny in a tacky way, but pretty much every scenario is just "oh shit I was too lazy to do my job and now I'm about to be called out, better throw it into ChatGPT literally ten seconds before I get in trouble."
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
ilaksh 12 hours ago [-]
What do you think would be a non-tacky and effective marketing approach targeting office workers?
black_knight 13 hours ago [-]
Feels like this is exactly what people do in meetings I am in these days. "Claude give me this…" is the start of a lot people's contributions to the meeting. It is then followed by the most inane, over-designed artifact with no real content.
ceroxylon 12 hours ago [-]
It is bizarre and bordering on offensive, if an employee throws some last minute slop at me, I would wonder why I am enabling a meat proxy and not prompting the LLM myself.
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
Bluestein 12 hours ago [-]
Heck, and at least in the 90s ad, at least one of the parties involved seemed to have some actual, human-gathered, reliable information that, sure, they threw into the cell chipper during the elevator ride, but at least they did not hallucinate.-
cmrdporcupine 11 hours ago [-]
Yep, plus a pile of "ha ha look at the GenX old timers with CRTs" age discrimination content.
noir_lord 13 hours ago [-]
Funny but Dystopian, the message is "Half arse your job, the clanker has you".
astro-lizard 12 hours ago [-]
the message is definitely intentional, the goal is to de-skill office workers.
pmg102 12 hours ago [-]
You sure the message isn't just aimed at the median (ie low skill) office worker who hungoveredly half-asses this stuff all the time?
"Now a robot can half-ass it for you so you don't have to!" is the perfectly targeted message for them.
aenis 9 hours ago [-]
Big progress, for my types of workloads.
When given my benchmark suite, which involves a couple of long horizon tasks where the agent decides when to consider the task well executed, its the first OpenAI model that does not want to finish as soon as some minimal level of quality has been achieved. Like Fable, Opus and Sonnet it now runs the maximum allowed iterations and does not stop at 'satisfactory', gold plating the solution to my delight.
(I don't work like this on work projects, but for everything hobby, I am testing how much autonomy I can give to the coding agents, and thats what my benchmark suite tests; I have the basic suite which completes in 2-4hrs, and the long running suite which takes about a week or so; the other one I just triggered, but the first iterations look way better than prev gen Sol/Astra at the same effort level).
undershirt 9 hours ago [-]
really curious what your benchmark suite looks like then
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
This might be in other pelicans, but 6.2 Sol/Astra is the first time I'm noticing a model shade the leg further in the frame darker to mimic depth (or a shadow).
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
glenjamin 12 hours ago [-]
Interestingly, on the mech thing it seems like only Fable really nailed the brief - the others might look cooler, but Yonkers and the mech aren't directly connected in the same way.
Melatonic 1 hours ago [-]
Odd that the smartest Astra starts making the left arm/wing drive the bike with the right side (toward us) facing backwards like a bird would normally look
ryanschaefer 12 hours ago [-]
I think this is the first time I’d call the pelican benchmark solved. All levels of effort look good.
arcastroe 11 hours ago [-]
one defect I noticed is that the back leg still bends a different direction than the front leg in many of those images. In both Astra and Sol.
Something a human can recognize as inconsistent, but not the SOTA leading frontier model.
gibspaulding 10 hours ago [-]
I find it somewhat amusing that it can’t quite decide which way the legs should bend since it actually is pretty debatable. Birds knees bend the opposite way ours do (technically it’s their ankle, but it’s what you’re looking at here), it just looks bizarre to draw something like that in a bike.
Maybe the next models will train on these comments and change the bicycle design itself to better fit how a Pelicans legs would locomote
lukevp 6 hours ago [-]
But is the model given the visual and an attempt to iterate? If I sat you down at a keyboard and asked you to write raw SVG code would you be able to draw a pelican? That’s the evaluation, not the analysis of the resulting image.
TiredOfLife 10 hours ago [-]
You need Sol at minimum xhigh to get 2 legs.
While Astra only low looks good to me.
prmoustache 11 hours ago [-]
Pelican is one legged on gpt 6.2 sol below high and has both legs on same side of the bicycle on high.
fekunde 11 hours ago [-]
Wow, surprisingly Gemini 4 Carbon really nailed the Yonkers prompt. I like how the sections of Yonkers held shape in the mech. Fable tried but gemini is more detailed.
iandanforth 10 hours ago [-]
It did not. A robot appears and then bits of a map stick to it. Only Fable did well.
RGS1811 12 hours ago [-]
The emergence of the beach setting and the scarf is interesting.
kzrdude 12 hours ago [-]
I think a scarf or similar is relatively common with bicycle cartoons, to indicate speed or movement.
okrad 11 hours ago [-]
Looks like high/xhigh both get the back leg z-index incorrect relative to the back wheel of the bike and frame.
DenisM 9 hours ago [-]
The frame/leg problem was discussed here previously and likely trickled into the training set.
The problem you pointed out was never discussed on hacker news, and it clearly shows they still lack spatial reasoning. I wish you didn’t mention it. :)
boutell 12 hours ago [-]
Fable gets at the mech Yonkers idea in a much more entertaining way than the others.
prescriptivist 11 hours ago [-]
Turning Beech Hill into shoulder mounted cannons is pretty clever.
adverbly 12 hours ago [-]
What's with everyone saying these Pelicans are great?
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
preommr 12 hours ago [-]
I have to agree, those results are surprisingly bad and I am not seeing an improvement.
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
svachalek 12 hours ago [-]
The highest Astra images give the pelicans wings instead of arms, which does seem like a valid interpretation.
adverbly 11 hours ago [-]
Do you mean on X-High where its got both legs on one side of the frame?
Or on Max where its for some reason only using its left arm/wing - which looks totally different from its right arm/wing - and is grabbing the right side of the handlebar with its left hand for some reason?
Look. I think at first glance these look pretty good. I'd be astounded with this level of quality a few years ago.
But people saying that this benchmark is done are completely missing the point that accuracy, consistency, and details matter.
I'm not willing to say this benchmark is done until I see consistent and accurate details.
foltik 6 hours ago [-]
It’s a recurring theme with all of these generative models. The images, UIs, music, code, prose, etc. that they produce all seem high quality, but only if you squint.
Quick glance at the table, wow pelican benchmark is solved! Oneshot a complex web app, wow software engineering is solved!
Of course upon closer inspection it’s always an incoherent mess. A poor imitation (i.e. lossy compression) of quality.
globular-toast 12 hours ago [-]
Do we know why they always end with a scarf?
phoghed 11 hours ago [-]
They are a classic motif for pilots, birds fly, and there are famous cartoon birds with a scarf, like the ones from The Rescuers or Duck Tales come to mind.
pinkyclouds2 12 hours ago [-]
winter is coming
12 hours ago [-]
boutell 12 hours ago [-]
Interesting that pelican quality does not go up monotonically within Astra. Honestly, I think low was the best one.
tintor 12 hours ago [-]
Sol 6.2 max is still failing on pelican benchmark. Left leg is jammed in chain.
Astra 6.2 max did a similar mistake on the left bicycle pedal.
frangonf 12 hours ago [-]
But Astra 6.2 low and medium didn't make this mistake.
BrokenCogs 11 hours ago [-]
No idea what a yonkers is but that fable 5.2 animation blows the gpt 6.2 models out of the water
dllu 11 hours ago [-]
Pelicans still not saturated. No chain or front fork offset, and Astra added a rather disconnected detail on top of the pelican's head. Still, we've come a long way since the early days. Very exciting for the future of pelican cyclists!
clzerv 11 hours ago [-]
The Fable5.2 version is the only one that allows user interaction with the model, it's also very clean and minimal. I've been using Sol-5.1-high for a dumb web project and it tends to create really baroque code and UIs. I haven't used Claude in a while and this makes me want to try it again.
It seems like the models have a kind of built in aesthetic sense and I wonder what part of the training influences this. Maybe if more RL time was spent on artistic tasks the models could not only make better visuals but also code.
Currently they all seem to care little about code quality and maintainability (by humans), just functionality, which is fine I guess if you can expect to vibe code forever. But what if there is a critical bug and the models are temporarily unavailable for some reason? It would be hopeless to expect a human to wade through the giant bowl of clanker soup and fix it anytime soon.
zapkyeskrill 7 hours ago [-]
Astra xhigh .. so not quite ready to take over the world then.
BoldColdHold 12 hours ago [-]
Wow the yellow is leaking into their llms too.
firemelt 12 hours ago [-]
any index link for all ur pelicans i want to see fable and opus
They all seem very good, with a lot of attention in detail !
Thx for the Pelicans!
EugeneOZ 12 hours ago [-]
still the best benchmark! thanks
p11p 13 hours ago [-]
Are you getting any signal out of this anymore? All I can see is "good pelican" and "dogshit pelican". These ones look pretty good.
EugeneOZ 12 hours ago [-]
I do. 6.2 Sol xhigh and max show the results comparable to Astra 6 max, and Astra 6.2 only shows any improvement on "max" reasoning effort level.
adverbly 12 hours ago [-]
You're kidding right?
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
jay_kyburz 12 hours ago [-]
When are the Pelicans going to stop being rendered 2d like a child's drawing. We are a million miles away from what a professional artist would come up with.
Update: Here is some Bluey characters riding bikes as a reminder how far these models have to go.
I'm not sure if this is a useful benchmark any longer
InsideOutSanta 12 hours ago [-]
They're extremely useful. In fact, I refuse to make infrastructure decisions in a pelican-blind environment.
jorl17 13 hours ago [-]
I really want GPT models to be good, but Sol models are so slow I cannot use them. Maybe the 500$ subscription (or enterprise packages) are better, but it is unbearably slow. I've tried the fast variant and low reasoning levels and there isn't a significant improvement.
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
ativzzz 12 hours ago [-]
Does speed really matter? Most of the time I get the AI to do something, I immediately switch to another one of my things I have in progress and then come back at some point later. I'm never sitting there waiting for it to finish
edvardas 12 hours ago [-]
Personally, my cognitive RAM is limited, so I'd rather work on a few tasks at a time.
The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed
kristofferR 7 hours ago [-]
Try T3 Code, it's way better to manage your brain RAM with it. The sidebar is the best in the business, it's so nice to be able to settle/snooze threads.
treis 12 hours ago [-]
It'd be a lot better if you didn't have to switch. It's not nearly there for coding tasks but for other things instant responses are a game changer.
Pedro_Ribeiro 12 hours ago [-]
It depends on the task. For frontend work, for example, it's table stakes.
epolanski 11 hours ago [-]
Yes, if you want to do something interactively and the task deserves more than the usual slop AI produces, speed matters.
jorl17 12 hours ago [-]
It depends on the project and task, but for the kinds of projects I'm throwing at it, it matters quite a lot. I have often been focused on building just one project, without the possibility of many parallel tasks due to machine contention (and unfortunately offloading to the cloud is not an option). This means all tokens go to the same project, and they all go sloowwwwwly.
The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.
easterncalculus 8 hours ago [-]
When RL-fried GPT models just don’t actually do the work or choose to do something different for 40 minutes speed matters a ton.
0xcoffee 17 minutes ago [-]
I can't say this has been my experience. I played around with Luna/Terra and DS 1017, but while the others tend to start spewing out tokens quicker, I find Sol generally does the 'right' thing while the others go into the weeds. So overall I finish my work quicker with sol
vtail 11 hours ago [-]
Vibe-checking the new GPT 6.2 Astra: I asked it to audit complexity in my production code, and then discuss the result with Fable 5.2.
Astra's had identified 12 issues, including 8 P2s, but that changed to just 8 P3s after the discussion with Fable. Fable was able to persuade Astra that it's OAuth reasoning was incorrect ("had a missing link" as Astra put it).
So: overly eager but not very diligent, and (based on this little anecdote) doesn't really offer a compelling reason to try it more.
EspressoGPT 10 hours ago [-]
> So: overly eager but not very diligent
Sounds like GPT-6 Astra.
amluto 12 hours ago [-]
I am right now being reminded of a problem with GPT-6 Astra: it surprisingly regularly completes part of a (not very complex) task, gets a bit distracted, and declares itself done. The fact that the original task is not at all complete doesn't seem to bother it, and it kind of seems like this bad state is sticky in the context and it struggles to recover.
I wonder if the newer models are better about this.
valdiorn 9 hours ago [-]
This was the biggest shock when I moved to Anthropic models. I didn't realise that there were models that could do actual long-term work. I'd give chatgpt a 5 stage project and after doing step 1 it would just go "yeah I'm just quitting now, if you want to keep going here's what you need to do, but I'm done working". It would just never work more than 20-30 minutes.
jswelker 6 hours ago [-]
That was the case 6 months ago but not since Sol. (Terra and Luna definitely do though.)
Although Sol does still give up after a few hours when there are still many items left on the to-do list and tell me what next steps are instead of just doing the next steps.
clarus 11 hours ago [-]
Ask it to create and run a goal about the task. This will make it stick to it.
amluto 11 hours ago [-]
I’m sure this could solve it, as could a fancier harness.
But I’m talking about a task that is a two-sentence prompt. I would like the model to be able to keep track of straightforward instructions without harness assistance :)
But the issue they’re describing isn’t what I’m talking about. If the model wants to stop and request clarification, fine. In fact I sometimes explicitly ask for this.
But sometimes it actually thinks it’s done when it isn’t. This isn’t “I did a step and I have a question”. This is “I did what you asked” and then, after two more prompts, “oh, sorry, I did not complete the task”.
drnick1 12 hours ago [-]
Recent Anthropic/OpenAI models are increasingly displaying warnings or outright refusing to perform rooting, hacking, and binary reverse engineering. At this pace, they will be completely useless soon at entire classes of tasks I have found them most useful for.
kccqzy 7 hours ago [-]
What? Just today I forgot to tell Opus 5.5 where the source code repository is in a multi-repo setup, and Opus 5.5 proactively disassembled the binary to figure out what the code does. I didn’t even intend to have it do binary reverse engineering and yet it did that.
oilkillsbirds 12 hours ago [-]
Good.
sunaookami 12 hours ago [-]
GLM is perfect for these kind of things. GPT and Claude shit themselves even on stuff for very old devices that are no longer sold, nor "normally" usable.
drnick1 12 hours ago [-]
> GLM is perfect for these kind of things.
Which provider? I found the z.ai lite plan to be pathetic. Consumes 5h and weekly limits extremely fast when driven by Codex, and now also refuses RE work.
Computer0 10 hours ago [-]
Opencode go
Systemerror7A69 11 hours ago [-]
Mistral is good. "Only" 250 Euro, but GLM is cheap af, and mistral has absolutely zero of these weird-ass "5h limits" that became the defacto standard for some reason.
summarity 11 hours ago [-]
DeepSeek Pro is also happy here. Or at least happy to take your word for it "OK, I assume the user is authorized ..."
InsideOutSanta 12 hours ago [-]
Fingers crossed that Zhippu doesn't jump on the "we've nerved security" bandwagon.
goyozi 11 hours ago [-]
My dad was curious about AI game building skills and now I have my own useless model benchmark - building a breakout game similar to DX Ball we used to play many years ago.
I use both astra and fable quite a bit for my work and I find that astra is much weaker with maintaining long horizon tasks and making decisions on my behalf / exposing those to me when appropriate. Astra seems more likely to do exactly what I ask for but nothing more, even if there is an obvious add on task it should have suggested. It often does not report on what it has done as comprehensively. I am admittedly more experienced with fable but is this because of something I’m doing? Am I missing some key feature of Astra?
9 hours ago [-]
zzleeper 7 hours ago [-]
I run an internal OCR benchmark for historical research (old texts.. newspapers, typewriter, but also handwritten)
Google is BY FAR at the top of the Pareto frontier, perhaps b/c of Google Books.
But Sol 6.2 (low) is the first non-Google model that grazes that frontier. Sure, it's still more expensive than Hydrogen, but not too far! (and that's at the current low Hydrogen prices)
YuechenLi 11 hours ago [-]
The heavy focus on 3D modeling and CAD is what I've expected from OpenAI recently ever since the news that they disbanded their standalone robotics team, as I would expect them to go all in on physical AI and game/visual asset generation via LLMs.
One of the consequences to that is I think the first wave of AI CAD startups are getting hit by the bitter lesson right now, including the startup companies that trained their own VLM for CAD like Zoo or companies that tries to build CAD specific harness like Adam.new/MecAgent, because the increase in capability from GPT/Claude/Gemini for 3D modeling as a specific area of focus for benchmarks means that the early moat gets eroded super quickly.
01100011 11 hours ago [-]
Is this version still hamstrung by endless Daybreak false positives? It’s becoming impossible to use 6.1-sol to fix bugs because of the endless daybreak warnings which block me from seeing the output. These aren’t security related bugs either. Our software has no security model and none of these bugs have any relevance to security.
samtp 8 hours ago [-]
I've only seen 1 in months of using it constantly with multiple sessions running at once. And just switched to DSF4.1 to finish of that one task.
kingstnap 13 hours ago [-]
Seeing them keep prices low is awesome. Was worried for a while that it would be like the GPT5.x series where progressively the models just got more and more expensive.
minimaxir 13 hours ago [-]
The GPT-6.2-Astra 20% price reduction is a match to Fable 5.2's 20% price reduction (although Astra is $0.40 MTok cache read vs. Fable 5.2's $0.20 MTok cache read).
jbellis 13 hours ago [-]
Good point, but also note that OpenAI has the same "discount rate" on subscription for Astra as the other models, while Fable's is much much lower
Opus: about 50x discount (98% off)
Sonnet: about 40x
Fable: about 5x
All GPT: about 20x
Rperry2174 13 hours ago [-]
I feel like the fact that they're not public lets them play the subsidization game though. And the same playbook that Anthropic started at that.
Would be awesome if it stays this way though
oh_no 13 hours ago [-]
we've seen anthropic numbers, inference is not being subsidized by these companies
mrandish 12 hours ago [-]
> inference is not being subsidized
Inference doesn't have to be sold below direct compute cost to be subsidized. If inference pricing doesn't cover amortized capex and the cost of capital it's still using investor money to buy market share.
suddenlybananas 13 hours ago [-]
Did you read what they wrote carefully?
charlesabarnes 13 hours ago [-]
Its been stated often, but I appreciate that OAI is now bringing the topic of conversation to "task completion cost" rather than brute forcing solutions
ksec 10 hours ago [-]
As someone who dont follows closely. How do I know at what point were the other GPT model tested?
Let say they just nerfed GPT 6.1 a few weeks ago. Now they are using that as a reference score against 6.2? Or were those results tested when 6.1 relesed. What about 6.0?
And that is the same with Claude. How do we know they are not cherry picking the date and test on these release when they are constantly "enhancing" their model.
We also know some of these benchmarks are largely irrelevant once they were known ahead of time and become a game of benchmaxxxing. It just follows Goodhart's Law "When a measure becomes a target, it ceases to be a good measure."
So how should I know what has actually improved?
therealdrag0 10 hours ago [-]
There are benchmarks that retest over time and look for nerfing. iirc they disconfirm the vast majority of suspicions about this.
I've been testing sol, and opus, almost weekly as new models are released. I haven't seen any evidence of the nerfing conspiracy theories. https://slopcop.com/power-ranking
Edit: But it does look like you get about 20% less tokens for your subscription with OpenAI than you did until a couple months ago.
0o777 9 hours ago [-]
Do you use subscriptions or API for this testing? I suspect it's only the subscriptions that are impacted.
jbellis 8 hours ago [-]
As an independent, I can only afford subscription-based testing. T.T
Had some major issues with 6.2 Astra this morning. Got stuck in a loop where it was taking a relatively simple bug fix and going deep into the weeds with overly-complex invented edge cases. It was in a running loop for 15 minutes, I stopped it, gave it another prompt - still couldn't get to a simple solution. I switched back to 6-Astra and it had no trouble.
superjared 3 hours ago [-]
What effort?
reassess_blind 3 hours ago [-]
High. It's happened again twice just now. Thinks for extended periods of time, seemingly gets stuck in a loop. When 6-Astra High takes over it reverts and does the same fix in 30 seconds.
mchusma 10 hours ago [-]
Just tried it side by side Astra to Opus, Astra was not very eager and Opus crushed it on my one test. I asked it to look at the other hacker news post about speeding up CI, and speed up my CI. Opus made a bunch of good solid changes. Astra just kind of told me this a suggested problem area. I had this issue with 6.1 (sol and astra), its hard to get them to do work. They do well when supervised by Opus though.
The best feature of 6.1 sol was it would actually do stuff for you that Opus refused though. I did use that often.
blm126 9 hours ago [-]
You and I have the exact opposite requirements. My workflow is that I always like to talk through the problem and the solution first, before codex just starts ripping through the coding portion. I used to have consistent issues getting that kind of workflow. The more recent models from OpenAI are so much better at building a plan collaboratively.
enlyth 9 hours ago [-]
I wonder if in the future we'll get to adjust a bunch of sliders about our AI's personality, not just effort level. Would be cool.
alienbaby 9 hours ago [-]
curious, i haven't experienced what you describe so far. Astra and Sol 6.2 have both gone off and done jsut what I wanted, and as the set of prompts I test against become more vague and handwavy, they both went on to fill in the gaps and produce somehting I was satisfied with.
HyperL0gi 13 hours ago [-]
My company can only hold one subscription per employee, and I still can't find a reason to go back to OpenAI after migrating to Claude once Opus 5.5 was released. It's super efficient, I almost never run out of tokens with the Teams Premium seat, and Fable is fantastic. They lost me when Astra was released and tokens simply evaporated with a few prompts.
firemelt 12 hours ago [-]
ehat effort do you use ur opus5.5?
anentropic 12 hours ago [-]
Medium just works for most stuff
placebo 11 hours ago [-]
I'm still amazed by the pace at which these models are released. On the same news page, just a few hours ago Microsoft announces its own competing frontier model. Interesting times.
I did some first hand tests of 6.2 Astra just now. I had a task that doesn't reproduce on the system I ran it (chosen purposely), ran both 6 and 6.2. The 6.2's responses were noticeably better. Instead of using the extremely long and rather annoying format 6 used, 6.2 led with the most important things first, the plan sizes were readable, and suggested a reasonable fix. Still doing some more tests
EDIT: also 6.2 is seemingly faster than 6, could be due to more selective context ingestion and smaller context window though.
wds 13 hours ago [-]
The promo video seems to be a pretty strong rip on the 90's Microsoft Excel elevator ad. [1]
The “training data,” as in, the humans who made the video were inspired by it?
astlouis44 12 hours ago [-]
GPT 6.2 Astra seems to be setting a new standard for 3D game creation! This is a Godot WebGL game linked in their announcement post, gives Wipeout vibes! A lot of fun to play too and visual are top notch.
It is not a lot of fun to play. Like all these games it is much too easy.
Foobar8568 10 hours ago [-]
Funny, I thought at first...oh a boring F-Zero /yawn
claiir 13 hours ago [-]
> Due to their increased capabilities, we’re launching GPT‑6.2 Astra and Sol with tighter safeguards for higher-risk cybersecurity assistance.
This is unfortunate—that was one of the big advantages of Astra.
jeffbee 12 hours ago [-]
I can't even get 6.1 Sol to summarize the log of my own fuzzing fixture running on my own project. I have to use Gemini models to handle my fuzzing tasks. It's ridiculous.
claiir 12 hours ago [-]
Ran into a false positive within the first two minutes of using 6.2 Astra.
unixhero 12 hours ago [-]
System engineering = Fable 5.2
Programming = Fable 5.2
Office work = Fable 5.2
I haven't found Astra to be better at any of these
iforgotmypasswo 6 hours ago [-]
6 or 6.2?
minimumvector 11 hours ago [-]
I haven't found Astra to be better than Claude yet, but my dot replaced Muse 100% for me. So maybe GPT will continue winning consumer, while Claude continues winning coders and enterprise.
czhu12 11 hours ago [-]
But isn’t dot paid while muse is free? Why make that change? I have both and I’m trying to figure out how to justify dot
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
pvab3 13 hours ago [-]
Cool, looks like 6.2 Sol is cheaper than Sol 6.1. I've stopped using Astra because it also goes and does things that I didn't want or didn't ask for, and burns a bunch of credits while doing so.
billypilgrim 13 hours ago [-]
I have yet to find an application for Astra vs Sol where the former is noticeable better. Genuinely asking, does anyone feel a big difference in their work between the two? If so, what are you working on?
whimsicalism 13 hours ago [-]
Subagent coordination, Astra also performs better on Opus/Fable thinking/modeling shape problems imo, also computer use/thinking about shapes. Astra is able to hear waveform images which is pretty nuts.
For just doing things or coding things up, Sol all the way.
zeroxfe 13 hours ago [-]
Wow, that's the opposite of what I see. I always simply use Astra (high), despite the cost and latency overhead -- it's just a lot more reliable. I find that on average it uses fewer tokens for a task (and I lose less hair per task), simply because I don't have to steer it as precisely .
itvision 13 hours ago [-]
I have a 1.5 MB C++ codebase, and at least a third of the tasks that Sol completed required cleaning up by Astra.
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
5555watch 11 hours ago [-]
Yes, Astra Ultra is just less friction. I task it and when it finishes it's done.
Once I had very little usage left, so switched to Sol (5.6, 6.1), and also tried asking it to delegate to Luna subagents to save credits. It delivered objectively less quality work and needed much more steering. Such that I eventually gave up on them and just default to Astra for everything. I just occasionally ask it to minimize certain testing and it's good.
chaos_emergent 13 hours ago [-]
I find Astra to have a lot better ability to coordinate subagents compared to Sol. I haven’t found a distinction in the complexity of work they can handle.
juliusdaniel7 13 hours ago [-]
[dead]
gunalx 13 hours ago [-]
EE/circuit design is way better with astra.
Marha01 13 hours ago [-]
For creative writing, Astra is better than Sol. Bigger models tend to be better than smaller ones for such tasks.
whimsicalism 11 hours ago [-]
I've actually had the opposite experience, which is odd because I would generally assume the same. Astra wrote in this ridiculously terse big-brained style
zf00002 13 hours ago [-]
astra makes slightly better small 3d models than sol for me.
vessenes 13 hours ago [-]
Audit
Cu3PO42 13 hours ago [-]
I find that 6 Astra already had very good taste in frontend design, but relatively poor taste in "nice and readable code". I really hope this release cracks that. Excited to try it out!
seizethecheese 12 hours ago [-]
Nice incremental benchmark improvement at the same cost GPT‑6.1 Sol -> GPT‑6.2 Sol:
FrontierCode 2.0
39.0% -> 43.8% vs Opus 45.5%
Terminal-Bench-Science 0.1
57.0% -> 68.0% vs Opus 63.3%
Artificial Analysis Intelligence Index
54.6 -> 57.3 vs Opus 59.1%
Sol 6.1 when it was released showed really nice benchmark improvements [1] but my impression from HN has been that people haven't found it to be a very nice model [2][3].
To try to close the gap between benchmarks and real world SWE use, we've been building a battle system on top of opencode [4], and Sol 6.1 has been a real laggard there. Sol 6.1 is statistically equal with DeepSeek 4.1 Flash at 10x the cost and 25% slower.
Am I reading these charts wrong? It seems like in most cases 6.2-sol : 6.1-sol Isn't that big of a change? It is a little smarter but costs a little more.
edit:
Take a look at Frontier Code 2.0.
I generally run 6.1-sol at High. That would be 34.9% at $0.65
62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
Isaackoz 13 hours ago [-]
Looks like 6.1-sol high+ is equivalent to 6.2-sol medium at the same price. So a bit smarter at a cheaper cost.
jotato 13 hours ago [-]
which benchmark are you looking at?
Isaackoz 12 hours ago [-]
I was loosely looking at the average of all of them. Regarding your edit, it does look like the difference for frontiercode is negligible; the difference of 3% in score is meaningless to me. Guess we'll see how it performs today on my project.
derac 12 hours ago [-]
I think 1. the problems on FrontierCode are incredibly hard. 2. it's 3% in absolute terms, but it's a 10% improvement.
tintor 13 hours ago [-]
Behind both Opus 5.5 and Fable 5.2 on software development benchmarks.
scilro 12 hours ago [-]
Only on max effort though. It seems to be alright at more common effort levels:
Effort Model AA Index TerminalBench FrontierCode $/task
------ ------------- -------- ------------- ------------ ------
Medium GPT-6.2 Astra 57.0 58.6% 46.6% $1.62
Medium GPT-6.2 Sol 52.7 56.1% 41.4% $0.41
Medium Opus 5.5 48.2 52.5% 33.1% $2.45
High GPT-6.2 Astra 60.1 60.1% 47.1% $2.07
High GPT-6.2 Sol 54.9 58.6% 40.2% $0.56
High Opus 5.5 51.8 56.6% 35.9% $3.46
Xhigh GPT-6.2 Astra 61.6 64.1% 49.0% $2.52
Xhigh GPT-6.2 Sol 55.7 56.1% 42.8% $0.65
Xhigh Opus 5.5 57.4 59.6% 43.8% $6.05
But of course, benchmarks can't capture that "it factor."
59nadir 9 hours ago [-]
I basically only use Opus 5.5 on medium. If 6.2 Sol trounces it that thoroughly for ~6 times less price and it doesn't "behave" like a moron I don't see why I would ever use Opus 5.5 over it. The problem is that caveat part, mostly, and I suppose the only way to find that out is to try it out.
b--l 6 hours ago [-]
6.2 astra scores well on verbosity. I may need to give it a try. I cannot stand the wall of inane junk you get from anthropic models.
The stain of distrust I have for OpenAI makes an announcement like this fall flat for me.
I cant use codex confidently without expecting my work to be trained on or at least exposed to someone
And maybe other vendors do this too, but the gut feeling of "its not safe to use GPT models" has ruined these cool announcements for me
Such a shame
mrandish 13 hours ago [-]
Recently, I've complained about OpenAI (and other tech company's) product landing pages communicating virtually no useful information about what the product is or why it might useful. So, I have to concede this is at least a step in the right direction.
muzani 5 hours ago [-]
Sol 6.2 seems worse on benchmarks, but cheaper.
I think it's more of a ChatGPT update rather than Codex, because one of the bigger changes is artifact creation.
Astra 6.2 looks like a substantial improvement.
huybuidac 3 hours ago [-]
Astra is very expensive. I am currently using two Ultra 20x accounts for work.
yanis_t 12 hours ago [-]
Looks like a giant step in terms of efficiency.
Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
mandog2000 10 hours ago [-]
Model rankings based on user's experienced described in this thread. I apologize in advance if this form of post is frowned upon. The rankings help me get a birds eye view of so many inputs.
It's probably just a coincidence, but I have read so many people say that the models get dumber just before a release. Just last night while working astra made a really stupid mistake and then did the "Yeah, that's on me." meme.
What could possibly cause this? Is it possible that maybe the post-training that the models go through slowly rots them, and pre-release they deploy the final version of the old model?
thesnarkitecht 10 hours ago [-]
Given that there is a new leader on the artificial analysis index every two weeks, I think that I will just wait for the newer Opus model, expected around the 21st. I usually, even with hard engineering tasks, get better results with Opus and Sol class models rather than Fable and Astra. Also spend around 50% less
cat-whisperer 2 hours ago [-]
why are both the model companies agreeing on their numbers? .2 why?
hmate9 13 hours ago [-]
Will need to play around with it to get a feel but those benchmarks with those costs look incredible!
It seems like they were saving these for just before ANT's IPO
uvdn7 11 hours ago [-]
I found the ads video interesting. It is targeting the users (not the bosses). I am guessing that the IT apartments are not making the decisions of which models/companies to use. Employees are voting with their feet. If that's true, a lot of the assumptions of SaaS stickiness might not hold.
aeagentic 10 hours ago [-]
Why is ultrafast just the same speed as normal Anthropic TPS? Does Anthropic have so much more compute?
Lucasoato 9 hours ago [-]
I don’t have benchmarks but it’s crazily fast. I couldn’t imagine reaching that speed with frontier models. I suspect they are leveraging Taalas or something to “print” the model directly to the chip, somehow analogously to ASIC miners for cryptos.
arfelious 12 hours ago [-]
That may not be the main subject of the plot in this case but I found results of Gemini 4 Carbon in FrontierCode 2.0 interesting. Besides the cost staying practically same between different reasoning levels, the performance decrease in high reasoning seems a bit weird. That usually happened in models with xhigh/max reasoning where they overthinked the solution
edumucelli 8 hours ago [-]
Less tokens for the same task. Pro plan already has a huge usage limit. But for cheaper plans that is basically doing more for the same price. Improvements like that are more than welcome.
thimabi 13 hours ago [-]
This release promises three things that I am pretty excited for:
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
jrflo 13 hours ago [-]
Opus has far better design tastes IMO, and is already great at 3D work
derac 12 hours ago [-]
This comes off as very partisan when you're commenting on a model that is minutes old that you haven't used.
jrflo 11 hours ago [-]
Sorry I should’ve clarified, I’m commenting on the previous version
e2e4 5 hours ago [-]
> also bringing Ultrafast to Pro 200 subscribers
just upgraded from pro-100 to pro-200 for the ultrafast
In the past, I've had to wait a while to see what "vibes" are (which is actually more reliable than benchmarks).
Today, I have so many projects that hit a plateau in terms of previous model capabilities, that I run them through the new model to see if they make further improvements.
Very early signal, but promising so far.
XCSme 10 hours ago [-]
Really cool hamsters, especially astra, really made it look like a painting/image, hard to tell it's an SVG[0], it's quite insane.
Interestingly, they seem to have changed how reasoning levels work. Low now reasons a and costs a lot less than previous versions on low[0].
They're much improved on fundamentals, but they've adopted the typical color and tone problems of a GPT diffusion model image even though they're svgs. Interesting.
XCSme 10 hours ago [-]
SVG logic doesn't seem bad though, for the net it used:
I assume that's just a rectangle with some CSS to draw the net mesh?
Miraste 10 hours ago [-]
Oh no the logic seems fine, I meant visually it's got the characteristic gpt-image-1.5 "piss filter"
clint 10 hours ago [-]
Site doesn't load :(
XCSme 10 hours ago [-]
Seems to load fine for me, refresh doesn't work? What browser?
Wait, now I'm thinking maybe it is the SVG? Is it mobile? maybe the astra-generated SVG is too heavy on the browser?
jswelker 6 hours ago [-]
I knew a new release was coming because for the past week and a half 6.1 Sol was suddenly slow as mud.
enlyth 8 hours ago [-]
Looks like OpenAI forgot to publish Windows x64 binaries for the 0.169.1 codex-cli update, it breaks it completely for me.
koruse 3 hours ago [-]
Openai we need better Luna too
krm01 8 hours ago [-]
So if I can summarize correctly: it’s slightly better in terms of output but it’s significantly cheaper and faster.
criemen 12 hours ago [-]
Fascinating that they're going after the mechanical engineering market now, too. Does anyone have experience in that space? How much is AI changing the day-to-day work there?
ramoz 12 hours ago [-]
The sentiment on x right now is interesting.
The influencers who get early access are using words like "worst", "terrible", "disappointing"
>`ts-rust` (rewrite of ts in rust) is a project I started to bench new OpenAI models during early access (ty for the unlimited tokens btw).
>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
user43928 8 hours ago [-]
That guy runs a company and develops a product. I don't think he has the time or intention to oversee and direct some rewrite of TypeScript in Rust.
So yes, obviously he would be instructing the model to improve that port autonomously.
djeastm 9 hours ago [-]
ts-rust was his benchmark for "task he once thought was impossible for AI to do". Opus, however, was able to do it, apparently. So he's seeing now if the latest OpenAI model can improve it at all. Seems like he's not impressed with the results.
suddenlybananas 9 hours ago [-]
but it's quite unclear what he is actually asking the models to do. I don't understand how you spend 300k without getting anything.
etherlink 6 hours ago [-]
[dead]
i_have_an_idea 9 hours ago [-]
It's almost as if it is some kind of an organized PR campaign
coolfox 7 hours ago [-]
initial impressions are that it seems fine-tuned for brevity and quicker responses at the cost of slightly reduced intelligence in some areas. I think this is something one could adjust for through loops or prompts aimed at completeness.
I'm hearing tons of mixed reviews on both of these, is it opus 5.5 level?
Robdel12 13 hours ago [-]
Nice! I still use GPT-6 luna for a ton of stuff, it's super solid (for me). So I hope there's a little bump for that model, too, soon
11 hours ago [-]
trefoiled 9 hours ago [-]
Tested both models, no change in speed from the 20-80tps I was seeing with the previous Astra/Sol models. I'm not sure when this happened exactly but it's strange to me that OpenAI's models are now around 5x slower than Anthropic's. Is this the new norm? Are other people seeing the same thing?
rcr-anti 13 hours ago [-]
Odd, doesn't look like they're available on the cloud providers, only first party. Not a single mention of Azure or AWS.
munksbeer 13 hours ago [-]
Are they intentionally lowering the cost of these models to the extreme, because of the Anthropic IPO? This just seems insane?
minimaxir 13 hours ago [-]
The entire industry is lowering costs, not just OpenAI/Anthropic, both due to a) open-weight model pressure and b) genuine efficiencies. Per-token profit margins are large enough to allow price competitions.
I'm still waiting for the shoe to drop. It's literally impossible for the prices to remain this low for much longer.
Miraste 12 hours ago [-]
We can't know for sure what OpenAI's models look like, but from public research it seems that models genuinely have become more efficient. Much more.
eli 12 hours ago [-]
Why? Pretty sure inference is still profitable.
Though admittedly probably not so profitable it can cover the cost of a next gen of training runs.
rico735 9 hours ago [-]
These are businesses run by some right bastards, not charities. They'll want to maximise costs, and not pass any savings on to the customer. Enshittification is pretty much guaranteed, isn't it?
As soon as we have a winner (probably Anthropic imo), they're going to squeeze us like Uber did their drivers. I know there's the Chinese model argument, but DeepSeek are also looking to IPO and will also want to make a profit at some point. Again, not a charity.
It's only cheap now because they are competing with each other for dominance and they need to win a popularity contest.
rendang 13 hours ago [-]
Surprising that their top model would be still be so cheap, and yet not surpass Fable in the high cost part of the curve
Aeolun 12 hours ago [-]
I just don't have access? Updated the codex cli and all, but the best available is still gpt-6.1?
rednb 8 hours ago [-]
Great job, this is the first release delivering UI/UX mockups being as good if not better than the ones produced by Claude. As other commenters have pointed out, GPT 6.2 Sol does a better job at iterating on its own instead of saying "done" after making the first draft. Matters a lot for UI work, but I also see that it is much more thorough for backend work without overengineering.
OpenAI has just become credible again for agentic coding on complex enterprise codebases as far as i am concerned. Looking forward to playing with it a bit more by powering through our roadmap using my stacked resets. The quality had been so bad that it wasn't even worth it to use GPT Sol 6.1 as i was spending more time fixing slop after waiting forever to get anything done. By contrast, so far Sol 6.2 feels superior to Opus 5.5 so far, and more cost efficient.
ryzvonusef 12 hours ago [-]
The names remind me of Pokemon, for some reason. I want the GPT that has Charizard in it.
SillyUsername 12 hours ago [-]
I'm still waiting for a new version of their open source model...
zergrush 11 hours ago [-]
This is my feedback of GPT 6.2 Astra med and Sol xhigh:
- It's still overengineering especially Sol. This was the biggest complaint by everybody who has been using Sol since GPT 5.6. OpenAI is not listening to us.
- Astra still writes complete slop code and at this price this is unacceptable. It's also chasing tail end of things that really has low value and irrelevant to the overall project. I'm working on a game and same issue I had with GPT 6 Astra where it would go down these rabbit holes. Opus 5.5 threw out much of its code and approaches.
I don't think Anthropic needs to release Fable 5.5 or Opus 6 at this point seeing how bad GPT 6.2 is. For OpenAI to release this even with early access testers who had access to it for weeks earlier shitting on it now on X, I am starting to understand why Anthropic sounded so confident about becoming a monopoly.
The other point of annoyance is the attitude of OpenAI staff on X. They seem very arrogant and seems to care more about growing their follower account and shitposting.
127 11 hours ago [-]
Those linear charts are shitty and hard to read.
Jimmc414 9 hours ago [-]
Gemini has quietly supplanted OpenAI in the neck and neck battle with Anthropic for the top frontier model.
kart23 12 hours ago [-]
wish they published some source code/Cad files or cost for the RC buggy. looks cool but theres no specifics
5555watch 12 hours ago [-]
Another anecdote. I've recently commented [0] my observations on Gemini 4 Carbon. I thought it gave really good answers, which were aggressively critiqued by Claude and Astra. Consequently, I've subscribed to AGY+Carbon, and Carbon continued delivering, yet Astra kept criticizing and listing why the results look good but are bad.
I've just tried Astra 6.2, and now they don't have any problems with Carbon's ideas, are willing to pick them up and continue...
I care less about the models being good and more on if I can fallback to them if Opus 5.5 gets a massive price hike after the IPO
otterdude 2 minutes ago [-]
number go up!
geauxvirtual 13 hours ago [-]
Website pegs my GPU at 100%. Are they mining for crypto with this announcement?
enjoyyourlife 13 hours ago [-]
Trying new ways to create distributed systems to train their LLM
kingstnap 13 hours ago [-]
The announcement video is really cringe imo.
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
vohk 12 hours ago [-]
That's about half of my problem with AI currently - much like the explosion in cheap overseas manufacturing allowed us to produce oceans of garbage [1], we can now trivially produce sagas of garbage slides and proposals that other people will have AI read for them. The volume of work output is increasing, but much less so the volume of actual work done.
The first graph (Agents' Last Exam v1.1) clearly got mangled by the marketing department so you can't read the cost of most models. Here is a version with a logarithmic horizontal axis:
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
fakwandi_priv 12 hours ago [-]
I hope it's faster than 6.1. It's not as slow as Gemini Carbon, but yeez..
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.
solarpunk_enthu 10 hours ago [-]
The promo video for this release is people procrastinating and being irresponsible. Then saving the day with chatgpt. Is that the world we are all dreaming to build?
machiaweliczny 11 hours ago [-]
This "efficiency" is likely why these models feel dumb, they lose reading of nuance when overtrained for specific tasks. It's like autists understanding of normal speech.
qwerty2020 9 hours ago [-]
Unfortunately the sentiment from our favorite, perhaps ill-chosen, tech influencers has reached a critical threshold of value to these labs - the economic incentive to sway these opinions renders them ~meaningless.
For anyone lost in the sauce re: polarized reactions, I'd recommend just taking a half hour to try the models yourself.
nprateem 52 minutes ago [-]
I wonder if it can design UIs that don't look like they were designed by a 4 year old like when I tried 3 weeks ago.
lexh 13 hours ago [-]
Before the pelicanistas arrive, allow me to present pelican-zapper:
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
gpugreg 12 hours ago [-]
Alternatively, here is a uBlock Origin one-liner that hides comments that contain the word pelican (also includes our comments)
The UX on the slider comparison isn't great.
What do you mean i have to individually move each slide deck to the same slide number?
And what's with those super similar blue shades for Astra in the charts?
11 hours ago [-]
leafingensemble 9 hours ago [-]
I have read the announcement post and will continue to use Deepseek v4.1 Flash.
The consumer API for this model WILL let OpenAI keep your prompts.
matt3210 13 hours ago [-]
> frontends that look and work better
The website doesn't event show the graphs properly.
mynameisvlad 13 hours ago [-]
The graphs look fine on my computer. Can you elaborate on what exactly isn't "shown properly"?
finger 13 hours ago [-]
Unreadable on the phone
mynameisvlad 13 hours ago [-]
Unreadable how? If it's the width, you can always... turn your phone sideways?
wasabi991011 12 hours ago [-]
And the slides and other pdfs don't load next pages or alt documents for me
rozenmd 9 hours ago [-]
Biggest win I've noticed so far: you no longer have to bloody remind Sol to push their damn code every single turn!
11 hours ago [-]
popcar2 13 hours ago [-]
I know the video is a comedy but it's so tone deaf that they're advertising it as "you should lie to your boss and make the AI do your job!"
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
algoth1 12 hours ago [-]
They should have made the opus and gemini chart lines a bit darker because i can still see them
lwarfield 13 hours ago [-]
I wonder if these will have the same crazed rotwiler feel that previous OpenAI models have had.
solomonb 12 hours ago [-]
Anyone else really put off by the video? Its giving de-skill and discourage effort.
lousken 10 hours ago [-]
This is all nice and stuff, but new oss models when?
guybedo 13 hours ago [-]
Benchmarks look good, it most often is the case with Astra and Sol.
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
maxnevermind 12 hours ago [-]
Black sphincter strikes back, orange sphincter is in disarray.
drbscl 12 hours ago [-]
> Meridian Sound Racing
They still love "meridian" huh
eiiot 7 hours ago [-]
Am I the only one who thinks this ad is about two years behind where AI is today? It seems kind of reductive to show that ChatGPT can make spreadsheets now... we know. I get that its a play on the famous Excel ad, but I feel like its a strange way to sell the capabilities of a new model, given that Sol 6.1 could absolutely do the same thing.
sreekanth850 13 hours ago [-]
So they release 6.2 just before Anthropic IPO.
morningbrew 12 hours ago [-]
This is too dangerous to release
deathtrader666 13 hours ago [-]
Waiting for the pelicans benchmark.
foolserrandboy 13 hours ago [-]
With the red scarf and fish in the front basket...
joegibbs 7 hours ago [-]
From the reactions I've been seeing it seems like they're atrocious, absolutely terrible. People who've been using it, what do you think? Is it actually that bad? Surely it has strengths right?
dbg31415 12 hours ago [-]
I really dislike the marketing videos on this.
"Look at all the 20-30 somethings who didn't do their jobs, but got saved by AI in the time it took their older incompetent bosses to plug in a TV."
As someone in his 40s, who uses AI daily (and I'm constantly showing younger folks how to do things with it), the messaging here really rubs me the wrong way.
Trying to be specific... it just feels wrong to me to try and instill more generational rift. Promoting stereotypes for no real reason, other than the marketing guys didn't have any other ideas other than to go low. The sooner we can get away from labeling people from what generation they were born in the better.
My 2 cents... I'll probably continue to use ChatGPT for slides, and written proofreading. But I'll stick with Claude for coding. ChatGPT has been behind for a while on coding, and their 6.1 just made a complex mess of every technical task I gave it.
giuseppegambino 11 hours ago [-]
Finally a version that compete with Fable 5.2 and Opus 5.5
whalesalad 12 hours ago [-]
Right out the gate 6.2-Astra (med fast) is giving me Sonnet 3.5 energy. I asked it to analyze a few CSVs on disk in the context of a larger project where we were seeing some odd rate limiting behavior. It immediately got stuck trying to get gcloud auth sorted out even though absolutely none of what we were looking at touched anything Google/BQ related whatsoever.
tasuki 10 hours ago [-]
The first graph on that page "Agents' Last Exam v1.1" is atrocious. Has no one told them about log scale?
mwt 10 hours ago [-]
I guess the PhD-level intelligence skipped high school?
ImJasonH 13 hours ago [-]
500 | Internal Server Error
sam0x17 10 hours ago [-]
Honestly a much more exciting announcement from my perspective would be "current astra/fable quality at 1/3 the price"
kardassnp 11 hours ago [-]
We are using the platform at https://studio.sinapsdata.com.br to evaluate the interaction between two agents regarding complex topics in Physics.
montenegrohugo 13 hours ago [-]
Nice 3D effect. Runs surprisingly well even on on brave on my linux pc without hardware acceleration
kinfish 11 hours ago [-]
i hope this astra family can get the work done tho.
jr3592 12 hours ago [-]
So, are the leaps and bounds of progress not going to happen? Seems we're settling into an iPhone cadence release cycle, where each model is definitely better but nothing you'd tell your relatives about.
Can we expect more "telling my relatives about this!" exciting models in the future, or is this the way things play from here on out?
firemelt 13 hours ago [-]
do astra or fable that have better html ui frontend design?
bananaflag 13 hours ago [-]
Is this the Bel model?
wahnfrieden 12 hours ago [-]
No
d_burfoot 12 hours ago [-]
I found the video off-putting almost to the point of being offensive. I've never been significantly anxious about being chastised by a supervisor. So I found the characters in that video scrambling around in fear to be ridiculous and cringe-worthy. Also, CRT monitors? Huh?
revolvingthrow 13 hours ago [-]
I really hope Sol can compete with opus 5.5. I dislike giving money to ANT due to it seeming like a strange cult rather than a straightforward "I just want money" from OAI, but opus 5.5 has been such a delight that I didn't even feel like fable 5.2 was a notable upgrade despite the benchmarks.
MassiveOwl 12 hours ago [-]
Yeah I don't understand the crowd that love Anthropic for "the safety" and "ethics". The products they offer are the same as OpenAI - they're literally substitutes, ANT just like to say the word safety a lot.
sfink 10 hours ago [-]
Heh, I'm the opposite. I don't use or give money to OAI due to its "ok for unlimited killbots and mass surveillance as long as we get legal cover from people who say it's fine and are totally worthy of your trust" stance, and prefer to stick with the condescension of the strange cult. Sure, they may come across as a gang of righteous know-it-alls, but my ego can handle that.
It sounds like some things are much cheaper in OAI land, but I'll never know. I'll probably start playing with other providers' models (non-ANT, non-OAI) at some point soon, though.
leothetechguy 9 hours ago [-]
what I find interesting is that there is essentially no improvement on CritPT v2 over GPT-6 Astra. What a shame.
zf00002 13 hours ago [-]
is this finally their answer to opus 5.5?
ryanisnan 11 hours ago [-]
It is wildly disappointing to see the marketing hubris associated with these releases.
This is such an unprofessional way to engage LLMs - to effectively pitch this as how to use them is sad.
Imustaskforhelp 10 hours ago [-]
From the point of benchmarks (which can or cannot be trusted), GPT 6.2 astra beats Opus 5.5 and is neck to neck with fable 5.2 and gpt 6.2 sol just a few points lower but pretty close while being cheaper ~2x for astra 6.2 and ~10x for sol 6.2 while outputting less tokens and being more token efficient and being less time consuming.
Claude itself recommends people to use Opus 5.5 instead of fable 5.2
This is pretty damn impressive especially if sol 6.2 does perform on a similar level to opus 5.5 (+/- a few points is alright but I think that opus 5.5 is a beast)
All within a few days just before Anthropic is getting IPO'd. Oh this is gonna be interesting and I am all for it!
Though I have to say that AA benchmarks can slightly not be trusted because they change it so much so unless we find people's real usage on things we can't say for sure but there are benchmarks like frontiercode and terminal-bench to be more representative of real work and both of these models perform good
All the more points to more models coming but if OpenAI stands to be close to opus 5.5, we are gonna find many companies switching.
The moat of AI is incredibly thin. Even if you are the very best at something (as Anthropic was for sometime with coding models), there is always gonna be a catch up and the moat is incredibly thin for all types of models.
The more the players in the space the better (atleast for us the users), I hope that Anthropic still has a good IPO just so that competition takes place and not any single company (OpenAI) takes the whole market. It might also be setting itself for its own grave because if Anthropic's IPO goes extremely bad (because of this model release and Jensen's recent affairs), then people are gonna be spooked of OAI's IPO as well. Both of these companies futures are interlinked and its gonna be interesting how this all pans out. Why am I having a feeling that because of all this competition, whoever wins might even end up combining with the other but the frontier is pacing rapidly from all companies and these are just such interesting times that we live in...
There are decades where nothing happens and there are months (especially in the case of AI)/years where decades happen. I just have to wonder what an historian's POV would be seeing this model release. Would it be just a release like all the rest of others or would it stand out differently. Well, time will tell what happens next but incredibly interesting times indeed.
onlyrealcuzzo 13 hours ago [-]
TL;DR - from the little I could gather, Sol 6.2 looks particularly impressive.
But... is it just me, or does this product launch page look like AI Slop?
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
suddenlybananas 12 hours ago [-]
I wouldn't go as far as to call this slop but the lamp builder thing was super weird and unclear for me too! Even clicking the other tab cause my browser to scroll down a lot which I found very jarring.
jdw64 11 hours ago [-]
I’ve used it a bit, and I think the GPT model writes code well, but it’s always overly defensive code—though that’s ultimately a matter of personal preference—and it tends to write too many tests. I can understand the defensive code, but it feels like if you give it a task of size 1, it writes tests of size 3.
varispeed 11 hours ago [-]
Quick quick run your prompts, before it gets nerfed.
avazhi 12 hours ago [-]
Not impressed, sadly. Anthropic sure must be having a giggle right about now.
I_am_tiberius 13 hours ago [-]
No reset?
zb3 13 hours ago [-]
> we’re launching GPT‑6.2 Astra and Sol with tighter safeguards for higher-risk cybersecurity assistance
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
dyauspitr 13 hours ago [-]
I beseech the incoming Democratic politicians to not curtail progress in LLMs.
Kinrany 13 hours ago [-]
Simply optimizing the current models to be faster without making them smarter would already give us a decade of improvements to build on top
dyauspitr 12 hours ago [-]
I’m not interested in that. I need to see these models become superhuman in the vast majority of fields.
voidmain0001 13 hours ago [-]
I hope the video advertisement on that page was made with ChatGPT and a video plug-in otherwise if it wasn't then what's the point of Astra and Sol?
k9294 12 hours ago [-]
Wow... I have to test this, but from the examples in the slides, it looks like they managed to teach Astra how to write non-AI-slop texts.
avelone 12 hours ago [-]
6.2 вышел чтоли?
hugodan 9 hours ago [-]
for a fair comparison their current Terra and Luna should have been included... how am I supposed to pick between them now?
timedude 10 hours ago [-]
I go to sleep, i wake up, another model released
vegnus 12 hours ago [-]
I still pretty much only use 5.6 on medium. Am I the only one?
Guyadou 13 hours ago [-]
Those AI labs got to chill. almost every day there’s a new model! Lol
Ersho 13 hours ago [-]
Another wave of Openai just killed ....
Also, is it website very laggy or is it just me?
rolymath 12 hours ago [-]
It's super laggy if you mess around a bit, play videos, etc.
Aerroon 11 hours ago [-]
Just like chatgpt website itself. If the bot answers with pictures and rich text the performance of the tab tanks.
rico735 9 hours ago [-]
It crashed my browser twice.
AGI.
rico735 8 hours ago [-]
The racing game doesn't load in Firefox and crashed my Chrome after taking about 3 minutes to load. 2/10.
blinded 12 hours ago [-]
luna slaps
badmonster0 12 hours ago [-]
how does it compare with fable
thesnarkitecht 10 hours ago [-]
Two or three points lower on Artificial Intelligence Index. I personally don't find Fable or Astra output to be better than Sol or Opus in any regard. Waste of time and money, unless you are going after 1: 3D graphics, or 2: Hardware Engineering
ex1fm3ta 13 hours ago [-]
I thought OpenAI would drop an awesome model on the day of Anthropic IPO, just to troll them
zarzavat 13 hours ago [-]
It's actually in OpenAI's interests that Anthropic's IPO is successful.
SomeonesAccount 13 hours ago [-]
explain?
rose52152 13 hours ago [-]
If Anthropic is a valuable company, then OpenAI is just as valuable, if not more.
rico735 8 hours ago [-]
Is it not a Nintendo vs Sega situation?
They are too similar for both to win, and the current cost of competition is unsustainable.
zarzavat 12 hours ago [-]
OpenAI is doing an IPO too. Both Anthropic and OpenAI need to raise money.
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
erkamyaman35 8 hours ago [-]
seems cool
kwkwkwkxdkxk 10 hours ago [-]
ty
mococa 10 hours ago [-]
Meh...
petesergeant 13 hours ago [-]
"Agents' Last Exam v1.1" did a human really look at this and say "nah, a linear scale is definitely the right way to show this"
jrflo 13 hours ago [-]
They probably want to show how far to the right Fable is
georgestones 22 minutes ago [-]
[dead]
7 hours ago [-]
7 hours ago [-]
11 hours ago [-]
lsch1033 13 hours ago [-]
[flagged]
11 hours ago [-]
36 minutes ago [-]
13 hours ago [-]
13 hours ago [-]
zhshhan 9 hours ago [-]
[dead]
reachjalil 8 hours ago [-]
[flagged]
11 hours ago [-]
ItsAScam 10 hours ago [-]
[dead]
11 hours ago [-]
sick_of_slop 13 hours ago [-]
[dead]
e2e4 9 hours ago [-]
[dead]
onesandofgrain 11 hours ago [-]
[dead]
ftchd 12 hours ago [-]
To whom it may interest, I usually run out of tokens a few times per day (5 hour limit), so I made something and been using it for the past 2-3 months.
Works with other AI CLIs like Claude, and other various CLIs like Vercel (if you're managing multiple accounts like me), Netlify, Github etc
specked-citrus 12 hours ago [-]
The price improvements stem largely from relying on CoT less, which trades off directly with monitability. OpenAI continues to backslide on safety even after the Hugging Face incident.
Actually I was really wrong. The model is not expensive!
enraged_camel 13 hours ago [-]
Dead on arrival, judging by the FrontierBench scores. Astra 6.2 is five percentage points below Fable 5.2 xhigh.
skybrian 13 hours ago [-]
Whatever you’re doing, it’s unlikely to be what most users are doing. Fable and Astra aren’t in my price range so I’m paying attention to Sol.
f3408fh 13 hours ago [-]
The point is that it's a fraction of the cost.
enraged_camel 13 hours ago [-]
I think "second place but also cheaper" isn't what OpenAI needs right now.
SwellJoe 11 hours ago [-]
Given the increasing cost of infrastructure and decreasing availability of cheap money to build it, I think efficiency actually is pretty important. Second place at a sixth the price and in a quarter of the time (both according to Artificial Analysis for Astra 6.2 vs. Fable 5.2, both max effort) definitely has a place.
swiftcoder 13 hours ago [-]
"Close second, and also cheaper" is however a pretty solid framing when your competitor is trying to IPO
enraged_camel 13 hours ago [-]
That remains to be seen. I think if OpenAI's goal was to torpedo Anthropic's IPO, they should have released a model that is significantly more intelligent and capable. I don't think investors will be impressed by this release, which I think is simply OpenAI trying to stay in the race.
swiftcoder 13 hours ago [-]
I don't think torpedo'ing the IPO is actually in their best interests? OpenAI also wants to IPO at some point, and if Anthropic fails spectacularly, investor confidence in a second big IPO in the AI space is going to be shaky
rico735 8 hours ago [-]
I guess OpenAI are seen as the normie one, and Anthropic as the coding one. They're basically the same, but that's PR for you. Coding is seen as the profitable bit, whilst OpenAI's advertising model has failed.
OpenAI must realise that the clock is ticking and they need to be seen as the coding one, and seen as that quickly. The idea of being the one with the most users is no longer a selling point.
Could just be getting desperate, knowing that if Anthropic are too successful with their IPO, they are going to struggle raising more funds. Seems potentially a lose-lose situation for them.
It certainly seems too close to the IPO for it to be a coincidence.
jrflo 13 hours ago [-]
Astra has always been a step below Fable so I'm not surprised.
Similar accuracy in both. It's cheaper both because of token reduction (20%) but it also took less tool calls to answer the questions. In some cases, it "finished early" and didn't find the answer we were looking for.
Also very fast. Both Astra & Sol are the 2 fastest models I've tested that got at least an A in the exam.
[1] https://plotly.com/blog/gpt-6-2-sol-plotly-data-analytics-be...
* We can see in the Chinese open weight models that there has been a clear trend of moving to more efficient hybrid attention formats over the last year (see e.g. https://sebastianraschka.com/blog/2026/full-to-hybrid-attent...), and presumably the closed labs have been following a similar path, perhaps using the innovations the Chinese labs are using, or perhaps developing their own. And at this point the models themselves are good enough to accelerate developing all sorts of efficiency improvements which can then be used in the next generation.
* The most dramatic pricing cuts are often on cached read tokens, which suggests KV cache efficiency improvements. Other token types haven't had pricing cut so much. If this was just a subsidy they'd probably have cut pricing more evenly across the board.
Finally, I do not think it is in the interests of the big labs to slash their margins on inference for api (i.e. enterprise) pricing. Much of their high value comes from having very big margins on inference (and we now know for sure that they do have them, see Anthropic's S1). There is presumably a strong shared desire to not have a race down to commodity margins. So most likely if they are cutting prices, their cost of inference has fallen. Individual subscriptions are a totally different matter, they are heavily subsidized compared to api pricing, and those are indeed about generating hype and developer mindshare. But they are not where the labs are getting their revenue.
All the LLMs have dropped significantly in writing quality, to the point that YC even created a RFS and everyone started using ASD-STE100. Claude was incomprehensible because it was using token-dense terminology.
I think it's an intended consequence of RL on long-term tasks, where most of LLM writing targets itself
Silicon Valley understands how to build ad networks, and those systems have extreme margins that will trivially subsidize the other business costs.
Google already knows this works, that's why they're full speed ahead on killing search traffic in exchange for LLM answers at the point of search.
The accuracy according to your link dropped from 100%(Sol v6.1) to 93%(Sol v6.2), not sure if that can be called similar.
If 100% is 0% error and 90% is 10% error. From this view, 90% is 10× worse than 99% in terms of remaining error or failure rate. Going from 99% to 99.9% is another 10× improvement, and so on. The differences are orders of magnitude, not linear percentage points.
I am not entirely sure I will accept the cost cutting measure of both Sol And Astra. And if people say this is not considered nerfing I think I will disagree.
Different products and services will have different expectations. But If a product has already reached 100% then all subsequent efficiency improvement should reach 100% if not 99.9% while being faster or cheaper.
I just tried out Codex for the first time in a few months, using GPT-5.6-Sol. It keeps doing work in small chunks and stopping. So far it feels pretty different from last time, when it would just persist to absurd lengths and do way more than I asked for. Now it's not even doing what I asked and just keeps stopping.
I'm doing a RE task now (x86-64 ELF to C++) which I haven't tested before (mostly did features and greenfield stuff) so maybe behavior is different there, but it's a bit annoying.
The models in Codex go all the way back to GPT-5.6-* so I'll have to do some more tests with different models and tasks.
EDIT: I even asked it why it keeps stopping, and clarified that I want it to continue, and it keeps stopping anyway. What the hell. I hope the older models aren't like that.
Maybe that point is a large overlap though, depending on the task
It's high time for OpenAI to assert competitiveness, regardless of the expense it incurs.
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
After a couple hours of testing, we reverted back to 6.1 for basic document generation, proposals, marketing content.
And if you want something that is purely objectively bad (cause anything up to this point could be dependent on taste), 6.2 added an ugly, unnecessary, unprofessional looking grey background layer behind "1 Executive Summary" for no discernible reason that'd sink any pitch [2].
Yes, professional decks don't all have extensive, unnecessary, overly branded multicoloured graphs that provide little useful information and I do view those as a blight against readability myself.
But professional decks also rarely if ever should contain artefacts from slap-dash execution like this example does. The GPT-6.2 Astra result is inferior to its own predecessor in the labs own example, simple as that.
[0] https://imgur.com/a/yRwNy6q
[1] https://imgur.com/a/L41WaW3
[2] https://imgur.com/a/MDWUutw
Besides the AI-isms in the headings, the 6.0 example uses meaningless words and makes unactionable and vague recommendations. What is growth on a "directional" basis (growth is already directional)? Where is the value-add in "convert visits into larger baskets"? Compare to 6.2's "ugly arrow" recommendations that are actionable, if not insightful.
Worse, 6.0 seems to conflate "traffic" with "transactions," when they are hugely different things.
And no, I will not give the "AI-isms" a pass. Execs and boards approve paying a banker (potentially) millions of dollars in fees for their insight and advice; having analysis so obviously outsourced to AI would kill credibility far far faster than the grey background layer you claim would "sink any pitch."
And again your nits don’t align with a lot of important decks getting put together. It’s not that it’s invalid but the examples they have show much stronger language that reads more like what an analyst/associate would put together and good decks often have stark design choices like gray rectangles.
That "ex-Refund measures are review calculations" reads as pure slop. What does that even mean?
I will however at this stage also have to defend GPT-6 Astra. No, I'd never use any model writing from GPT-6 Astra (and doubt 6.2 will change that) in a presentation or documentation, but I have never read that model be this unintelligible. If going from that worst case to slightly less bad is worthy of celebrating in the announcement, I am still very unconvinced on the office work front.
The writing is much better though. Still sloppish but at least recognizable as language.
GPT-6.2 Astra scored 92 in 6.6 min for $2.04, vs Kimi K3.1 Pro (93 in 11.5 min for $0.68); it lost points for the web-page layout around the maze and an orange ghost that often never leaves the house.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-astra
Closest: https://jonclegg.github.io/pacman-bakeoff/#kimi-k3.1-pro
PacBench results.
GPT-6.2 Sol scored 87 in 6.3 min for $0.49, vs Muse Wonder (87 in 5.4 min for $1.17); it plays close to the arcade but lost points for a web-page layout, glowing navy-filled walls and no steady siren.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-sol
Closest: https://jonclegg.github.io/pacman-bakeoff/#muse-wonder
HARNESS OpenRouter / Claude Code RELEASED Oct 23, 2026 PHASE 2 RUN claude-code-openrouter__moonshotai--kimi-k3.1-pro EFFORT high REQUESTED moonshotai/kimi-k3.1-pro ACTUAL moonshotai/kimi-k3.1-pro WALL TIME 11m 29s INPUT 52,140 CACHED 729,344 CACHE WRITE 0 OUTPUT 42,703 THINKING 21,997 TOTAL — COST $0.68 HTML SIZE 39.7 KB
It mostly happens when thinking is set to max (and that is where the qwen models work best). It happens far less frequently at lower thinking levels.
But Flash Next seems to be very fragile/sensitive to small differences in quantization and chat template. For example, I am fine leaving it on xhigh thinking because I don't see any overthinking. The reasoning traces are very orderly and well calibrated to the immediate task. But if you look at online comments, people are clearly having different experiences with it. (I'm using the original Nvidia NVFP4 quant and vLLM 0.33.0, for what it's worth.) Edit: one other thing I don't do is use YaRN. I stick with native 262k context.
What I have seen is the occasional hallucinated URL fetch to aliyuncs.com.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
That’s the main difference. It’s smart enough for most of my tasks, but so cheap that I can use it far more than I could before.
Also, for long contexts, Haiku 5.5's direct competitor is probably Terra 6.1; Terra has similar cost per task with longer cache TTL and equal or better capabilities for long contexts.
I think Anthropic's problem is that they could get a lot of price-conscious people on board by making Haiku more attractive to use as a primary model, but then they'd probably lose money by having more people switch from Opus or Sonnet to Haiku.
I mostly use it to cheaply find out why a build failed (saves me about 5-20 minutes as I can have it run in the background).
It used to tell me which commit caused the failure. Now even if I ask it to, it stops well short with something similar to: "Well the logs say there's an error. I couldn't work out why.".
Not on a few important tasks in our app that I've tested it on
Neither provider’s models are perfect, but they keep pushing each other forward. As an individual, it’s relatively easy to switch between them. At a large company, you’re often committed to one provider for a long time, so some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to.
1. If you make an app, most of the happy users say nothing. Whereas seemingly all of the unhappy users will be vocal complainers.
2. The grass is always greener on the other side of the fence.
I feel the "grass is greener" mentality is particularly silly in this case, because for an extra $20/month you can just keep a cheap subscription to the "other" one. I realize not everybody has $20/month, but I think most people who can afford <= $20 for one subscription can probably afford another $20/mo for the other one.
If I don’t like a tool I just find a way not to need it.
I would complain about Fable’s writing style, to illustrate, but after some tense days we have found our common ground!
My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
I would not put Google in the same league as OpenAI and Anthropic. It is like 3 times slower at completing tasks. It also has the highest failure rate out of any of the models. Like sometimes it can do what Astra and Fable can do but sometimes it just misses completely.
And obviously you don't run all the tasks through these models *locally*
And yes - they all run locally on similar machines and with the exact same setup in terms of files, installed tools, etc.
codex: blah blah blah blah - that's the problem.
[stops]
me: continue
codex: blah blah blah blah - compile failed.
[stops]
me: wtf continue
does the same with /goal as well.
Plenty of people reporting it -> https://www.reddit.com/r/codex/comments/1xuyggs/codex_premat...
I shouldn’t have to write it this way, but it seems to solve the problem.
That seems like the root cause of their issue.
(Note that the GH issue is closed without a fix)
It is a step up from the last stuff I tried from Google, but it is still excruciatingly slow, and makes far too many architecture and performance mistakes. I left it for 5 hours trying to fix a performance issue it had introduced. Gave it to Opus and it was solved in 15 minutes.
I would love to not rely on Anthropic's models, but they seem so far ahead for whatever I'm doing. Not even in the same league.
However, given that Carbon is much faster than Sol on my stuff (though much slower than Opus/Fable), and didn't seem to be worse than it, I may start to use it when I want to save up on Anthropic tokens.
Astra 6.1 and Opus 5.5 spent days on it without fixing it. Gemini 4 Carbon fixed it in about 3 hours. Only Gemini Carbon decided after getting the same result over and over that maybe the bug it assumed it was, was actually something different and tried a different idea on its own initiative.
Maybe Fable might have got it, but probably not before I ran out of credits.
Gemini 4 is _really_ good at fixing bugs. Whether the OpenAI 6.2 models are that good remains to be seen.
How did you gain this sense? Do you use all of them interchangeably on tasks that are similar in size and complexity?
I think "they're all pretty much the same" is a good working hypothesis due to the intense level of competition between these three companies. When someone says "OAI suck; Claude is good" or the opposite (as you very often see in these threads), I think that's likely rationalization at work rather than a meaningful comparison.
For OpenAI, they have not been in a good place for the last two months. I switched from Astra to Opus 5.5 and it's been a clear upgrade, nicer to use, and much more generous with the usage limits on top.
This has nothing to do with rationalization. Opus 5.5 is just the better model, and that's a wide consensus. In the past I used the $200 subscription for both Anthropic and OpenAI, nowadays I downgraded Codex to $20.
From the comments on this release, it appears OpenAI rushed something barely usable out of the door in an attempt to stop bleeding customers to Anthropic. This would check out with the reports back then that the 6.1 Astra release was cancelled because of poor alignment and lack of usability as a coding agent.
I would not be surprised if the issue is still present to some degree in this 6.2 release.
For the most part the majority say the same thing but with different wording.
Only model that I have found that has, what I describe, "humanly creative takes" has been opus 5.5 so far.
But that was until gemini carbon came along, and I've decided to bench all the other frontier models for the time being, well except Astra 6.2, I've just added it in the room.
The 80% overlap is expected since I'm not doing particularly deep or novel work. But the 20% divergence is significant -- OpenAI's models routinely spot bugs that Anthropic's don't spot, and vice-versa.
That's interesting. In your setup, since they're all in the same room, is it possible that groupthink emerges since they're all reading each others' output? Or do you have things set up to prevent that?In my scenario, it's relatively low-tech... I'm just cut-and-pasting between independent Claude and Codex sessions.
(This could easily be automated, and I have written skills to do so, but I actually prefer the manual approach for now because it helps me to comprehend things)
Still making my way through the entire Fable 5.2 thread and would love to switch over to an updated thread at almost the same scale.
HN admins could help facilitate this… maybe a monthly model talk thread or something.
(I am testing the hypothesis)
(Sorry, couldn’t stop myself.)
There is a big spectrum from pair programming with AI to I don't even read the PRs and your experience with the various models can vary greatly depending on where you fall on it.
I find in a pair programming situation a lot of them are very similar. Its when you let them off the leash and run unsupervised that you tend to start to see the differences.
To me that’s a sign of an authentic discussion. When I see social media full of comments about a model nobody could use until that moment, it’s obvious that labs are seeding release content.
`gpt` is the the overly pedantic team member that no one invites to game night, while `claude` is the "idea" person and the reason company meetings now requires a meticulously scoped itinerary down to the minute. If you know how to work with both types and which type of work each is better suited for each, you can get a lot more out of them. As another comment made: gpt is depth-first, while claude is breadth-first.
Re-using agent files doesnt work. Give claude detailed task boundaries about what a feature should do (and only do) and then a hard config setup. For gpt you tell it the feature scope in noraml discussion terms without worrying about coming back to find a simple rest crud form is now a single page web-app -- even though your users instructions say no javascript -- with 50 ways to ingest a 6 field form that step over each other and looks like it works but probably has bugs waiting to bite you.
For claude: strong settings, well designed hooks, skills, linters and code checkers fully set up to your standard, and a rules & standards system that is emphasized in every single instruction and agent file. Also a standards agent or skill that is used with a critique/review run that is a requirement of the `/goal` (this can go in a hook too).
For gpt: that isnt as important or needed and will probably start to piss you off because of how literal it is. Your rule for online activity: Respect robots.txt; ask the agent to use clojure cli with deps.edn to add a library; agent replies (the dependency repository's) robots.txt says it does not allow automated access. Explain to the agent that means bots and web crawlers not resolving a dependency from the repository. So less really is more. Instructions and prompts is where verbosity helps. Tell it you want a 6 field rest crud form and it will make a 6 field rest crud form. It will also give you directories full of measurements and checks and performance for each one of those 6 form fields. Restrict it at the user instructions and task level spec. If you dont want all 50 checks for each form field, tell exactly what you don't want in its instructions. It's a fine line but the pedantic types will probably love it.
gotta be different and appreciate deepseek as more moral and optimal choice
Maybe this will change after the IPO but for now, should probably put the money into Anthropic for another account and get more out of it while it lasts.
They seems to be adjusting expectations properly for the real world after everyone getting hooked on all these various companies cyberpunk drugs
The more oblivious the “masters” the more intense the ant/human’s loyalty wars will get.
So many possible futures, how do we pick between them? /h /!h
Just my 2 cents:
I think that our first reactions to seeing these companies model releases is that of skepticism and somewhat rightfully so because these companies have done a lot of things to burn a lot of trust.
But AI is still important and so if someone is on Anthropic, they might feel as if Opus 5.5/Fable 5.2 is good and so if OAI is saying Astra 6.2 is better/comparable to them, a persons first instinct is to not trust OAI which might mean siding with Anthropic.
And if Anthropic releases a model, then there would be more discussions on pricing because somehow that is also related to bubble popping. I mean, supposing I can get good enough model at 10x the cheaper price of Anthropic from OAI for very marginal points, ofc people will comment about that as well.
And within both the threads we will find an intersection of discussion of open-weights models as well.
But maybe this is that if a model is just released and as such we are not having real use and word of mouth after real use from other people (which takes time), so our first instincts is to not trust any of the company so much and benchmarks can be cheated through and sometimes inaccurate of real world usage so we are in for a surprise when a new model drops and so we just drop back to those first reactions that I mentioned previously.
But this was a good question that you asked! and I would be curious to hear other person's observations
but mine (TLDR) is that: people distrust both OAI/ANT and other AI companies in HN. (Maybe we are technical people and we tend to look beyond the BS of marketing and so we think that we wont be impacted by that BS so we want to call out that BS but we dont know for sure if its BS or not but judging by the fact that these AI companies have incredibly low trusts, we as default don't really trust them and consider them BS and so the chain follows.)
HN audience is a tough audience to satisfy from first impressions (real world usage imo becomes a much better indicator and we do see it as people mention other models in this thread and others) ;)
I've heard from several users here that they're offered up to $200-500 per comment sometimes to promote a relevant product when a first page post is about a competitor product
I can be a very positive person.
But the idea that people are still paying people to write slop just doesn’t add up for me. We have tools for that now. Those tools have tools.
Or maybe Anthropic does astroturfing on OpenAI release threads and OpenAI does astroturfing on Anthropic release threads?
HackerNews is being botted to death since a year or two.
The system works well for me!
https://artificialanalysis.ai/models?speed=output-speed-over...
Starting September 22nd, data collected by the T3 Code platform shows that Anthropic has absolutely taken away the developer market from OpenAI (https://slopalytics.com/usage), moving from about 50% of the usage to close to 80%. I personally hope that the value delivered by the latest OpenAI models levels the playing field once again, because proper competition is super important.
Also, all of these subscriptions are still heavily subsidized, whether it be R&D, marketing, or elsewhere. Also, in Anthropic's case, they WANT to subsidize. it is in their best interest to pull folks from Open AI and elsewhere.
Side note, No conflict of interest in this scenario: I have no horse in this race. I've a high end gaming PC and run Qwen locally. It's "good enough" and our electricity in the SE USA is dirt cheap and rates have been the same for at least 10 years (our electric bill is unchanged with my AI usage). If I had to use a cloud AI provider and couldn't use deepseak, etc., I'd probably choose something from Anthropic. Luckily, I don't, since I don't work anymore.
They released their S-1. That excuse has been proven false. On to the next...
If someone spins up a website with the corrected figures they will be doing god's work.
Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.
the only impressive thing about claude is seeing how much it costs
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
If we missed anything, feel free to point me to it and we can add it.
Have you not considered log scale for the "Agents' Last Exam v1.1" plot? Like this the left is so crowded I can't see a thing...
Would be great to see this on the plot as well
Right, but you gotta level with us that this affects all models equally.
OpenAI and Anthropic probably do.
(Again speaking only for myself)
Assuming good intent they can run it internally a few times and post it with a disclaimer, but engineering a harness is a bit of a bigger ask.
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
I still use Claude on a personal level, but it really feels like OAI wins on a product level.
ANT launches these proprietary tools that just feel extremely clunky like Claude Design which from my perspective diminish the brand value.
OAI feels like Apple and ANT feels like Microsoft/Android and open source like Linux.
Also with GPT Sol 6.1 I still get this feeling that it does only/exactly what you tell it to; meaning some times it misses a change there, a document here, that would have an impact if left (i.e. a bug), while Claude more often than not would make the requested change and suggest to you to change those other places that would also require a change. This in my view feels more "intelligent".
If you see things from a technical angle you will have an objective perspective of which model is "the best." But, just like operating systems (iOS/Android -- MacOS/Windows/Linux) it all comes down to what each person feels comfortable with no matter what the spec sheet says.
This is my preference and I'm in a similar situation as you. I do get lost a bit in the UI and lots of mental overhead when running multiple projects.
I love the dot. A while back my family was sitting around and I noticed the phone icon. I said lets call Big Boy (thats what i named my dot). Call picks up in a flirty female voice. My wife was not happy haha
such a great title that made my day.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
I can definitely see why that behaviour would be irritating in most code related work.
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)
Genuinely, just excited about what we're building and think it adds information about these models that are missing.
Do my comments get tiresome, though? None have been downvoted to hell, since they are all adding to discussion. If you think so feel free to email me, I don't want to be annoying.
You have six different comments on this story that all basically link to your product. That's a bit hamfisted.
BTW check out my totally real project that helps you detect native advertising on your favorite platforms:
https://github.com/blah/blah
What kind of stuff were you throwing at it and where was it letting you down?
So what have __I__ been doing with it, then?
I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.
So yeah, 6.1-sol is good.
Where it falls down is on the boring stuff - writing tight PRs for features / changes in large production codebases (data area in my case). Without a lot of handholding, tendency is towards 'bolting on' new code resulting in overcomplexity & missing obvious opprtunities for simplificatation.
I think you are correct, OpenAI tools need more handholding.
However, I think that is actually intentional from OpenAI... they stick to what you the POH ask them and deliver what you ask. They embellish less. Anthropic tools by contrast embellish a lot, and they're very good too.
Perhaps start explicitly asking the OpenAI tools to embellish or think outside the box. That may unlock that experience and results you prefer. If you already did this, would you mind sharing more detail on what effect that had (or not) for your workload?
General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.
Have a look at the videos comparing opus 5.5 to others.
https://www.youtube.com/@ArenaAIOfficial
Astra and sol are quite behind, hence I question your motives, unless that's truly your perception, which may be genuine, but honestly, odd and against the grain of many, many others' experiences.
I am 100% sure they do. My internal evals of Opus-5.5 dropped a few weeks ago, then recovered to launch levels once I got added to the security program.
https://maxxbench.unlap.ai/research/why-maxxbench/
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
Either people really just want to buy and hold or they see it as a gamble chance like with SpaceX Superintelligence LLC.
> I have no idea why they released this. It needed a lot more iteration.
Maybe the goal is to make Anthropic's models look good by comparison. :-)
Here's a more balanced tweet from another early reviewer noting it just seems to be very uneven: https://x.com/emollick/status/2122755693191287009?a=hzSJAw9H...
The review claimed that the writing in the output is incomprehensible.
It also said that with /goal, the model set the goal to blocked to ask for the filename reporter_3423 should write to.
Lastly the reviewer said it made no progress on a large project.
Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.
Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)
It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive
I thought it was interesting that all the Sonnet 5.5 have a weird light effect on the bowl. XHigh has a nice film grain too.
Also confirms grok is absolute garbage
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
Dario says a lot of things. He just said they'll cure cancer too.
If you really mean it though and think about top x things to do first, i would probably try to research, invent and patent some superconductor and medicin.
ASI might not mean it can solve everything immmediadly but it could in x years with x amount of compute.
With an IPO and more capital, you reduce your risks
[] IA Stockholm Syndrome.
https://www.reddit.com/r/threejs/
Anybody remember https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B
"Oh, the quality is bad because we uploaded the wrong weights."
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
Cost is the most important factor now.
We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?
Not totally sure what your point is. Is it incremental that OpenAI dropped the 722 math proofs or solved Navier Stokes or disproved ETH etc? Yes, in a sense it is and the trend-lines predicted it for ages (albeit it seems to be quite a bit earlier than the trend-lines predicted), but it's also a mind-blowing result.
When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two, yes, it will have just been incremental progress, but it also seems like somewhat of a relatively big deal, no?
That would seem like a big deal because it’s not going to happen.
Feel free to return to my comment in 2 years time.
I expected significantly bigger improvements from Fable after Opus 5.5, this did not manifest.
And OpenAI is apparently in a bad and desperate place.
https://botbowlbench.com/
Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.
Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.
And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.
I don't mean to come across as argumentative, it's just been my experience that the supporting docs and plan are very important.
The prose is still shit though.
Sounds like I'd like it. I'm really tired of claude introducing pointless constants everywhere.
I've always used AI tools to write very constrained and small functions and modules, and then stitch it all together myself. So far. Over the last year it's become evident that many people let the AI drive 100% and there are proponents (DHH notably) that this is the way.
I'll be interested to see how compilers and LLM's compare at generating assembly over the next decade. Assembly offers enough abstraction, why do we need C? \s
We're building a battle system for OpenCode (https://agentduel.com/leaderboard) and have data on how large the diff was compared to opponents for both:
Opus 5.5: 0.42x (based on 299 battles) Sol 6.1: 1.42x (based on 265 battles)
Data on this is still pretty small. P=0.14. It's also affected by large differences, this is average not median.
I've seen two projects where developers didn't curtail this kind of overengineering, and recently had to spend a very significant amount of time and tokens to debug their deployment issues, because even for advanced agents there's a point where each issue could be caused by 12 things. This is especially pronounced in DevOps, but I see it in code as well; the suggestions are all reasonable individually, but you quickly end up with a code base that is too complex for a human to reason about, and it currently only kind of still works with agents because capabilities are growing so quickly.
Typical convo:
Me: "Hey, I see behavior X, which seems like a bug. What is causing that?"
Astra: "That's caused by a verification failure; we should be verifying the hash of this JSON output along with a permanent audit record. Let me know if you'd like to make that change."
::ask it what caused X to happen 6 or 7 times::
Astra: Tool use was limited to 1GB but the Rust compiler needed more than that, and that caused some PROBLEMS
Me: Remove that constraint
Astra: OK
In addition, I have spent some time to fine tune my custom instructions for Codex. It went like every time it violated my desired behavior, I asked why it did not understood my instructions and did this and that. In the end, the resulting instructions seem to work.
In the process, the model explained that the master instructions built in the Codex just tell it to push to the end and do as much as possible to satisfy a user. And these master instructions have precedence, so it's better to put the custom instructions into the developer_instructions section in the toml config file, as it has more weight, and not (only) in AGENTS.md. I ended up having both, with more details in the system AGENTS.md.
It also reassures me about the longevity of my career. I have sustained a theory that there simply isn't the data to teach these things to be really good, so I love to get the evidence to back it up.
After having it write around 13k loc i had to tell it to remove most of the complicated bs, we went down to 3k loc.
I actually have the problem with Fable stopping too early on tasks but Astra way overengineers things.
My favorite model for this at the moment is Gemini 4 Argon, which writes very naturally and isn't too expensive. If you're not using Cursor or some other harness that allows you to choose non-Anthropic models, then I don't know what to suggest.
I have to correct stuff Opus suggests during planning every single day, not even for major tasks. Very useful still, but last thing I would suggest to anyone is to trust its "gut instinct".
Small wonder why we have news of outages everyday now.
you really can't. unless you are working on something simple or what has been solved 100 times before
Also to allow it to express any concerns or highlight possible areas where we might be making an assumption about my intent.
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
[1] https://www.youtube.com/watch?v=HegoJaXOIqQ
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
"Now a robot can half-ass it for you so you don't have to!" is the perfectly targeted message for them.
When given my benchmark suite, which involves a couple of long horizon tasks where the agent decides when to consider the task well executed, its the first OpenAI model that does not want to finish as soon as some minimal level of quality has been achieved. Like Fable, Opus and Sonnet it now runs the maximum allowed iterations and does not stop at 'satisfactory', gold plating the solution to my delight.
(I don't work like this on work projects, but for everything hobby, I am testing how much autonomy I can give to the coding agents, and thats what my benchmark suite tests; I have the basic suite which completes in 2-4hrs, and the long running suite which takes about a week or so; the other one I just triggered, but the first iterations look way better than prev gen Sol/Astra at the same effort level).
And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
Something a human can recognize as inconsistent, but not the SOTA leading frontier model.
https://en.wikipedia.org/wiki/File:American_White_Pelican.jp...
While Astra only low looks good to me.
The problem you pointed out was never discussed on hacker news, and it clearly shows they still lack spatial reasoning. I wish you didn’t mention it. :)
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
Or on Max where its for some reason only using its left arm/wing - which looks totally different from its right arm/wing - and is grabbing the right side of the handlebar with its left hand for some reason?
Look. I think at first glance these look pretty good. I'd be astounded with this level of quality a few years ago.
But people saying that this benchmark is done are completely missing the point that accuracy, consistency, and details matter.
I'm not willing to say this benchmark is done until I see consistent and accurate details.
Quick glance at the table, wow pelican benchmark is solved! Oneshot a complex web app, wow software engineering is solved!
Of course upon closer inspection it’s always an incoherent mess. A poor imitation (i.e. lossy compression) of quality.
Astra 6.2 max did a similar mistake on the left bicycle pedal.
It seems like the models have a kind of built in aesthetic sense and I wonder what part of the training influences this. Maybe if more RL time was spent on artistic tasks the models could not only make better visuals but also code.
Currently they all seem to care little about code quality and maintainability (by humans), just functionality, which is fine I guess if you can expect to vibe code forever. But what if there is a critical bug and the models are temporarily unavailable for some reason? It would be hopeless to expect a human to wade through the giant bowl of clanker soup and fix it anytime soon.
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
Update: Here is some Bluey characters riding bikes as a reminder how far these models have to go.
https://www.bluey.tv/characters/uncle-rad/
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed
The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.
Astra's had identified 12 issues, including 8 P2s, but that changed to just 8 P3s after the discussion with Fable. Fable was able to persuade Astra that it's OAuth reasoning was incorrect ("had a missing link" as Astra put it).
So: overly eager but not very diligent, and (based on this little anecdote) doesn't really offer a compelling reason to try it more.
Sounds like GPT-6 Astra.
I wonder if the newer models are better about this.
Although Sol does still give up after a few hours when there are still many items left on the to-do list and tell me what next steps are instead of just doing the next steps.
But I’m talking about a task that is a two-sentence prompt. I would like the model to be able to keep track of straightforward instructions without harness assistance :)
But the issue they’re describing isn’t what I’m talking about. If the model wants to stop and request clarification, fine. In fact I sometimes explicitly ask for this.
But sometimes it actually thinks it’s done when it isn’t. This isn’t “I did a step and I have a question”. This is “I did what you asked” and then, after two more prompts, “oh, sorry, I did not complete the task”.
Which provider? I found the z.ai lite plan to be pathetic. Consumes 5h and weekly limits extremely fast when driven by Codex, and now also refuses RE work.
Latest entries are:
Astra 6.2: https://goyozi.github.io/ax-ball/
Sol 6.2: https://goyozi.github.io/sx-ball/
Fable 5.2: https://claude.ai/artifact/UszKFvuonWAT5psjMttAEg
Opus 5.5: https://claude.ai/artifact/UuT7P936dU1XoPByYZ7xyD
Google is BY FAR at the top of the Pareto frontier, perhaps b/c of Google Books.
But Sol 6.2 (low) is the first non-Google model that grazes that frontier. Sure, it's still more expensive than Hydrogen, but not too far! (and that's at the current low Hydrogen prices)
One of the consequences to that is I think the first wave of AI CAD startups are getting hit by the bitter lesson right now, including the startup companies that trained their own VLM for CAD like Zoo or companies that tries to build CAD specific harness like Adam.new/MecAgent, because the increase in capability from GPT/Claude/Gemini for 3D modeling as a specific area of focus for benchmarks means that the early moat gets eroded super quickly.
Opus: about 50x discount (98% off) Sonnet: about 40x Fable: about 5x All GPT: about 20x
Would be awesome if it stays this way though
Inference doesn't have to be sold below direct compute cost to be subsidized. If inference pricing doesn't cover amortized capex and the cost of capital it's still using investor money to buy market share.
Let say they just nerfed GPT 6.1 a few weeks ago. Now they are using that as a reference score against 6.2? Or were those results tested when 6.1 relesed. What about 6.0?
And that is the same with Claude. How do we know they are not cherry picking the date and test on these release when they are constantly "enhancing" their model.
We also know some of these benchmarks are largely irrelevant once they were known ahead of time and become a game of benchmaxxxing. It just follows Goodhart's Law "When a measure becomes a target, it ceases to be a good measure."
So how should I know what has actually improved?
https://www.bridgebench.ai/nerf-bench
https://github.com/ninjahawk/livenerf
Edit: But it does look like you get about 20% less tokens for your subscription with OpenAI than you did until a couple months ago.
The best feature of 6.1 sol was it would actually do stuff for you that Opus refused though. I did use that often.
Edit: It's obviously not on the same news page - I was referring to what I read on the Reuters page (https://www.reuters.com/legal/litigation/microsoft-unveils-a...)
EDIT: also 6.2 is seemingly faster than 6, could be due to more selective context ingestion and smaller context window though.
Maybe it found its way into the training data?
[1]: https://www.youtube.com/watch?v=HegoJaXOIqQ
https://meridian-futuristic-race.openai.chatgpt.site/
1: Make this real 2: Make it look better
https://compo.tv/w/after-rain
This is unfortunate—that was one of the big advantages of Astra.
Programming = Fable 5.2
Office work = Fable 5.2
I haven't found Astra to be better at any of these
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
For just doing things or coding things up, Sol all the way.
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
Once I had very little usage left, so switched to Sol (5.6, 6.1), and also tried asking it to delegate to Luna subagents to save credits. It delivered objectively less quality work and needed much more steering. Such that I eventually gave up on them and just default to Astra for everything. I just occasionally ask it to minimize certain testing and it's good.
FrontierCode 2.0 39.0% -> 43.8% vs Opus 45.5%
Terminal-Bench-Science 0.1 57.0% -> 68.0% vs Opus 63.3%
Artificial Analysis Intelligence Index 54.6 -> 57.3 vs Opus 59.1%
Sol 6.1 when it was released showed really nice benchmark improvements [1] but my impression from HN has been that people haven't found it to be a very nice model [2][3].
To try to close the gap between benchmarks and real world SWE use, we've been building a battle system on top of opencode [4], and Sol 6.1 has been a real laggard there. Sol 6.1 is statistically equal with DeepSeek 4.1 Flash at 10x the cost and 25% slower.
[1]: https://artificialanalysis.ai/articles/gpt-6-1-sol-replaces-...
[2]: https://news.ycombinator.com/item?id=50129992
[3]: https://news.ycombinator.com/item?id=49897054
[4]: https://agentduel.com/leaderboard
edit:
Take a look at Frontier Code 2.0. I generally run 6.1-sol at High. That would be 34.9% at $0.65 62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
https://artificialanalysis.ai/models/gemini-4-carbon#token-u...
I cant use codex confidently without expecting my work to be trained on or at least exposed to someone
And maybe other vendors do this too, but the gut feeling of "its not safe to use GPT models" has ruined these cool announcements for me
Such a shame
I think it's more of a ChatGPT update rather than Codex, because one of the bigger changes is artifact creation.
Astra 6.2 looks like a substantial improvement.
Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
https://chatgpt.com/share/6afcb990-f9cc-83ea-ad04-2724c7dba1...
What could possibly cause this? Is it possible that maybe the post-training that the models go through slowly rots them, and pre-release they deploy the final version of the old model?
It seems like they were saving these for just before ANT's IPO
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
just upgraded from pro-100 to pro-200 for the ultrafast
Today, I have so many projects that hit a plateau in terms of previous model capabilities, that I run them through the new model to see if they make further improvements.
Very early signal, but promising so far.
Interestingly, they seem to have changed how reasoning levels work. Low now reasons a and costs a lot less than previous versions on low[0].
[0] 6.2 Astra vs 6.2 Sol: https://aibenchy.com/compare/openai-gpt-6-2-astra-max/openai...
[1] 5.6 Sol Low vs 6.2 Sol Low vs Luna High: https://aibenchy.com/compare/openai-gpt-5-6-sol-low/openai-g...
<path d="M0 0q228 8 456 0v43q-228 7-456 0Z" fill="url(#mesh)"></path>
I assume that's just a rectangle with some CSS to draw the net mesh?
Wait, now I'm thinking maybe it is the SVG? Is it mobile? maybe the astra-generated SVG is too heavy on the browser?
The influencers who get early access are using words like "worst", "terrible", "disappointing"
>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
So yes, obviously he would be instructing the model to improve that port autonomously.
I see it here:- https://developers.openai.com/api/docs/models/gpt-6.1-sol
but not here:- https://developers.openai.com/api/docs/models/gpt-6.2-sol
https://news.ycombinator.com/item?id=50236102
Though admittedly probably not so profitable it can cover the cost of a next gen of training runs.
As soon as we have a winner (probably Anthropic imo), they're going to squeeze us like Uber did their drivers. I know there's the Chinese model argument, but DeepSeek are also looking to IPO and will also want to make a profit at some point. Again, not a charity.
It's only cheap now because they are competing with each other for dominance and they need to win a popularity contest.
OpenAI has just become credible again for agentic coding on complex enterprise codebases as far as i am concerned. Looking forward to playing with it a bit more by powering through our roadmap using my stacked resets. The quality had been so bad that it wasn't even worth it to use GPT Sol 6.1 as i was spending more time fixing slop after waiting forever to get anything done. By contrast, so far Sol 6.2 feels superior to Opus 5.5 so far, and more cost efficient.
- It's still overengineering especially Sol. This was the biggest complaint by everybody who has been using Sol since GPT 5.6. OpenAI is not listening to us.
- Astra still writes complete slop code and at this price this is unacceptable. It's also chasing tail end of things that really has low value and irrelevant to the overall project. I'm working on a game and same issue I had with GPT 6 Astra where it would go down these rabbit holes. Opus 5.5 threw out much of its code and approaches.
I don't think Anthropic needs to release Fable 5.5 or Opus 6 at this point seeing how bad GPT 6.2 is. For OpenAI to release this even with early access testers who had access to it for weeks earlier shitting on it now on X, I am starting to understand why Anthropic sounded so confident about becoming a monopoly.
The other point of annoyance is the attitude of OpenAI staff on X. They seem very arrogant and seems to care more about growing their follower account and shitposting.
I've just tried Astra 6.2, and now they don't have any problems with Carbon's ideas, are willing to pick them up and continue...
[0]: https://news.ycombinator.com/item?id=50442303#50442457
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
[1] Literally: https://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch
https://asdf10.com/gpt-6.2_agents-last-exam.svg
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.
For anyone lost in the sauce re: polarized reactions, I'd recommend just taking a half hour to try the models yourself.
https://github.com/lexh/pelican-zapper
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
[1] https://www.pacingthefrontier.com
The UX on the slider comparison isn't great. What do you mean i have to individually move each slide deck to the same slide number?
And what's with those super similar blue shades for Astra in the charts?
The consumer API for this model WILL let OpenAI keep your prompts.
The website doesn't event show the graphs properly.
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
They still love "meridian" huh
"Look at all the 20-30 somethings who didn't do their jobs, but got saved by AI in the time it took their older incompetent bosses to plug in a TV."
As someone in his 40s, who uses AI daily (and I'm constantly showing younger folks how to do things with it), the messaging here really rubs me the wrong way.
Trying to be specific... it just feels wrong to me to try and instill more generational rift. Promoting stereotypes for no real reason, other than the marketing guys didn't have any other ideas other than to go low. The sooner we can get away from labeling people from what generation they were born in the better.
My 2 cents... I'll probably continue to use ChatGPT for slides, and written proofreading. But I'll stick with Claude for coding. ChatGPT has been behind for a while on coding, and their 6.1 just made a complex mess of every technical task I gave it.
Can we expect more "telling my relatives about this!" exciting models in the future, or is this the way things play from here on out?
It sounds like some things are much cheaper in OAI land, but I'll never know. I'll probably start playing with other providers' models (non-ANT, non-OAI) at some point soon, though.
This is such an unprofessional way to engage LLMs - to effectively pitch this as how to use them is sad.
Claude itself recommends people to use Opus 5.5 instead of fable 5.2
This is pretty damn impressive especially if sol 6.2 does perform on a similar level to opus 5.5 (+/- a few points is alright but I think that opus 5.5 is a beast)
All within a few days just before Anthropic is getting IPO'd. Oh this is gonna be interesting and I am all for it!
Though I have to say that AA benchmarks can slightly not be trusted because they change it so much so unless we find people's real usage on things we can't say for sure but there are benchmarks like frontiercode and terminal-bench to be more representative of real work and both of these models perform good
All the more points to more models coming but if OpenAI stands to be close to opus 5.5, we are gonna find many companies switching.
The moat of AI is incredibly thin. Even if you are the very best at something (as Anthropic was for sometime with coding models), there is always gonna be a catch up and the moat is incredibly thin for all types of models.
The more the players in the space the better (atleast for us the users), I hope that Anthropic still has a good IPO just so that competition takes place and not any single company (OpenAI) takes the whole market. It might also be setting itself for its own grave because if Anthropic's IPO goes extremely bad (because of this model release and Jensen's recent affairs), then people are gonna be spooked of OAI's IPO as well. Both of these companies futures are interlinked and its gonna be interesting how this all pans out. Why am I having a feeling that because of all this competition, whoever wins might even end up combining with the other but the frontier is pacing rapidly from all companies and these are just such interesting times that we live in...
There are decades where nothing happens and there are months (especially in the case of AI)/years where decades happen. I just have to wonder what an historian's POV would be seeing this model release. Would it be just a release like all the rest of others or would it stand out differently. Well, time will tell what happens next but incredibly interesting times indeed.
But... is it just me, or does this product launch page look like AI Slop?
https://www.vals.ai/comparison?m=claude-fable-5-2,gpt-6.2-as... seems like a better comparison than what I could get from their page.
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
Also, is it website very laggy or is it just me?
AGI.
They are too similar for both to win, and the current cost of competition is unsustainable.
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
I've decided to open source it: https://github.com/FTCHD/switcheroo
Works with other AI CLIs like Claude, and other various CLIs like Vercel (if you're managing multiple accounts like me), Netlify, Github etc
https://deploymentsafety.openai.com/gpt-6-2-astra-sol/monito...
Actually I was really wrong. The model is not expensive!
OpenAI must realise that the clock is ticking and they need to be seen as the coding one, and seen as that quickly. The idea of being the one with the most users is no longer a selling point.
Could just be getting desperate, knowing that if Anthropic are too successful with their IPO, they are going to struggle raising more funds. Seems potentially a lose-lose situation for them.
It certainly seems too close to the IPO for it to be a coincidence.