Rendered at 07:21:28 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lcprk 10 hours ago [-]
It's fascinating to see these benchmarks being saturated where, a priori, solving it seems to require human-level general intelligence, yet when you try the models they are still clearly not broadly human-level. I am very curious to see if a particularly clever set of benchmarks is all you need to elicit genuine human-level intelligence.
Aperocky 9 hours ago [-]
Maybe the problem is the concept of "benchmark" itself.
Maybe we do better at things that cannot be a benchmark, i.e. uncertainty and randomness.
You cannot put things that cannot be graded into a benchmark.
imimayj1337 9 hours ago [-]
I think also there's an element of these benchmarks being static (and therefore gameable) which is holding us back? I know we do multi turn assessment yadda yadda but I wonder if we had an adaptive framework of sorts which could actually adapt to model behaviour whilst dynamically leveraging some kind of internal scoring criteria knowledge graph it might at least address the issue slightly? Reality is just not matching these benchmarks (or any of those which I've built privately for assessing models even!)
interstice 9 hours ago [-]
And yet we make children take exams to prove their value
PoignardAzur 10 hours ago [-]
Looking forward to see what they come up with for ARC-AGI-4.
datadrivenangel 10 hours ago [-]
100% 'with adaptor' and 70.3% in standard mode...
rzk 2 hours ago [-]
[dead]
coffeebeqn 12 hours ago [-]
So is this AGI?
rzk 2 hours ago [-]
François Chollet answered that here:
> Many of you will ask, "if it saturates ARC 3, is it AGI?"
Maybe we do better at things that cannot be a benchmark, i.e. uncertainty and randomness.
You cannot put things that cannot be graded into a benchmark.
> Many of you will ask, "if it saturates ARC 3, is it AGI?"
https://x.com/fchollet/status/2095599835932135919