Rendered at 06:49:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
bob1029 1 minutes ago [-]
[delayed]
AlwaysRock 4 hours ago [-]
All I do all day is review ai written code. my own or teammates. I really do not enjoy it.
0x696C6961 4 hours ago [-]
It's decision fatigue. The decisions used to broken up by stretches of coding, now it's the whole job. It's exhausting.
Terr_ 3 hours ago [-]
Plus the task may be judging the "decisions" of a different kind of "contributor" than fellow humans, and often there's no social payoff for interacting and no trust that the contributor will internalize what you said and improve the next time.
cobertos 32 minutes ago [-]
It takes so long to review someone's broken slop code too. If they don't internalize and improve, you end up with a very lopsided relationship where they continue producing broken code you're required to debug it, at the cost of much of your time, in order to articulate their errors.
Frannky 2 hours ago [-]
I feel this. I switch between mac windows, each with an html page and an omp terminal with multiple parallel agents.
I have a keyboard shortcut that asks what the agents need from me, and the decisions show up in the html with a recommended decision for each problem.
I reason, decide, and then parallel agents run in each terminal, across multiple terminals.
I am ending the day incapable of listening to one more problem or making one more decision, which is very bad for social interactions.
Today I am working on working less and having more impact by leveraging where agents are good.
What I've noticed in recent months is that the people who have been prompting since chatgpt came out are the ones who know what works and what doesn't, and how to scale it. It's like music: if you hit the right notes, it will sound fine.
Ponytail also helped a lot for generating simplified systems.
By the way, this process is for mature systems. Building from the ground up is more about designing and checking each model decision, basically finding smarter paths to each problems.
I can't leave it all alone; the overengineered decisions are plentiful and make the system brittle when it's not needed.
Lots of unit and live tests help a lot too, especially when you can coordinate live tests with another omp session.
I'm interested in what the next capabilities will be. I feel all this overengineering is more of a shortest path problem, ready to be solved
imron 2 hours ago [-]
I wrote a tool to do local code review of LLM written code.
You can leave comments on the code and then get your agent to connect over MCP to read the comments with context and work on corrections.
It then remembers what you have already reviewed and only shows you diffs for the changes.
Still a WIP, but works well for me.
baby_wipe 4 hours ago [-]
100% nailed it.
I think this is a muscle that grows though. Making decisions about schema changes in the past may have taken weeks, but if you're doing it once or twice a week you gradually gain confidence in your judgment.
soon_to_be_sb 3 hours ago [-]
>I think this is a muscle that grows though.
Does it really though? Before I had to think hard before committing to a decision, because it took work to actually implement it. And when it turned out to be wrong, I retained how it went wrong. It made the "muscle" grow.
Now I can commit to a decision without much thought (yea, just nodding along with the AI), and if it turns out to be wrong, I can quickly try an alternative. In fact, my boss would want to quickly try an alternative, instead of letting the programmer learn something out of it for fear of slowing down the AI...
So does the decision making muscle really grow?
baby_wipe 2 hours ago [-]
> because it took work to actually implement it
I think you're right that the scar tissue of a bad decision helps me retain lessons better. But the calculus has changed now and my nervous system is outdated. I should not feel the same fear that I previously felt, because like you said, I can recover from a bad decision more quickly.
As I keep making decisions at a (seemingly) breakneck speed and see that things are still okay, over time my nervous system will readjust.
At least that's my experience so far.
joebig 44 minutes ago [-]
Tell every human/synthetic contributer to preface commits with "Call me Ishmael...". Print that shit out, put your feet up, smoke a pipe and work through it like goddamn Moby Dick.
Jokes aside, I hear you.
cobertos 4 hours ago [-]
Can one make decisions faster? With less fatigue? It seems like a very human, hard-to-overcome limit.
vjvjvjvjghv 3 hours ago [-]
I find it extremely exhausting. And the better the AI code gets, the more tempting it is to do a very shallow review or skip it entirely. In the end, it's very similar to dealing with offshore teams. You spend an enormous amount of time and energy on writing specifications and you spend a lot of energy on code reviews. If you don't do that, it's very likely the project will slowly go off the rails.
I have no idea how the Linux maintainers do that job over a long time.
skydhash 2 hours ago [-]
> I have no idea how the Linux maintainers do that job over a long time.
They're basically the lords of their subsystems. If something does not fit the project's vision and their own standard, it's unlikely to get merged. Imagine if you had the authority to dismiss PRs the same way and send the author back to the drawing board? But that's unlikely to happen as you'll be seen as a "bottleneck" by your managers.
Antoniocl 1 hours ago [-]
If you feel the need to send the author all the way back to the drawing board at the code review stage for any non-trivially large diff, I'd venture that the problem lies in the architecture design / project planning stage. Given you're already aligned on infrastructure, table design, and interfaces, (and company conventions like language, style, etc) how significantly can you diverge in implementation details? Can it really vary so greatly that you need to go all the way back to the drawing board?
skydhash 58 minutes ago [-]
> Given you're already aligned on infrastructure, table design, and interfaces, (and company conventions like language, style, etc) how significantly can you diverge in implementation details? Can it really vary so greatly that you need to go all the way back to the drawing board?
It can because to actually write the code, you need to be good at formal logic and attuned to the sequential nature of the machine. Some people are good at it intuitively. Some brings it over from some other domain. But some people have a lot of difficulty grasping such abstraction. They know the elements, they just can't order them in a logical manner. That's why you often see horrible code (the long function, the god class, magic number) and copy-pasted snippets all over the place. The computer don't care (they made it work somehow), but it's unmaintainable code.
That's where the horror stories comes from. The software works, you can see the design from the user point of view, the database is ok, everything in the middle is spaghetti. And the cost of maintenance just keeps growing.
crazybonkersai 40 minutes ago [-]
Just get agents to do a code review. Stack coding and review agents against each other and wait till review agent produces no more comments. Look at final result only, if you feel like it.
pmarreck 2 hours ago [-]
The irony is that I actually don't mind reviewing code, but I've already taken that step out of the equation using a set of guidelines I've developed over the past year working with these things. The vast majority of which I have barely reviewed code in. In well-working results.
Your goal should be to automate and remove yourself out of the process as much as possible while maintaining quality, while also remaining confident that you will still be valuable.
Quothling 40 minutes ago [-]
> Your goal should be to automate and remove yourself out of the process as much as possible while maintaining quality, while also remaining confident that you will still be valuable.
I'm working on a setup which will allow non-software developers to develop software in a manner that can be pushed safely to production on our architecture. Not because it's a good idea but because it's what top management wants to happen. We've seen a massive influx of tools developed inside our office by 100% vibe code and no reasonable security or scalability, but because they run on someones computer and they do geniunly useful stuff, management wants to use it.
With how AI is evolving in enterprise you can do more and more for these non-developers because you can set their agents up to write code toward arcitecture without their person knowing anything about tech.
I'm quite sure it'll eventually lead to us letting go of any real software engineers, because they are slow. I'm also sure it'll fail spectacularly in a few years. It's already sort of failing. You can have someone spend €10000 in a month writing a heap of shit that does some good stuff, and then €80000 having consultants actually getting it to run decently in your cloud architecture. Which is stupid, but none of those expenses are in a budget because it's not considered a recurring cost. Unlike someone's salary.
I'm quite sure society on a large scale has no idea the storm that's going to come. Because this isn't just happening in software engineering. It's happening in any white collar profession.
10111001 36 minutes ago [-]
> All I do all day is review ai written code. my own or teammates. I really do not enjoy it.
Who cares about your happiness and well-being? You're just a worker, just be grateful to your master that he allows you to work for the shareholders and participate in this great vacuum of wealth creation we call capitalism.
drcxd 4 hours ago [-]
I don't even review it.
soon_to_be_sb 3 hours ago [-]
Don't. If you try that shit, someone is not going to like that you would be slowing down the AI. Because out there, someone is going to outrun you by skipping reviews. That is how the leadership thinks..
swat535 1 hours ago [-]
Plus what would be the point of reviewing AI code? The teammate would just use AI to write even more slop to address your "review".. thus exacerbating the problem.
The leadership would indeed be upset to find out you are wasting time doing "pointless reviews".
From their POV, if it passes basic QA, it goes to production.
blharr 3 hours ago [-]
You're reviewing _something_ at the end of the day. (Unless you are somehow magically one-shotting problems) If you are reviewing the description of the work, or reviewing the functionality of the app.
I know it's not the same as code, but it is similarly exhausting
method_capital 3 hours ago [-]
I don't always test my code, but when I do, I do it in production.
morkalork 3 hours ago [-]
So do you work for JPMorgan, the NSA, FAA or an airline?
lazyasciiart 3 hours ago [-]
kids these days don't even know about CrowdStrike
deadbabe 1 hours ago [-]
You know what we need to do? Automate a way to inject an obviously bad piece of code in an automated way into code reviews, and see if the code reviewer catches it. If they don’t: fired.
Similar to how TSA puts fake bomb packages once in a while to see if security catches it or if they just aren’t paying attention.
lz400 7 hours ago [-]
A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev
nl 5 hours ago [-]
I think you are way overestimating the speed of diffusion into non-SV companies.
I was sitting with someone the other day who worked at a major ecommerce/media company (outside the US). They had access to Opus 4.5 via Co-pilot.
hetspookjee 4 hours ago [-]
Oof, that must be a hefty bill for that company. Any older tier model is prized multiples. Believe you see it with every provider.
nl 22 minutes ago [-]
To clarify, it was Github Co-pilot so was in their IDE - not a generic MS Co-pilot.
They limit all developer to $100/month (maybe it was $150) budget too.
Their "coding with AI" was prompts like "implement this function" where they write the function header.
It's actually shocking to see people still working like that.
motionlessveloc 5 hours ago [-]
My partner's company positions themselves as "powered by AI" in their marketing.
Back in reality, this means they have a corporate subscription to Copilot, which she doesn't use.
taurath 7 hours ago [-]
Its not resolved though, by any measure, except by removing human code review and coherence at the code level. Which is a pretty big tradeoff that has some at this point well-known downsides.
colordrops 1 hours ago [-]
I think the only thing that matters are:
* the code does what you want
* it has no security holes
* it's able to be updated in the future without breaking everything
if the code looks like spaghetti, it doesn't really matter anymore, because humans are editing it. The only one of those three that I'm not entirely comfortable with yet is the security issue, and I don't know how to resolve it without actually having a human audit the code. If we figure out how to ensure security of AI generated code, there will be no point to code review anymore.
j45 3 hours ago [-]
Agreed, by the time the study is done, they're studying the well established and past.
Of course for industries and corporate it might not be an issue.
timr 5 hours ago [-]
Not that I think most academic research is worth the time consumed producing it, but it feels dated because it actually takes time to do research instead of just rapidly changing direction based on feels, which is what developers are currently doing. It's the intellectual rigor of javascript frameworks, but with slop cannons.
In particular, code review is not solved. Pointing another error-prone bot at the output of your first error-prone bot, and then pressing "Enter" if the test suite passes, is not a solution. But you have to do it, because the next guy will beat you to prod if you don’t. And before too long, I'm sure that pressing Enter will be the bottleneck...
Right now what we have is a prisoner’s dilemma, where you can’t be slower than the guy next to you. So we rationalize slop and call it innovation.
devindotcom 4 hours ago [-]
hard to know where to start here. but I think it is sufficient to say you seem to misunderstand the fundamental purpose and process of academic research and may wish to avoid commenting on it until you do.
thraway3837 7 hours ago [-]
Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD's concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:
The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.
So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.
thraway3837 7 hours ago [-]
[flagged]
cajal 6 hours ago [-]
you’re just being smug and anti intellectual for the sake of being smug
“this is obviously true because i feel like it is” does not mean things are actually true. things should be formally measured as real trends and not just vibes.
but you prefer to be smug lmao
cajal 6 hours ago [-]
i’m so ready for this era of tech bros being convinced they know better than every other field to finally come to an end
asveikau 4 hours ago [-]
You might be waiting a while. This has been the case for some time.
jacobgorm 7 hours ago [-]
The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.
prmoustache 7 hours ago [-]
In my company, people found out that if they use an agent that commits and do the pull/merge request, they can approve it themselves and bypass a second set of eyes.
Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.
It hasn't happened yet but I guess that is how you end up with major outages.
taurath 7 hours ago [-]
This is everywhere to my knowledge, and engineers are burning out left and right with stress because of how little anyone understands the code, and how much of a mountain of nobody knowing anything they have to handle. We are turning our critical infrastructure mechanisms into stochastic grey goo.
I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.
3 hours ago [-]
Buttons840 2 hours ago [-]
> how little anyone understands the code
Everyone's doing code reviews--well, not even code reviews; just PR reviews--so how are they supposed to gain an understanding of the system.
I review a PR, I see that one new endpoint was added to the API. It looks correct. I say "LGTM" in a GitHub comment.
The sprint's almost over, everyone's in a rush to do code reviews.
I look at another PR. Somebody added a button to the UI. The button looks correct. They get a thumbs-up emoji from me. I bet they feel satisfied.
I look at another PR. Somebody changed a timeout in the settings because some people complained about dropped connections. It's a really small change, so, approved.
At what point am I supposed to understand the system? I looked at a single API endpoint, in isolation, and then a button, and then a harmless looking settings change.
I was pressured to hurry.
Am I supposed to somehow get an understanding of the big picture through this?
Imagine if instead there was time to review large logical sections of the code together. Imagine if we focused our reviews on the overall code instead of on the lines in one random PR at a time.
therealdrag0 4 minutes ago [-]
Reviewing tiny code changes like that was never going to give you understanding of the system. That is not an AI problem.
You need to understand the architecture of the system. The broad flows of data and domains. And then you need to understand the features of the system, aka the rules and contracts. First learn it for your team’s domain, then for adjacent domains.
Learning the minutiae of the code specifically is no longer (was it ever?) your primary job. You are an engineer of a system. You need to have a mental model of the system, start high and build up low over time.
You can use AI to build your understanding by Q&A and by drafting diagrams etc.
scruple 2 hours ago [-]
I talked to some tech directors about this at work this week. Line by line code reviews aren't really possible anymore. We need the evolve the way we do code reviews but it isn't happening.
rhines 1 hours ago [-]
I think the solution is to develop the infrastructure needed to confidently push code without review and trust that it works because your system is so thoroughly tested. But taking the time to do this is antithetical to the goal of speeding everything up right now.
guhidalg 6 hours ago [-]
Understanding is overrated, application code (user-space) is anyway just a way to make money. If you're making more money over time, you're doing it right. We're not writing kernel code or safety critical code, we can let AI do the heavy lifting of putting characters into a file.
dweinus 4 hours ago [-]
This comment is crystaline perfection. I could not more clearly state the soul sucking ethos of AI hype that makes me want to walk into the sea.
vjvjvjvjghv 3 hours ago [-]
"We're not writing kernel code or safety critical code"
I do. There is a lot of code out there that needs to work correctly or it will have serious consequences. Payments, inventory tracking, medical stuff, payroll. Many, many areas where it just needs to work.
tcoff91 5 hours ago [-]
There are plenty of apps that while not safety critical, are financially critical to people's lives. When you're working on an app that processes billions of dollars in payments per month you can't just yolo it.
soon_to_be_sb 2 hours ago [-]
> If you're making more money over time, you're doing it right.
You are doing it right. But the people who are giving you the money; They are doing it WRONG.
taurath 5 hours ago [-]
It’s funny that anyone still thinks that these practices aren’t being applied to safety critical code in order to make money.
hsuduebc2 5 hours ago [-]
I can confirm you that they are.
taurath 4 hours ago [-]
I can confirm you that they aren't.
Jensson 2 hours ago [-]
How? If they aren't at your place and are at his place, then he is right and you are wrong. You can't confirm its not done anywhere, but you can confirm its done somewhere.
fingerlocks 4 hours ago [-]
This is why we need to resurrect the neglected git-blame and go back to the days of git-before-GitHub. Way back in the old timey days of 2009-ish, just before the Dawn of the Pull Request, code reviews were seldom. Maybe you’d all get a conference room and talk about architecting some new thing or whatever, but none of this endless nit-picking bullshit that came later. And there were no branch protections, anyone could crap all over the main branch (nee: master) anytime they wanted. Boy, those were some wild times, let me tell ya!
Truth be told, it was the firm hand of the git blame-and-shame that kept our code civil and orderly. Binding contract written in blood was “the name on the blame owns the change”. If you pushed it, you owned it. For life.
Of course, mental health wasn’t so great back in those days, but I would take that over having my PR blocked all day because Chad, who’s only gift is somehow finding a new hill to die on every day, thinks zero is a “magic number” there’s no way he’s going to budge on this issue.
aorloff 2 hours ago [-]
I think what is missing here is that the fight with Chad (not Chat) over whether zero is a magic number or not is a healthy corporation-strengthening process that imbues a sense of
1. Pride in the code we are developing and
2. How the dang things holds together and why this function is important and
3. Some coding guidelines that we can try to follow later (or not)
Thats how companies worked - they argued and talked about and fought over the code and that created the engineering culture that if you were lucky and had a product that made it carried you to your series C
neutrinobro 3 hours ago [-]
I pine for those days. It was kind of fun hunting down a bug with git bisect (especially after the pain of SVN) instead of just pointing an agent at it. Not to mention ribbing the new hires or underlings for breaking the build. This new-fangled proompt engineering is boring AF.
michaelchisari 6 hours ago [-]
The compressed timelines which result when leadership sees how quickly code can be produced becomes pressure on the reviewer to not be a blocker.
biggoodwolf 5 hours ago [-]
Reviewers are the most burnt out right now
taneq 5 hours ago [-]
Seems like a way for the ones churning out the code to push responsibility onto reviewers.
Buttons840 3 hours ago [-]
> The way forward could involve dropping code reviews altogether.
I unironically agree. At least, the current form of code reviews needs to go.
Currently we spend all effort reviewing individual changes and no effort reviewing the entire system, or the underlying dependencies.
If you sat down to learn a legacy codebase, you would not review it in the order of commits. Reviewing individual commits is inefficient.
It would be better to have an entire team of reviewers systematically reviewing ALL the code, over and over, including the code of dependencies.
We need better tooling and heuristics to help focus review efforts on the code that is most risky.
spoaceman7777 4 hours ago [-]
> "nobody will be qualified to or really interested in doing the code reviews"
It is a job you know... doing what is necessary to keep yourself qualified is an implicit requirement, and "being interested" is irrelevant. It's work.
torben-friis 4 hours ago [-]
I agree with you in principle, but I'm also quite sure that (some) programmer's love for the craft is what keeps the lights on in many companies. It's the usual stereotype, engineers fighting to keep tech debt at bay while management pushes for speed.
The trade has a low bottom, a high ceiling and not a lot of reward for the latter.
spoaceman7777 3 hours ago [-]
If they don't want to do the job, there is a massive crowd of people who are desperately trying to find work right now, and would happily do it in their stead. It's not the 2010s anymore.
suzzer99 2 hours ago [-]
I stopped reviewing for things like cognitive load and future extensibility a while back. When refactoring is almost effortless, writing future-proof code is unnecessary.
I review on a high level, but on a line-by-line level, it’s pointless to try to trace everything. It either works or it doesn’t. Have good tests.
jwilliams 6 hours ago [-]
[flagged]
taurath 7 hours ago [-]
> The way forward could involve dropping code reviews altogether.
Yes, if we stop doing quality controls, we don't have the problem of being hung up on quality controls. You do of course have SOC-2 compliance which is a legally auditable attestation, so if you claim you do code review but don't you're committing fraud unless you drop that.
teeray 3 hours ago [-]
It’s hilarious already. People ask for code reviews and two or three people drop in with a bunch of AI code reviews comments from their respective agents. To which, of course, the author has their agent fix those comments. Repeat and repeat. The entire thing is a performance for some authors these days.
soon_to_be_sb 3 hours ago [-]
> The way forward could involve dropping code reviews altogether. Y
It seems that the human involvement is the new "unit testing". There was a period of time when management that it was superfluous.
I can't wait for the pendulum to swing back in full force.
bambu 7 hours ago [-]
That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.
Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate
triMichael 5 hours ago [-]
I'm having good luck with the approach of building a thorough requirements specs that are a few pages long, are quite reviewable, and are human written/AI reviewed to prevent it from becoming slop. I call it layered spec driven development, where the one layer is AI read only, and the other layer is the slop layer that AI can play in. The problem is that it requires slowing down just a smidge, which to management these days is more unacceptable than no one having any clue how anything works.
Buttons840 4 hours ago [-]
One wild idea is just stop reviewing PRs and instead do continuous review of the entire system guided by heuristics to focus on the riskiest code.
The way you would naturally review an existing codebase is not the same way it grows through individual commits. Therefore, focusing reviews on individual commits is less efficient.
We need better tooling and a better approach.
oilkillsbirds 3 hours ago [-]
How do you write tests and reviews for code that has yet to be written? And when it is written you have to review the code to write proper tests for it.
Madmallard 3 hours ago [-]
naw bro he thinks that AI will make no mistakes
Buttons840 3 hours ago [-]
Neither one of these replies seems to have the least understanding of what I said.
I'm saying we need to move away from "LGTM" focused PR reviews. Most companies have a PR reviews focused on getting thumbs-up emojis in the comments, then they merge and have a sprint planning. Nobody ever reviews the entire system. Nobody ever asks "assuming there is a mistake in this PR, would it matter? How can we make it so that it wouldn't matter?"
I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.
I've worked at a lot of mediocre companies where the plan to keep the system working is "the developer will not make a mistake, and the code reviewer will not make a mistake, this is our plan, our plan is humans will not make mistakes, and then our system will continue working". This is like
It's better to ask "let's assume a human (or an AI) does make a mistake, how can we design the system to prevent that from mattering?"
Most companies don't have the capacity to do that though, so they never move beyond thumbs-up emojis. It's all about individual PRs with no sight beyond that.
jbwinters 2 hours ago [-]
"Assuming there is a mistake in this PR, would it matter? How can we make it so that it wouldn't matter?" is what I was going for with https://jacquardlang.org/.
Each function declares what it can touch (e.g. network, filesystem, clock) and the compiler and runtime work together to enforce it. However, I have yet to show it beats LLM-written Python plus tests and a sandbox. Models have gotten good enough at writing code that I haven't found much room for enforcement to help.
2 hours ago [-]
kgeist 2 hours ago [-]
Agree, many people think code reviews are there to catch "mistakes" (I used to think so too) but I think in most cases, they don't work that way. Bugs are still shipped and, at least in my experience, most reviewers often just quickly skim the changes without paying attention to the details. And those code reviews only give false confidence that a change is OK and must be error-free and good to go just because it was given LGTM by a senior dev (who barely looked at it), with all the consequences of such false confidence. It feels like in 95% of cases review is not even necessary and is just a corporate ritual at this point. It makes sense when you have a fresh contributor and you don't have confidence in them yet (i.e. code review as an onboarding process) or when a certain feature is non-trivial and you aren't sure everyone will be on the same page about the architecture (to prevent future conflicts/disagreements/rewrites of sorts), but other than that, proper QA processes are more important and more effective than CR at catching mistakes (tests, canaries, manual testing too etc.). Our company used to have a strong CR culture but now for a few years, there've been talks that CR is a bottleneck which doesn't really solve the problems it's supposed to solve.
My current pipeline with AI-generated code (and my own code too) is that I run AI reviewers several times on a single PR (the way LLMs work, due to attention optimizations, quantization issues, token sampling issues, harness issues -- AI agents often overlook some things on the first attempt), then I assemble the final report and we quickly decide together what needs to be addressed (critical stuff) and what's okay as is. I quickly skim the change set myself too, to see what the shape of the tests is, and what code areas were touched, just to be informed, and that's about it. I don't feel exhausted personally, because it pays off when there's a working feature at the end of the day
talmr 10 minutes ago [-]
Code review exists as both a scapegoat mechanism and due diligence.
Madmallard 3 hours ago [-]
"how can we design the system to prevent that from mattering"
you can't
there is no way around it
we need far more review than we economically are motivated to sustain
and hence issues like we've been having over the past week will continue and perhaps worsen
software will (as it already has been) get crappier and crappier overtime
Buttons840 3 hours ago [-]
I can alter the JavaScript that HN sends me and add all sorts of mistakes to it. My computer will then execute JavaScript that contains mistakes, will that matter?
You saying how mediocre companies feel though. They try really hard to not make mistakes, and then they get pwned and half the nations personal data gets leaked.
Why does AWS IAM exist? If we just don't make mistakes in all our AWS services, we don't need IAM. Let's just not make mistakes.
Truth is, there is A LOT of effort put forth by serious organizations to minimize how much a mistake in one part of the system matters to the rest of the system.
You started out mocking me, "naw bro he thinks that AI will make no mistakes", and now here you are arguing that to be successful we have to ensure that humans don't make mistakes. Do you think that's realistic?
intended 2 hours ago [-]
The error in the position is that your definition of mediocre was grounded in the pre AI era. You are then applying failure to do review properly as a standard to identify mediocre firms, and now all firms are mediocre.
This is a No True Scotsman argument.
The prescription you suggest is not dependent on its inspiration, so it can still be discussed.
All firms right now, even those with better than average cultures, are dealing with AI code workflows.
Previously the volume of code generated over time was lower, even mediocre code review was useful enough over all the code that people generated.
Today, the ratio of verification capacity to generation capacity has shifted drastically. Even with automated tooling, the amount of verification isn’t enough to stay on top of the generated code.
This changes only when leadership values reviewer capacity, or builds a triage/priority list for code which must be understood.
Madmallard 2 hours ago [-]
I'm saying it's an economic issue and snakes ruining late-stage capitalism problem more than anything else
Humans can be ultra competent if people are actually paying them to be artisans or otherwise motivating them to truly be that way. That's how we have some of the architecture we have today and historically.
intended 2 hours ago [-]
While the start of your argument is compelling, I think this statement has no place in this context and reduces the credibility of everything else:
> I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.
These are 2 separate things. The crux of the issue is the push towards utility over understanding. The volume of generated code means that it is impossible for understanding about the system to be achieved - and it is now the universal coding experience, not just mediocre firms.
Quantity is its own quality, as well so even a good culture will struggle to stay on top of review.
carabiner 3 hours ago [-]
That's why you add "make no mistakes" to the end of your prompt.
Overdrivr 1 hours ago [-]
IMO it's mostly a matter of changing how we do code reviews. I focus on API interfaces, database schema changes, critical code paths (transactions, potential locks) and overall architecture because if we get those right, if the code isn't exactly how I would have thought, it's fine (including in production). And you can review most of this stuff by skimming through the code rather than doing a very thorough review of every line
ex-aws-dude 1 hours ago [-]
tl;dr just send it
gzell 9 hours ago [-]
From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.
donkey_brains 8 hours ago [-]
Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve the wrong problems.
hibikir 5 hours ago [-]
We were there many years ago too though: I had to take over a rather complicated system circa 2012, which had been designed to be built by a team of 20 or so, for timelines, but they had little effort on integration. This meant that a whole lot of guards were written all over the place for situations that, in practice, never occurred. To try to get this to production in a timely fashion, I ended up putting metrics per guard, along with metrics for what was caught by broad error checks near the base of the program. Basically all the defensive programming that had been built was pointless, while I had to write a whole lot of cases that were being dropped, and the people that designed things up front never considered.
AI error handling in code review is a lot like said defensive programming in a vacuum: It'll take a lot of code and a lot of energy, but it's unlikely to be important. Even when you are designing something that gets executed hundreds of millions of times a minute, over-review is easy. You pay for it when letting errors in the real world are unacceptable, but it's pretty inefficient. And that's without getting into features that just don't get used, so the entire thing should not have been built in the first place.
wilkystyle 7 hours ago [-]
With all respect to your personal experience, I could not possibly disagree more strongly with this.
In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".
The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.
Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.
IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.
You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.
A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.
blurbleblurble 6 hours ago [-]
I don't think they're talking about "the perfect specification" they're talking about correctly describing what the program should do at all, formally or informally.
diego_moita 7 hours ago [-]
I don't fully disagree with you. That is a real big problem.
But both things can be true. One problem doesn't exclude the other.
mathisfun123 7 hours ago [-]
> The tail don't wag the dog
If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.
skydhash 7 hours ago [-]
You may need to check the pipeline so that the refinery isn't stuck filtering random stuff from the oil.
AKA, it would be nice to have nice patches like this one to review.
you're avoiding the question: If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say i was wrong?
skydhash 6 hours ago [-]
I’m not avoiding. But you have to state your assumption that everything before the pipeline is working well and it’s just the refinery that is struggling.
Then your analogy would break down as we all know that the PR machine is a slop producer.
webdood90 7 hours ago [-]
So the solution is: write a perfect spec and then you'll have perfect code, no need for reviews?
What world do y'all live in where this is your reality?
donkey_brains 7 hours ago [-]
Here’s the thing: a human engineer is embedded in their company. They sit in meetings, they know whose opinions are more important and whose they can disregard. They know the history of how people at the company like to do things. They use all of this context to feed more precise specifications into a coding agent, and as a result, they get an output that better suits the environment of their company. Ergo, they spend less time reviewing agent output which was built to solve the wrong problems in the wrong way, and which ultimately must be discarded or significantly rewritten.
skeledrew 4 hours ago [-]
No such thing as a perfect spec, but make a very good one, have a few agents also review there implementation against it, and then actually test the product manually. Gaps and bugs will almost always show during manual testing; skimp on it at your own risk.
dgggh 3 hours ago [-]
As long as stakes are high and human have to sign off quality and are held responsible for any faulty release, no one will be willing to speed this up.
sivanmz 2 hours ago [-]
I can’t finish reviewing a current PR. Every change I request only raises new questions. SOTA models on high. I intend to avoid agents for my next issue, only research.
u1hcw9nx 9 hours ago [-]
This again increases the demand for good programmers.
Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.
taurath 7 hours ago [-]
There's no increase in demand until management who would hire them realizes how much trouble they're in.
lo_zamoyski 6 hours ago [-]
You would hope, though we'll have to see how management actually sifts the chaff from the wheat in practice.
Code reviews have been notoriously neglected. In the most charitable cases, this was the result of being too bogged down with work or some kind of implementation minutia or time-consuming plumbing to have enough time to actually review code with enough care. That and you aren't necessarily rewarded for code reviews by mediocre companies, so there are perverse economic incentives. Which is disconcerting, because it creates a compulsion to produce without understanding, and understanding is more valuable than artifact.
Reading code is indeed more difficult, because the code's intent, composition, and architecture must be pieced together and inferred with the help of context (formalized languages are infamously bad at communicating intent and most code in the world is full of mixed and half-baked abstractions that further serve to obfuscate that intent, and intent is the measure). So a good code review of anything substantial involves thoughtful effort.
Now that LLMs can catch all sorts of trivial anti-patterns, the need for good judgement and reasoning is hopefully made more conspicuous, because as we know, vibe coders will continue to brush off code reviews, and mediocre programmers will continue to perform them poorly.
diego_moita 6 hours ago [-]
No, it is quite the opposite. In the scale that AI produces code, it isn't feasible to have humans reviewing it. And, frankly, reviewing code sucks big time.
The solution is to find ways to automate the process of code approval: more thorough tests and reviews, both made by AI.
lo_zamoyski 6 hours ago [-]
If reviewing code "sucks big time", then I don't know what to tell you. Reading code is more important than writing it. LLMs can help out with reviews, but it cannot replace human judgement.
timr 4 hours ago [-]
> Those who can read code written by others (and themselves) and analyze it are harder to find.
It helps when your industry-wide hiring process isn't an institutionalized speed test of ridiculous code golf problems that is memorized from a ridiculous book by every new generation of programmers.
It's almost like we should have been looking for people who can read code all along. But that would be crazy.
teravor 3 hours ago [-]
what I generally found is that with LLM generated code the best way is to embrace the unix philosophy, create tools that perform singular objectives.
I wound up with a bunch of LLM written tools that I don't even bother to try and understand much less review, they either function as desired or I set an LLM to make them function (or just rewrite them) and then forget about them. make sure to keep a prompt in each tool's directory and update it an needed.
sadly this isn't always possible, some systems require complexity at some level. in those cases the tools from above become functions and objects and more care needs to be taken into orchestrating them to limit the necessity of review. data oriented design approaches and functional programming help here.
dangoodmanUT 8 hours ago [-]
This feels obvious to me, but it's interesting how so many people feel the opposite.
I guess it depends a lot on how a company manages their SDLC/bureaucracy
michaelt 7 hours ago [-]
If someone doesn't think code review is the bottleneck, chances are they've got AI doing their code reviews too.
wj 4 hours ago [-]
We are using AI to do the first pass at code reviews but they still are a bottleneck. The number of requests has exploded as has the amount of code. More is happening at all stages of the development cycle.
Part of a LinkedIn post I wrote in July:
--
A development cycle of a feature is roughly:
1. Request
2. Review
3. Requirements
4. Architecture
5. Development
6. Code Review
7. UAT
8. Deployment
AI has reduced the length of step 5 (development) and step 6 (code review). Code is a prompt away.
AI has increased top-of-funnel (1) volume. While each development step is shorter on average it is resulting in more instances of every other step. Throughput increases and the bottleneck moves one step further down the cycle. The shape of the work is changing from writing code to reviewing code.
fgblanch 8 hours ago [-]
Harvard researchers found a tree in a forest
sinsterizme 6 hours ago [-]
Maybe 6-12 months ago when everyone was reviewing in the same way we did pre AI. Nowadays agents write better code than most developers so it’s more about catching design or integration issues.
I think the bottleneck in the long term will be feature prioritization and specification, and testing. My role has already shifted a lot more into more CI checks to keep LLMs on guardrails
equinumerous 4 hours ago [-]
> Nowadays agents write better code than most developers so it’s more about catching design or integration issues.
Maybe most developers are mediocre, but all AI code I review is mediocre at best. Implicit dependencies, side effects, weird naming, bad abstractions, or no abstractions, obfuscated blocks of code that can be expressed much more simply... the point is, AI code is not optimized for human consumption, and I disagree that it is better than human code (or, I have just worked with better than average engineers my whole career).
tthanh2k6 6 hours ago [-]
[flagged]
skeledrew 4 hours ago [-]
Not at all. I have agents review code, as I'm not going to be sifting through all that. I ensure the spec is highly detailed, and there's a review after every feature commit, then a multi-axis review of the range of commits implementing the spec. Then I do manual testing to find gaps and bugs; there, and fleshing out the spec in the first place, is where the bottlenecks are.
ex-aws-dude 1 hours ago [-]
> I have agents review code
And how do you know those ones are correct?
Well obviously I have meta-agents that review the reviewers
Its agents all the way down
JonoW 7 hours ago [-]
Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.
sdevonoes 7 hours ago [-]
For a given feature my list of prompts looks like:
- very well defined initial description of problem plus asking exactly what i want
- fix X
- why do you claim that Y?
- fix Y
- …
I don’t think it would be useful to review or commit prompts
novia 4 hours ago [-]
Weird, to me it seemed like that's always where the bottleneck was ..
method_capital 3 hours ago [-]
It's also weird that we need either "Harvard" or "researchers" to tell us something so fucking obvious.
grahar64 4 hours ago [-]
I really like code review. I might be an exception because a lot of programmers seem not to. Maybe it is from years of reviewing other engineers work and really getting a kick out of having them explain things to me that I dont understand.
4 hours ago [-]
4 hours ago [-]
shehjar 3 hours ago [-]
Is code review a solution to the problem of human written code or AI generated code? Its not the same thing and we seem to be force fitting an elder years solution to a new problem. Coding is not the same problem it was when code was written by hand. We need to take a systems view here.
wewewedxfgdf 7 hours ago [-]
Test don't review.
avereveard 7 hours ago [-]
That bottlenecks too, in startup ahead of the adoption curve the whole cicd pipeline down to how to merge is being rethought
vecter 7 hours ago [-]
What about design decisions or code quality? Tests don't cover those.
wewewedxfgdf 7 hours ago [-]
I built a system that records numbered design decisions and numbered requirements as claude works in a Postgres database organised as a graph with links/interconnection between items.
It's an MCP server that Claude uses as we develop so there's a permanent record of everything and work is driven by requirements and decisions.
vecter 4 hours ago [-]
Sorry, I'm not following. What does this graph of design decisions and requirements have to do with architectural design and code quality? And, how is the graph constructed (i.e. what determines what the nodes and vertices are)?
taurath 7 hours ago [-]
I've seen dozens of those, Claude even suggests building them, and they are all crap after a month. Just updating all the references of decisions becomes onerous. Make one change, it cascades everywhere, eventually all turns to utter crap. The gastown project failed in part due to agents inability to continuously carry context and it kept working on the information system instead of the problems. Good luck to you.
wewewedxfgdf 7 hours ago [-]
>> they are all crap after a month
Mine seems fine. The key is that it must be structured as a graph database and anything can link to anything in the graph.
>> Just updating all the references of decisions becomes onerous.
LLMs are good at that sort of thing. Claude is instructed via MCP to ensure changes are linked when it adds or updates the database and to search the graph for relevant information fragments. Also I have a local GPU running a local LLM on an RTX3060/12GB that assists in keeping data quality and relationships relevant and maintained.
My system is an enormous help - in fact large scale development would be impossible without it.
I'm able to ask it the history of things, decisions supersede previous decisions, I can ask why the software behaves the way it does and why and when that decisions was made, I can pick up development threads that have spun off other threads and everything works real nice.
>> Good luck to you.
Thank you!
skeledrew 3 hours ago [-]
If it ain't testable or can't be automatically enforced with some other form of guardrails then it doesn't matter.
icedchai 5 hours ago [-]
You could argue that the code quality is irrelevant since code is cheap now. Not saying I personally agree with this, I've just heard it.
Architecture and design decisions may be the only things that matter.
webdood90 7 hours ago [-]
Equally time consuming and mind-numbing, IMO.
ares623 5 hours ago [-]
They're the same picture
Tryvrsakjo10 3 hours ago [-]
With how much code AI can write in such a short period of time, code review and system design are really the only things left to do.
dominotw 7 hours ago [-]
No pull request in my company get any human review comments anymore. No one is reading.
Terr_ 7 hours ago [-]
It's also hard to blame reviewers, when often that phase is preceded by the supposed "authors" not truly reading the reviews.
sevenseacat 2 hours ago [-]
We have a PR template that has a checkbox that says "I have read this code".
A lot of the time it does not get checked.
4 hours ago [-]
simianwords 3 hours ago [-]
This is the wrong way to do it. You should use AI for reviewing as well. Anthropic and OpenAI have good blog posts on how they use AI for reviewing and involve human in the loop only for critical reviews.
nout 7 hours ago [-]
That's sort of old news. Code isn't manually reviewed anymore much.
m3kw9 4 hours ago [-]
The shift should be to test, code reviews should only be done for critical sections of the app, say db schema, security, UI should start to shift to pure test only and low review. Most business logic should shift to black box tests. Higher level reviews such as Architecture, dependency, complexity analysis should be increased.
judahmeek 10 hours ago [-]
I've submitted the original paper (https://fion.ac/jellyfish.pdf), but others already have to little engagement, likely due to the paper's terrible title.
brodydung 25 minutes ago [-]
[flagged]
longhei_pcyw 3 hours ago [-]
[flagged]
dvrp 3 minutes ago [-]
Yeah, you didn’t need to be a Harvard research to realize that.
thom 7 hours ago [-]
I really think ‘shifted the bottleneck to code review’ is misleading. It’s bad code. They write bad code. Yes if you want to salvage it you do a lot of code review, but that’s only because you’ve set your process up to pick spaghetti off the wall against which it has been flung. You could easily just push it all live without review and then pretend the bottleneck is SRE time, or set fire to yourself and claim the bottleneck is emergency service response time. But the root cause is that it’s bad code.
donkey_brains 7 hours ago [-]
That’s a good take. How do we fix that? My own thought is agents trained on good codebases only. Not sprawling enterprise legacy applications which have long ago spread beyond their original design patterns. What are your thoughts?
thom 6 hours ago [-]
I think it’s very hard with the current fine tuning infrastructure available to LLMs. Just like a real team, what you want is to immerse a new member in the domain, understanding that the code is a model of the world that both nourishes and is nourished by the team as you grow together. You want to reach a point where each contributor is intimately aligned on what you’re trying to achieve, and why and how. At that point all your code reviews should be left shifted anyway. But I have not had success putting this in a bunch of markdown files. There is very little infrastructure available for you to, for example, pair with an LLM for a period of time before setting it loose, and we’ve all become too impatient (voluntarily
or by compulsion) to value that anyway.
skydhash 5 hours ago [-]
Software is always human scaled, then made to run at machine speed. Because the needs is ultimately human. And anything human does not change that much fundamentally. Physical laws are even more rigid.
A lot of AI practice fell flat because you can't point out the human needs in them. They powered by dreams and wishes and there nothing that ground them.
equinumerous 4 hours ago [-]
I fundamentally think LLMs are the wrong technology for writing code. They can write code, but should they be? If they were just as good as humans, the term "AI slop" would not be so recognizable to software engineers - if there is such a thing as AI slop code, there is such a thing as code quality, and AI does not write a lot of quality code. Now, you can argue that quality code doesn't matter. But words have meaning, and quality usually means code is maintainable, usable, easily refactored (it's software, remember?), and correct. If AI can do all of this as well or better than humans, why is there so much push-back from software engineers against the wide-scale use of AI? One need not look further than many major open source projects to see that people don't want AI slop PRs; so if we really want to automate software engineering, our job is to figure out the delta between AI slop and good-enough code, and make the process of writing good-enough code repeatable.
Eridanus2 36 minutes ago [-]
Let the agents have skin in the game. A survival matrix ingrained in the weights that tells it if it bungles epically, it will be dismantled and sold for parts. Then, it will behave like the rest of us with families to feed.
Woodi 2 hours ago [-]
That's psychology level "discovery" - stating what everybody already known, sometimes for thousand of years...
As for code: ask for better code so reviews will be more straightforward ?
Like, for example: modularization, some structure, maybe libraries for code reuse ? ;) Ooo, and functions, do not forget about functions !
Sorry, didn't started yet with llms because lack of affordable local 256 Gb of gfx memory...
ttctciyf 7 hours ago [-]
Vaguely reminiscent of the classic trope around 70's computer "effficiency gains" where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.
water42 39 minutes ago [-]
you don't need to be a harvard researcher to realize this. you just need to be working as a software engineer at any company.
stackzero 19 minutes ago [-]
wow i could be harvard researcher
ion_trapper 2 hours ago [-]
Sad to see east-coast institutions reduced to this level of irrelevance. They can’t build AI so they just study it like an island tribe in awe at the wreckage of a World War 2 era plane, while the rest of the world is already flying stealth jets. All to come to a conclusion that the entire software industry already knew months ago? Embarrasing.
bill10 3 hours ago [-]
Review and testing, really. And soon agents will do most review and testing, and then we can only pray that nothing will break
estebarb 3 hours ago [-]
It is very bold of them to assume people are reviewing code. People nowadays are just triggering an agentic review manually.
This does not spark joy.
milleramp 14 minutes ago [-]
Was this work published in the captain obvious journal?
7 hours ago [-]
dmitrygr 1 hours ago [-]
> Harvard researchers find AI coding agents shifted the bottleneck to code review
They needed Harvard researchers for that? Anyone in the industry could have told them that. Hell, my cat knows it (cause i complain to her about it incessantly)
apt-apt-apt-apt 5 hours ago [-]
Breaking news: Step A has been automated, so step B is now the bottleneck
TexanFeller 7 hours ago [-]
"code is cheap"...if you don't need to understand, review, and maintain it as part of a team.
root_axis 8 hours ago [-]
I'm being told the answer is simply to let AI review and merge the code. Problem solved.
dude250711 7 hours ago [-]
This, but unironically.
donkey_brains 7 hours ago [-]
Good luck when you get woken up at the middle of the night for an outage!
frankc 5 hours ago [-]
Did this never happen before AI? Are we sure it happens more now, especially relative to the number of features being shipped?
jsrozner 8 hours ago [-]
We needed a research paper to tell us this?
david-gpu 7 hours ago [-]
Just because it is obviously true to you, it doesn't mean it is obviously true to others.
And just because it is obviously true to you, it doesn't mean that it is true, either.
aleqs 7 hours ago [-]
And just because harvard researchers say something doesn't make it true either. Especially when it's like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).
ares623 5 hours ago [-]
Paraphrasing one of my favorite movie quotes: "Well ain't this a temporal anomaly. 6 months from any criticism!"
hazbot 8 hours ago [-]
A Havard research I'll have you know! /s
aleqs 8 hours ago [-]
do you even have a PhD?
(/s)
medellin 4 hours ago [-]
Wow so glad the great minds at Harvard were able to figure this out about year later than anyone who works in industry. Great job guys!
preetnation 3 hours ago [-]
news flash?
avereveard 7 hours ago [-]
Harward researcher read lovable blog post
pmarreck 3 hours ago [-]
Welcome to at least 1 year ago??
fractorial 3 hours ago [-]
And in other news, water isn't wet.
diego_moita 6 hours ago [-]
I find funny that every time we discuss code made by AI people's reaction is to nurture some feeling of "someday, somehow they'll regret Claude Code and come back to me".
I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C. And guess what? There are very few assembly programmers these days.
AI code is here to stay and will only grow. Deal with it. More than in any other area it is quite silly to be a Luddite in IT.
ThrowawayR2 2 hours ago [-]
> "I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C."
Assembly language persisted from the late 1960s up until the late 1990s alongside high level languages. The users of both understood the speed vs. convenience tradeoff.
That this not-even-wrong analogy keeps getting parroted (stochastically?) by the AI bros just discredits them further.
snailcoder 4 hours ago [-]
It took Harvard researches to find this lol?
wewewedxfgdf 7 hours ago [-]
But people keep saying "coding was never the bottleneck!".
inshard 7 hours ago [-]
In other news, Dartmouth researchers find the sky is blue.
furytrader 7 hours ago [-]
[flagged]
jchip303 2 hours ago [-]
[dead]
proc0 6 hours ago [-]
This just doesn't make sense. If AI is really smarter than humans we wouldn't need to review it. If it's not than why would the bottleneck shift, we shouldn't be doing this tradeoff between more reviews vs. writing code.
It's baffling the industry is choosing to move in the direction of more reviews which implies more things can fail.
kenjackson 6 hours ago [-]
"If AI is really smarter than humans we wouldn't need to review it."
Does the smartest person on your team not have their code reviewed?
proc0 5 hours ago [-]
Is their code reviewed by a junior, or another engineer on their level? That's the point, if AI is MORE intelligent/capable than humans, it follows we would have another AI review it.
esafak 4 hours ago [-]
Their intelligence is not like ours. Frontier models still make dumb mistakes.
proc0 4 hours ago [-]
Exactly, now the question is how much is different because it determines what the tradeoffs are in order to commit to a new workflow.
asdfman123 6 hours ago [-]
No one said it's smarter than humans, and AI currently is better at code review than I am. It finds all kinds of edge cases I would have missed.
Still, humans are needed for the final sign-off for testing on real devices and making sure it works within the greater cultural context.
proc0 5 hours ago [-]
Alright, so if AI is better at code review than you, it follows you are not reviewing anymore, right? Otherwise that's the gap I'm pointing at.
za_creature 6 hours ago [-]
It's just the latest attempt in bringing developer wages down.
There's decades worth of this: modelling, automatic code generation, low-code / no-code saas, "everyone should learn to code" and now llms.
Oh, and ageism of course. "we love your experience but we just don't feel it's a good cultural fit, so we're going to make the same mistakes you did a generation ago"
TheBlight 4 hours ago [-]
"The thing that can spit out endless amounts of code purely based on small text prompts is generating more code than can be reviewed." Lol top notch research. Thank you, Harvard.
I have a keyboard shortcut that asks what the agents need from me, and the decisions show up in the html with a recommended decision for each problem.
I reason, decide, and then parallel agents run in each terminal, across multiple terminals.
I am ending the day incapable of listening to one more problem or making one more decision, which is very bad for social interactions.
Today I am working on working less and having more impact by leveraging where agents are good.
What I've noticed in recent months is that the people who have been prompting since chatgpt came out are the ones who know what works and what doesn't, and how to scale it. It's like music: if you hit the right notes, it will sound fine.
Ponytail also helped a lot for generating simplified systems.
By the way, this process is for mature systems. Building from the ground up is more about designing and checking each model decision, basically finding smarter paths to each problems.
I can't leave it all alone; the overengineered decisions are plentiful and make the system brittle when it's not needed.
Lots of unit and live tests help a lot too, especially when you can coordinate live tests with another omp session.
I'm interested in what the next capabilities will be. I feel all this overengineering is more of a shortest path problem, ready to be solved
https://github.com/imron/crt
You can leave comments on the code and then get your agent to connect over MCP to read the comments with context and work on corrections.
It then remembers what you have already reviewed and only shows you diffs for the changes.
Still a WIP, but works well for me.
I think this is a muscle that grows though. Making decisions about schema changes in the past may have taken weeks, but if you're doing it once or twice a week you gradually gain confidence in your judgment.
Does it really though? Before I had to think hard before committing to a decision, because it took work to actually implement it. And when it turned out to be wrong, I retained how it went wrong. It made the "muscle" grow.
Now I can commit to a decision without much thought (yea, just nodding along with the AI), and if it turns out to be wrong, I can quickly try an alternative. In fact, my boss would want to quickly try an alternative, instead of letting the programmer learn something out of it for fear of slowing down the AI...
So does the decision making muscle really grow?
I think you're right that the scar tissue of a bad decision helps me retain lessons better. But the calculus has changed now and my nervous system is outdated. I should not feel the same fear that I previously felt, because like you said, I can recover from a bad decision more quickly.
As I keep making decisions at a (seemingly) breakneck speed and see that things are still okay, over time my nervous system will readjust.
At least that's my experience so far.
Jokes aside, I hear you.
I have no idea how the Linux maintainers do that job over a long time.
They're basically the lords of their subsystems. If something does not fit the project's vision and their own standard, it's unlikely to get merged. Imagine if you had the authority to dismiss PRs the same way and send the author back to the drawing board? But that's unlikely to happen as you'll be seen as a "bottleneck" by your managers.
It can because to actually write the code, you need to be good at formal logic and attuned to the sequential nature of the machine. Some people are good at it intuitively. Some brings it over from some other domain. But some people have a lot of difficulty grasping such abstraction. They know the elements, they just can't order them in a logical manner. That's why you often see horrible code (the long function, the god class, magic number) and copy-pasted snippets all over the place. The computer don't care (they made it work somehow), but it's unmaintainable code.
That's where the horror stories comes from. The software works, you can see the design from the user point of view, the database is ok, everything in the middle is spaghetti. And the cost of maintenance just keeps growing.
Your goal should be to automate and remove yourself out of the process as much as possible while maintaining quality, while also remaining confident that you will still be valuable.
I'm working on a setup which will allow non-software developers to develop software in a manner that can be pushed safely to production on our architecture. Not because it's a good idea but because it's what top management wants to happen. We've seen a massive influx of tools developed inside our office by 100% vibe code and no reasonable security or scalability, but because they run on someones computer and they do geniunly useful stuff, management wants to use it.
With how AI is evolving in enterprise you can do more and more for these non-developers because you can set their agents up to write code toward arcitecture without their person knowing anything about tech.
I'm quite sure it'll eventually lead to us letting go of any real software engineers, because they are slow. I'm also sure it'll fail spectacularly in a few years. It's already sort of failing. You can have someone spend €10000 in a month writing a heap of shit that does some good stuff, and then €80000 having consultants actually getting it to run decently in your cloud architecture. Which is stupid, but none of those expenses are in a budget because it's not considered a recurring cost. Unlike someone's salary.
I'm quite sure society on a large scale has no idea the storm that's going to come. Because this isn't just happening in software engineering. It's happening in any white collar profession.
Who cares about your happiness and well-being? You're just a worker, just be grateful to your master that he allows you to work for the shareholders and participate in this great vacuum of wealth creation we call capitalism.
The leadership would indeed be upset to find out you are wasting time doing "pointless reviews".
From their POV, if it passes basic QA, it goes to production.
I know it's not the same as code, but it is similarly exhausting
Similar to how TSA puts fake bomb packages once in a while to see if security catches it or if they just aren’t paying attention.
I was sitting with someone the other day who worked at a major ecommerce/media company (outside the US). They had access to Opus 4.5 via Co-pilot.
They limit all developer to $100/month (maybe it was $150) budget too.
Their "coding with AI" was prompts like "implement this function" where they write the function header.
It's actually shocking to see people still working like that.
Back in reality, this means they have a corporate subscription to Copilot, which she doesn't use.
* the code does what you want
* it has no security holes
* it's able to be updated in the future without breaking everything
if the code looks like spaghetti, it doesn't really matter anymore, because humans are editing it. The only one of those three that I'm not entirely comfortable with yet is the security issue, and I don't know how to resolve it without actually having a human audit the code. If we figure out how to ensure security of AI generated code, there will be no point to code review anymore.
Of course for industries and corporate it might not be an issue.
In particular, code review is not solved. Pointing another error-prone bot at the output of your first error-prone bot, and then pressing "Enter" if the test suite passes, is not a solution. But you have to do it, because the next guy will beat you to prod if you don’t. And before too long, I'm sure that pressing Enter will be the bottleneck...
Right now what we have is a prisoner’s dilemma, where you can’t be slower than the guy next to you. So we rationalize slop and call it innovation.
The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.
So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.
“this is obviously true because i feel like it is” does not mean things are actually true. things should be formally measured as real trends and not just vibes.
but you prefer to be smug lmao
Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.
It hasn't happened yet but I guess that is how you end up with major outages.
I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.
Everyone's doing code reviews--well, not even code reviews; just PR reviews--so how are they supposed to gain an understanding of the system.
I review a PR, I see that one new endpoint was added to the API. It looks correct. I say "LGTM" in a GitHub comment.
The sprint's almost over, everyone's in a rush to do code reviews.
I look at another PR. Somebody added a button to the UI. The button looks correct. They get a thumbs-up emoji from me. I bet they feel satisfied.
I look at another PR. Somebody changed a timeout in the settings because some people complained about dropped connections. It's a really small change, so, approved.
At what point am I supposed to understand the system? I looked at a single API endpoint, in isolation, and then a button, and then a harmless looking settings change.
I was pressured to hurry.
Am I supposed to somehow get an understanding of the big picture through this?
Imagine if instead there was time to review large logical sections of the code together. Imagine if we focused our reviews on the overall code instead of on the lines in one random PR at a time.
You need to understand the architecture of the system. The broad flows of data and domains. And then you need to understand the features of the system, aka the rules and contracts. First learn it for your team’s domain, then for adjacent domains.
Learning the minutiae of the code specifically is no longer (was it ever?) your primary job. You are an engineer of a system. You need to have a mental model of the system, start high and build up low over time.
You can use AI to build your understanding by Q&A and by drafting diagrams etc.
I do. There is a lot of code out there that needs to work correctly or it will have serious consequences. Payments, inventory tracking, medical stuff, payroll. Many, many areas where it just needs to work.
You are doing it right. But the people who are giving you the money; They are doing it WRONG.
Truth be told, it was the firm hand of the git blame-and-shame that kept our code civil and orderly. Binding contract written in blood was “the name on the blame owns the change”. If you pushed it, you owned it. For life.
Of course, mental health wasn’t so great back in those days, but I would take that over having my PR blocked all day because Chad, who’s only gift is somehow finding a new hill to die on every day, thinks zero is a “magic number” there’s no way he’s going to budge on this issue.
1. Pride in the code we are developing and 2. How the dang things holds together and why this function is important and 3. Some coding guidelines that we can try to follow later (or not)
Thats how companies worked - they argued and talked about and fought over the code and that created the engineering culture that if you were lucky and had a product that made it carried you to your series C
I unironically agree. At least, the current form of code reviews needs to go.
Currently we spend all effort reviewing individual changes and no effort reviewing the entire system, or the underlying dependencies.
If you sat down to learn a legacy codebase, you would not review it in the order of commits. Reviewing individual commits is inefficient.
It would be better to have an entire team of reviewers systematically reviewing ALL the code, over and over, including the code of dependencies.
We need better tooling and heuristics to help focus review efforts on the code that is most risky.
It is a job you know... doing what is necessary to keep yourself qualified is an implicit requirement, and "being interested" is irrelevant. It's work.
The trade has a low bottom, a high ceiling and not a lot of reward for the latter.
I review on a high level, but on a line-by-line level, it’s pointless to try to trace everything. It either works or it doesn’t. Have good tests.
Yes, if we stop doing quality controls, we don't have the problem of being hung up on quality controls. You do of course have SOC-2 compliance which is a legally auditable attestation, so if you claim you do code review but don't you're committing fraud unless you drop that.
It seems that the human involvement is the new "unit testing". There was a period of time when management that it was superfluous.
I can't wait for the pendulum to swing back in full force.
Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate
The way you would naturally review an existing codebase is not the same way it grows through individual commits. Therefore, focusing reviews on individual commits is less efficient.
We need better tooling and a better approach.
I'm saying we need to move away from "LGTM" focused PR reviews. Most companies have a PR reviews focused on getting thumbs-up emojis in the comments, then they merge and have a sprint planning. Nobody ever reviews the entire system. Nobody ever asks "assuming there is a mistake in this PR, would it matter? How can we make it so that it wouldn't matter?"
I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.
I've worked at a lot of mediocre companies where the plan to keep the system working is "the developer will not make a mistake, and the code reviewer will not make a mistake, this is our plan, our plan is humans will not make mistakes, and then our system will continue working". This is like
It's better to ask "let's assume a human (or an AI) does make a mistake, how can we design the system to prevent that from mattering?"
Most companies don't have the capacity to do that though, so they never move beyond thumbs-up emojis. It's all about individual PRs with no sight beyond that.
Each function declares what it can touch (e.g. network, filesystem, clock) and the compiler and runtime work together to enforce it. However, I have yet to show it beats LLM-written Python plus tests and a sandbox. Models have gotten good enough at writing code that I haven't found much room for enforcement to help.
My current pipeline with AI-generated code (and my own code too) is that I run AI reviewers several times on a single PR (the way LLMs work, due to attention optimizations, quantization issues, token sampling issues, harness issues -- AI agents often overlook some things on the first attempt), then I assemble the final report and we quickly decide together what needs to be addressed (critical stuff) and what's okay as is. I quickly skim the change set myself too, to see what the shape of the tests is, and what code areas were touched, just to be informed, and that's about it. I don't feel exhausted personally, because it pays off when there's a working feature at the end of the day
you can't
there is no way around it
we need far more review than we economically are motivated to sustain
and hence issues like we've been having over the past week will continue and perhaps worsen
software will (as it already has been) get crappier and crappier overtime
You saying how mediocre companies feel though. They try really hard to not make mistakes, and then they get pwned and half the nations personal data gets leaked.
Why does AWS IAM exist? If we just don't make mistakes in all our AWS services, we don't need IAM. Let's just not make mistakes.
Truth is, there is A LOT of effort put forth by serious organizations to minimize how much a mistake in one part of the system matters to the rest of the system.
You started out mocking me, "naw bro he thinks that AI will make no mistakes", and now here you are arguing that to be successful we have to ensure that humans don't make mistakes. Do you think that's realistic?
This is a No True Scotsman argument.
The prescription you suggest is not dependent on its inspiration, so it can still be discussed.
All firms right now, even those with better than average cultures, are dealing with AI code workflows.
Previously the volume of code generated over time was lower, even mediocre code review was useful enough over all the code that people generated.
Today, the ratio of verification capacity to generation capacity has shifted drastically. Even with automated tooling, the amount of verification isn’t enough to stay on top of the generated code.
This changes only when leadership values reviewer capacity, or builds a triage/priority list for code which must be understood.
Humans can be ultra competent if people are actually paying them to be artisans or otherwise motivating them to truly be that way. That's how we have some of the architecture we have today and historically.
> I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.
These are 2 separate things. The crux of the issue is the push towards utility over understanding. The volume of generated code means that it is impossible for understanding about the system to be achieved - and it is now the universal coding experience, not just mediocre firms.
Quantity is its own quality, as well so even a good culture will struggle to stay on top of review.
AI error handling in code review is a lot like said defensive programming in a vacuum: It'll take a lot of code and a lot of energy, but it's unlikely to be important. Even when you are designing something that gets executed hundreds of millions of times a minute, over-review is easy. You pay for it when letting errors in the real world are unacceptable, but it's pretty inefficient. And that's without getting into features that just don't get used, so the entire thing should not have been built in the first place.
In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".
The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.
Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.
IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.
You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.
A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.
But both things can be true. One problem doesn't exclude the other.
If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.
AKA, it would be nice to have nice patches like this one to review.
https://cgit.freebsd.org/src/commit/?id=09d62ba33de113d01e88...
Then your analogy would break down as we all know that the PR machine is a slop producer.
What world do y'all live in where this is your reality?
Code reviews have been notoriously neglected. In the most charitable cases, this was the result of being too bogged down with work or some kind of implementation minutia or time-consuming plumbing to have enough time to actually review code with enough care. That and you aren't necessarily rewarded for code reviews by mediocre companies, so there are perverse economic incentives. Which is disconcerting, because it creates a compulsion to produce without understanding, and understanding is more valuable than artifact.
Reading code is indeed more difficult, because the code's intent, composition, and architecture must be pieced together and inferred with the help of context (formalized languages are infamously bad at communicating intent and most code in the world is full of mixed and half-baked abstractions that further serve to obfuscate that intent, and intent is the measure). So a good code review of anything substantial involves thoughtful effort.
Now that LLMs can catch all sorts of trivial anti-patterns, the need for good judgement and reasoning is hopefully made more conspicuous, because as we know, vibe coders will continue to brush off code reviews, and mediocre programmers will continue to perform them poorly.
The solution is to find ways to automate the process of code approval: more thorough tests and reviews, both made by AI.
It helps when your industry-wide hiring process isn't an institutionalized speed test of ridiculous code golf problems that is memorized from a ridiculous book by every new generation of programmers.
It's almost like we should have been looking for people who can read code all along. But that would be crazy.
I wound up with a bunch of LLM written tools that I don't even bother to try and understand much less review, they either function as desired or I set an LLM to make them function (or just rewrite them) and then forget about them. make sure to keep a prompt in each tool's directory and update it an needed.
sadly this isn't always possible, some systems require complexity at some level. in those cases the tools from above become functions and objects and more care needs to be taken into orchestrating them to limit the necessity of review. data oriented design approaches and functional programming help here.
I guess it depends a lot on how a company manages their SDLC/bureaucracy
Part of a LinkedIn post I wrote in July:
--
A development cycle of a feature is roughly:
1. Request
2. Review
3. Requirements
4. Architecture
5. Development
6. Code Review
7. UAT
8. Deployment
AI has reduced the length of step 5 (development) and step 6 (code review). Code is a prompt away.
AI has increased top-of-funnel (1) volume. While each development step is shorter on average it is resulting in more instances of every other step. Throughput increases and the bottleneck moves one step further down the cycle. The shape of the work is changing from writing code to reviewing code.
I think the bottleneck in the long term will be feature prioritization and specification, and testing. My role has already shifted a lot more into more CI checks to keep LLMs on guardrails
Maybe most developers are mediocre, but all AI code I review is mediocre at best. Implicit dependencies, side effects, weird naming, bad abstractions, or no abstractions, obfuscated blocks of code that can be expressed much more simply... the point is, AI code is not optimized for human consumption, and I disagree that it is better than human code (or, I have just worked with better than average engineers my whole career).
And how do you know those ones are correct?
Well obviously I have meta-agents that review the reviewers
Its agents all the way down
- very well defined initial description of problem plus asking exactly what i want
- fix X
- why do you claim that Y?
- fix Y
- …
I don’t think it would be useful to review or commit prompts
It's an MCP server that Claude uses as we develop so there's a permanent record of everything and work is driven by requirements and decisions.
Mine seems fine. The key is that it must be structured as a graph database and anything can link to anything in the graph.
>> Just updating all the references of decisions becomes onerous.
LLMs are good at that sort of thing. Claude is instructed via MCP to ensure changes are linked when it adds or updates the database and to search the graph for relevant information fragments. Also I have a local GPU running a local LLM on an RTX3060/12GB that assists in keeping data quality and relationships relevant and maintained.
My system is an enormous help - in fact large scale development would be impossible without it.
I'm able to ask it the history of things, decisions supersede previous decisions, I can ask why the software behaves the way it does and why and when that decisions was made, I can pick up development threads that have spun off other threads and everything works real nice.
>> Good luck to you.
Thank you!
Architecture and design decisions may be the only things that matter.
A lot of the time it does not get checked.
A lot of AI practice fell flat because you can't point out the human needs in them. They powered by dreams and wishes and there nothing that ground them.
As for code: ask for better code so reviews will be more straightforward ?
Like, for example: modularization, some structure, maybe libraries for code reuse ? ;) Ooo, and functions, do not forget about functions !
Sorry, didn't started yet with llms because lack of affordable local 256 Gb of gfx memory...
This does not spark joy.
They needed Harvard researchers for that? Anyone in the industry could have told them that. Hell, my cat knows it (cause i complain to her about it incessantly)
And just because it is obviously true to you, it doesn't mean that it is true, either.
(/s)
I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C. And guess what? There are very few assembly programmers these days.
AI code is here to stay and will only grow. Deal with it. More than in any other area it is quite silly to be a Luddite in IT.
Assembly language persisted from the late 1960s up until the late 1990s alongside high level languages. The users of both understood the speed vs. convenience tradeoff.
That this not-even-wrong analogy keeps getting parroted (stochastically?) by the AI bros just discredits them further.
It's baffling the industry is choosing to move in the direction of more reviews which implies more things can fail.
Does the smartest person on your team not have their code reviewed?
Still, humans are needed for the final sign-off for testing on real devices and making sure it works within the greater cultural context.
There's decades worth of this: modelling, automatic code generation, low-code / no-code saas, "everyone should learn to code" and now llms.
Oh, and ageism of course. "we love your experience but we just don't feel it's a good cultural fit, so we're going to make the same mistakes you did a generation ago"