If only one person knows the code, then PR time isn't going to save you.

I'm tech lead and I basically don't review PRs, and I tell people this, with a caveat - if you can tell me what you specifically want me to review, for what specific purpose, I'm happy to!

So "can you review this bit for race conditions" is great, love it. This forces people to actually think about what in their code they should be suspicious of, if anything.

"Can you review this" [link to PR] is getting a rubber stamp because humans have never been good enough at "just spotting bugs" to make this worth it and now any LLM is better than a human.

And for the purpose of understanding - PR time is too late. I have not reviewed a PR ("for real") in a long time and yet I could tell you how every system my people have built works down to a very fine level of detail. And it's because _we talk to each other!_ We don't just chill in the same slack channel and code independently, we all value each others brains and want each others inputs because we know it will improve our product and we value what perspectives others will bring.

Trying to learn via PR is a sad substitute for real collaboration and teamwork.

Missing my biggest issues as you ask the agents to do larger tasks with less up front planning.

Is there already a pattern or code on in in the existing codebase that handles this functionality,

Do we really need net new code to achieve this functionality?

Can existing code be extended or abstracted to more cleanly implement this feature or functionality.

- Do we want this? Cost/Benefit etc

- Is the change architecturally right?

Particularly the latter LLMs seem still pretty useless at.

Code review also transfers knowledge to the reviewers!

I think we're kind of missing a layer of testing, that should sit above unit and integration tests.

Something akin to "meta-tests", which are not about testing the code itself, but the approaches taken by the implementation - i.e. architecture, understandability, terseness, etc.

These tests would operate on the source code level, even when testing code for a compiled language.

> I think LLMs are okay at most of these, and worst at the first.

LLMs are worst at not realizing problems that I'd call "meta" problems. Here's one example to illustrate it:

I was allowed by my employer to work on a small project within the large collection of the projects which all constitute the product the company sells. Like a few dozens of other projects, it's written in Python. The company doesn't have any explicit policies about how Python projects have to be organized, it requires testing, linting, a CI code to package it etc, but the guidelines are very permissive. It just so happens that, beside the guidelines, there's a tradition: every other Python project in my company uses the typical Python bloatware, like masonry with a lot of insanity and mental flips going on in pyproject.toml, which is, in general, very typical for Python community at large.

My project used none of that. Instead, I wrote a ~100 lines setup.py file (no dependency on setuptools/distutils) that assembles the wheel and runs project maintenance tasks in the same way (interface-wise) things used to work decade or two ago (eg. "./setup.py test" if you want to run unit tests).

The AI reviewer didn't bat an eyelash. Found some typos in the comments, a problem with Base64 formatting, and generally OK'd the whole thing.

I knew I was on my way out. And I generally enjoy seeing people having a fit of rage when they know they are wrong (especially, together with many more like them), and scrambling for arguments that they know to be lies. I felt a little bit vindicated for the years of suffering I had to endure working with what might have been the dumbest and the most entitled manager I had in my life. :D

Anyways. My point is: the AI caught none of it. It was very happy with my approach to Python project management.

* * *

While my story is... more of an odd case, where this does have much wider implications is the AI-generated code. AI-generated code often fails to match these meta-requirements. I've seen AI reviewer OK'ing a PR containing AI-generated 10K loc Python file. I human would probably break after reading the first 1K lines. But AI doesn't get "tired", it just kept picking on typos in comments, criticizing short variables names etc. And there are other aspects in which AI-generated code is weird to humans in the ways that humans simply won't accept it, but AI reviewer would completely ignore as non-issue.

Well, I can tell you that at my current job we had something of a crisis over the summer when we realized that nobody could make changes without fear of breaking things anymore because we lost the ability to tell which existing behaviors were and were not safe to change.

Previously the knowledge needed to discern that sort of thing would be disseminated through both design and code review sessions. But plan mode and AI code review largely put an end to that.

So we put our heads together and came up with some new policies about project management and how we use AI, and things have steadily getting better since then.

(Though, in fairness, the one guy who seems to actually enjoy getting paged after hours seems to be having less fun.)

> If companies want to compete in the world of AI generated code, human code review has to go. The only question is, what replaces it?

You’re going a bit hand wavy for an answer by redefining the term into something that fits what you’re promoting.

> If companies want to compete in the world of AI generated code, human code review has to go.

This is a HUGE non-sequitur. It only follows if by "compete" you mean producing more LoC. How often does that translate to market fit or economic success?

This is the mindset that makes me want to leave this industry immediately. Somehow, an industry that already annoyed me with how much "mediocre is good enough" was an acceptable stance, with the emergence of LLM coding tools suddenly decided that "absolute dogshit is good enough" was just as acceptable, as long as everybody else is also fine with eliminating the few quality standards they might have had.

I guess out in the wild vibe coders will tell you code review is replaced by "prompt review"

People forget that pull requests and doing code reviews in the context of those is still a fairly recent thing. People did some code reviews before that of course but nowhere near as strictly. Same with testing practices, static code analysis, etc. Most of that wasn't all that common until beginning of this century. I remember using findbugs with Java around 2004. It actually found bugs in my code the first time I used it. No review had caught those. And we got lucky not finding them in production. But they were definitely bugs. Our system didn't have unit tests; it was all manual. Junit was a fairly recent system that hadn't been around for that long yet. Our build was done with Ant. There was no test phase. Our tech lead would of course check my work and correct & educate me (I learned a lot). But a lot of bugs slipped through as well.

Git did not exist either, I migrated out cvs to a beta release of Subversion. We only used branches for releases. We'd cut a branch just before a release. Test it (manually) and then ship. That was a process I helped put in place actually. After release, master would diverge quickly so back porting fixes was not really a thing. We'd support releases for as long as our customers used them. Often that involved just upgrading them to the recent version. We shipped when things were good enough.

I think the notion of people reviewing any meaningful amount of generated code is simply delusional. As you say, we do need alternative means to replace those checks. And a lot of that is going to be AI driven as well. AI driven testing, code reviews, and all the rest. Essentially all the stuff we used to do manually (poorly).

And we do have an important new tool as well: clean room code replacement. That used to be prohibitively expensive but now it's not. If you have something that is well specified through documentation, APIs, specifications, tests, etc. replacing it is fairly straightforward now. There are some early examples of people using LLMs to generate functioning replacements for things like Postgresql, browsers, compilers and similarly large and complex systems. While not perfect, these things seem to work, pass their tests, and generally not be completely horrible. It's only going to get better from here.

The notion that people are going to ever manually review code that was generated for such systems in mere hours/days is beyond imagination. How? When? Who? Why? It simply does not scale. It's only going to be more and more code. The amount of code no person will have ever looked at will soon dwarf the amount of code that is still manually inspected/created pretty rapidly.

"does the test coverage adequately demonstrate the functionality of the code?"

and is this a solved problem? If not, then the bottleneck is right here, if it is solved, then yeah we shouldn't need anymore software engineers other than the elites

This does not reflect my experience. AI code review has improved tremendously in the last years to the point that nobody in my team does manual code review any more. It used to be ridiculously bad two years ago, but not any more.

Outdated/bad CICD is surely extremely common in every org