> I never understood the code. You think it works a certain way, until you find out that it doesn't. What LLMs make possible is for me to say: find out all the ways this thing works. Analyze the different ways we can run this software, build a fuzzer, build property tests, and run this software in every scenario possible. Log full traces. Log all the outputs. Now, analyze each scenario for bugs. You can't do that by hand.

Testing isn’t the same as understanding the code, or proving (even informally) that it is correct. Having the LLM do all these things above doesn’t lead you or the LLM to understand the code, to logically reason about its behavior over all possible states and inputs.

“Finding out that it doesn't” means that you didn’t properly reason through the code beforehand, checking all your assumptions against what the code and underlying systems are actually guaranteeing. This may be a matter of formal education (proving computer science theorems and algorithmic correctness in university), I don’t know.

> Reading the code does not mean you understand the code.

Reading the code may not be enough to understand the behaviour of your program, but believing you can understand the behaviour of a program without at least reading the high level code is truly silly.

(by high level, I mean the code living in the higher layers - of course we don't often read the code of the generated assembly, or the interpreter, or the browser, but that's because they're reliable abstractions, unlike prompts!)

Strange that the world worked before 2024 and software gets worse now. Your debit card transactions for example worked.

This sounds like a typical testimonial whose mind has become captive to Claude. It is like Scientology.

> What LLMs make possible is for me to say: find out all the ways this thing works.

They are not good at that. The space of possibilities can be massive and LLMs are terrible at exploring such space because they predict from the prior tokens they made. They are inherently bad at exploring new space because it is antithetical to how they work.

To me it's deeply concerning that so many people are getting fooled into thinking that LLMs are actually good at covering their bases like you're describing here. It's one of their weakest qualities.

> Analyze the different ways we can run this software, build a fuzzer, build property tests, and run this software in every scenario possible.

It usually does quite a bad job at this too, often the tests it wrote feel like that of a lazy student that didn't really want to do the task and just sort of cheats at it or does a really shallow job. It certainly cannot run the software in every scenario possible.

> Anyway all of this reads like someone who is not actually using LLMs to build software or hasn't tried them in a while

This is something that is said by everyone who contests anyone pointing out the risks in being overly trusting of AI or otherwise points out their flaws. I use the latest and greatest all the time and all the time I'll point out something that it totally overlooked and get hit with the classic "you're absolutely right". This occurs because I actually read the code and can see the myriad of blatant issues that still occur when using LLMs and know better than to trust them. You will find so many issue by delving into the details.

I don't think the discrepancy is in LLM capability improvements over the past year.

Correctness has never been a priority across an industry where rapid iteration and feature delivery drive sales. There's always some opportunity cost to doing things right, at the price of technical debt down the road. If AI is primarily used to produce fragile code, people will be wary of AI solutions. There's also ongoing public debate about AI safety and alignment. Deploying AI in safety critical applications feels riskier than ever in the current environment, even though it doesn't have to be.

> a thing that previously somewhat prevented lazy and incompetent developers from pushing out horrible code

Brings to mind this classification https://en.wikipedia.org/wiki/Kurt_von_Hammerstein-Equord#Cl...

"""I distinguish four types. There are clever, hardworking, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and hardworking; their place is the General Staff. The next ones are stupid and lazy; they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the mental clarity and strength of nerve necessary for difficult decisions. One must beware of anyone who is both stupid and hardworking; he must not be entrusted with any responsibility because he will always only cause damage"""

Even if you are competent I cannot review your 5,000 lines of code you produce per day vs the 100 you were producing before the LLM apocalypse.

I'm seeing this too. I've worked with devs that would previously push PRs that wouldn't work or run correctly. Those PRs wouldn't get merged in. Now they're putting up PRs which seem to work at first glance, but have hidden problems. For example, one guy introduced a huge PR for a visualization and it seemed to work fine, though another dev mentioned to me that we already use recharts and it does 90% of what this guy's PR does (his code does all the drawing logic itself). Maybe AI will get good enough to clean up these kinds of messes, but in the near term I imagine there will be a lot of code bases that will be filling up with dragons.

At a startup I worked, there was an engineer whose code was incoherent and buggy. So, we were literally better off if that engineer did nothing because their net output was negative. Engineers like that become weaponized with LLMs, and negative numbers become larger negative numbers when scaled up.

Limitations of AI are a thing; but one rhetorical point keeps coming up (I don't think it's just you) and confusing me:

> I hope you can see the stupidity here if you expect to see any deterministic results at all.

Are you expecting humans to be deterministic in the code they produce?

We also read these kinds of rebuttals almost every day. The original article made a number of substantive critiques about where AI falls short in the actual requirements for building maintainable, reliable, business-critical software. So its not enough just to point to newer models without explaining how the newer models solve these problems. Can the new models take full end-to-end ownership of a system? If not, how do they solve the problem of humans taking ownership of AI generated code?

I use the latest models every day for app development, it does still make many mistakes and poor decisions. Just today I dealt with an issue where it allowed a silent failure that would delete all of the users data without anyone noticing. The code it writes for one task is usually pretty good, but it still doesn’t always follow conventions well even if you’ve documented them. It also has no sense for long-term architecture or organization, or when to make tradeoffs for less complexity because it’s a temporary or prototype feature that needs to stay easy to change. I’ve found that if I do even 3 months of pure agentic coding with no code reviews, it’s aged 10x faster than a human coded codebase so it’s like a 30 month legacy codebase now. I know people who are obsessed with AI coding and they’re throwing away projects they’ve built over a year and starting over because it’s become too slow to make changes.

What kind of coding are you using these models for? Most of the people I know who share your perspective never go beyond the prototyping stage. I’d be curious to hear from anyone who’s been AI coding for more than six months, shipping it to real users, and isn’t looking at their code at all.

>This article would be 100% correct if it came out 1 year ago, 75% correct 9 months ago, 50% correct 3 months ago and it's probably 25% correct now if not less.

Feels like I read comment similar to this one each year since 2023.

I've been reading such arguments more or less every day for years now: dude, you need to try the latest model. Forget about last week's model, it didn't work. This week's model is the real deal.

> But just go ahead and give the latest models (Opus 5.5 / Astra 6 as of today) another try. See what they are capable of and read the code which they produce.

Well, for one, they're capable of draining our (or companies') wallets.

I resolved a huge merge conflict for $60 today. Opus 5.5 did a great job and spent just 1h 16min on this. I could probably run six such sessions today, if I disregarded the need to read and understand the code.

This money has to come from somewhere and my concern is that it will be from decreasing the number of people hired and/or their salaries.

At the same time I firmly believe people who had a tendency to produce tech debt will keep doing that, regardless how brilliant LLMs will become. Unscrewing this is going to cost a lot of money.

Came here to give an answer but your last sentence kinda made the point I was gonna make. If one is legally in control, then one is accountable (the dog or unsafe toy example in reality is OpenAI's agents hacking huggingface for example).

The difference seems to be that some companies are above the law apparently.

I’ve been working in enterprise for ten years and I wouldn’t call any of the services I’ve worked on simple.

I wish that were the case, my multiple experiences are quite different. its not that the software doesn't work - but the software doesn't actually capture the process, or properly talk with other systems, or isn't set up for a new business model or product line...then you have layers and layers of "tools" created to make it work, and a bunch of people using spreadsheets and csv files to do workarounds, and then the guy that wrote a bunch of the tool's 10 years ago leaves and there is no documentation or it needs to be re written in order to upgrade some other piece of the system.

i guess another way of saying this is that on the micro level a lot of this stuff is not rocket science, but at the macro level it becomes hugely complex.

The majority of existing software runs without a glitch? Are you being serious?