I think the problem is that you're using basing your conclusion from the cheap/dumb models available on the free tier of services. I just asked GPT6-Astra in Codex and it replied:

"It’s ASCII art of a bare foot and lower leg, with the toes pointing to the right."

No tool calling, just an immediate reply with the correct answer.

For me, 6.1 Sol nailed it immediately:

> A bare foot and ankle, pointing right, with three little toes.

I wonder how much of the wide variation in perceptions of LLM capabilities is driven by the gulf between free models and frontier models. Luna getting something wrong is not always great evidence for LLMs be unable to do that thing.

Edit: for curious skeptics without access to 6.1 Sol, I tried 3 times and it got it all 3 times. Convo share link: https://chatgpt.com/share/e/6abeb955-7614-832e-a5e1-b1bd134f...

Mm, I kina agree with the AI on this one:

   (_)(_)(_) represents the wheels

They do look rather wheel-like; I have to assume you see them as toes though?

It's like the duck-bunny picture to me. If I focus on the "wheels", I see a steam train locomotive (but perhaps I'm only seeing that because I read your comment?); if I look at the ankle I see a foot.

Readers: before you vote or comment, look at that foot.

I think I would have failed this test!

To be fair if a human was given a linear sequence representing ascii art you couldn't tell either

Sigh, the whole "obviously the turing test is solved" meme is annoying.

Like, if we meant that it convincingly masquerades as a shitposter, ok. But everyone still bitches about AI slop, and everyone knows the writing is still bad. How does that even work if the turing test is obviously solved?

More to the point though, if you grill SOTA models on counterfactuals, causal world-models etc, you'll trip them up in a way that actually will not work on ESL students and children. Certainly there's no way to find a person that struggles with that and is also capable of cheerful fluent erudite discussion about astrophysics with perfect grammar. Yes, it's getting harder obviously.. but detecting machines with determined, focused and intelligent interrogation remains pretty easy. If nothing else, the models are cooperative where people wouldn't be and that's a signal too.

The best progress we've made is that most people do agree that this doesn't practically matter very much, i.e. we generally recognize the stakes were always overstated. But the constant vague appeals to common-sense that "of course it's a solved problem!" always feels naive or fake.

Well, I can answer this one: https://stoppels.ch/goalposts/?c=40662140

jerf, 2024: "If it could be solved with a Math Overflow-post level of effort, even from Terence Tao, it isn't what I was talking about as "high level math".

"I also am not surprised by "Consider a generation function" coming out of an LLM. I am talking about a system that could solve that problem, entirely, as doing high level math. A system that can emit "have you considered using wood?" is not a system that can build a house autonomously.

"It especially won't seem all that useful next to the generation of AIs I anticipate to be coming which use LLMs as a component to understand the world but are not just big LLMs."

The voting gloss: "An AI fully solves a research-level math problem on its own, not just suggesting an approach."

Yes, I'm satisfied. I don't even feel bad in hindsight. Coding assistants had a nice, gradual rise up the utility curve. Math went from "lol, can't add two six-digit numbers" to research-math level almost overnight in comparison.

Right - people on HN are generally reasonable about objective things. The vast majority of comments (outside those chosen for this website) are not "AI will never ..." but rather, "AI does not currently ...". Of course the further you go back (I'm seeing a lot of comments from ten years ago!) the more skeptical they get, obviously. That's a funny thing to go back and see with modern context, but it doesn't really call for snideness/mockery (something I think is sadly increasing on HN).

> cannot do precise things like coding software since humans will never be able to use natural language to specify their requirements.

To your point, this example. The issue expressed here is with humans, not AI. We are still pretty terrible at writing specs. TBF, the AIs are too but that wasn’t being voted on.

Yeah this shouldn't be flagged, it's a neat project.

Somewhat appropriate the site the OP links to is called „goalposts“ because as far as I can see, people keep shifting theirs.

In your case, the comment you link to says „business tasks“ and you expanded it now to „arbitrary new tasks“. Those are not the same. An LLM today sure can do many many many business-speak conversion tasks.

> reliably convert business-speak into efficient bug-free code

I actually think this would take AGI to solve, which makes me optimistic about the future of software development.

All the benchmarks are currently testing against automated tests the AI can use as an oracle

so the conditions for your prediction simply haven't been met yet.

if/when you can tell a model to do a thing and be confident that it did the thing, it's joever for 90% of knowledge workers.