Experience is not understanding, but an LLM does not reason so it cannot understand.

It can produce text that looks like reasoning. It can even produce text with mostly sound logic, but there is no internal experience or reasoning that occured there just the generation of language.

LLMs therefore tend to be very bad at tasks that involve meta cognition. I've yet to successfully convince one to tell me when it knows something.

> LLM is next token generator

> Isn’t the human brain also just a next token generator?

The two most cliche messages on this forum. They occur in every LLM discussion. It’s fascinating.

The word itself? There's little practical difference.

The concept, though, it's a very wide moat. You could call a unicorn "nyati" for all we care, we still know it's the concept of a magical flying horse with a single horn. We know, besides the literary corpus, the concepts of magic, horse, flying, and horns. It's fictional, yet we have a very good idea of what if would sound, feel, or even smell like. Ask an LLM to describe what a unicorn feels like, and it will ramble about forests and sparkles.

In fact, I asked Gemini (thinking, to see the process) to describe a mindful experience about meeting an unicorn in real life, and it did ramble about the event. When I reminded it about mindfulness being about experiencing with all your senses, and asked it to focus on the creature, to its credit, it even described the taste:

> Taste: Even the air surrounding the creature tastes different on your tongue—thin, crisp, and tinged with a faint, sweet metallic tang, like snow melting on limestone or fresh rain falling through high canopy.

Still nonsense, as a unicorn (or the air around it) will likely taste like horse, whatever that flavor is. It finishes with more nonsense, and I doubt the flash version will give better results.

> Every micro-detail of its anatomy becomes an anchor to the present moment. You are not thinking about what it means or where it came from; you are simply perceiving the texture, heat, sound, and weight of a living, breathing reality standing inches away.

Still, current LLMs can do amazing things given their inherent limitations, and that makes it easy for us to overestimate their capabilities.

Yeah. LLMs are great at convincing people they could be amazing, that they're so close to being amazing, if only they just figure out how to bridge that last gap... But that last gap is fundamentally unbridgeable because of the way LLMs work.

It's a very effectively designed human trap, in this and other ways.

> I just asked Claude

How well are LLMs able to reason about their own behavior?

Put another way, this entire post is about getting AI to not bluff. How do you know that it’s not bluffing in its response to you?

> I've never heard of grounding in this context, so it may help to describe what you want more clearly

Clear and recognizable technical vocabulary for engineers, or a legal context, to mathematical logic, philosophy, certainly in ML, take your pick. I would think it's pretty familiar to everyone who speaks English and if not still clear with context clues

> I've never heard of grounding in this context, so it may help to describe what you want more clearly

An eval of this is likely worthwhile;

Re: "Grounded in logic" and "Grounded in theory"

Ground and justify all of the responses with logic and theory and real observations from qualified experiments with citations.

Present a coherent argument borne of logical premises with extant sufficient proven evidence of support. Assess and critique the response given such criteria that all responses should be valid logical arguments, and revise before responding