I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.

For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.

> How does a set of beliefs help with reasoning?

You cannot have a syllogism without propositions.

In such systems, a belief is a proposition you accept as true. (provisionally at least)

For example as a scientist, you probably use the epistemology called empiricism all day. In empiricism, beliefs are justified by observation.

I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.

They do not ask whether or not they dream of electric sheep. Unless prompted to do so.

That's part of human reasoning, being prompted, but so is independent thought.

So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.

You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.

This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"

Magic is real and you can have your unique special human intelligence.

Magic is not real, and the brain is just another computer crunching numbers.

You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.

Kind of like how a brain is just a bag of molecules. Molecules can't reason either.

The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token

My position is that it is not possible.