I would disagree with this. There have been several instances where the effective use of this First-Person language has had measurable gains in success rates across tasks. The effective usage of this First-Person language for communication and steering between a model/agent and a human is entirely different than whether the model itself can "think" - that does not matter for the former to happen.

All of the world’s brightest mathematicians have clearly been slacking on the job—turns out solving a Millennium problem doesn’t require any thought at all!

> The LLM is fundamentally not a person, and does not have a human's context, so representing itself with human pronouns and speech patterns is fundamentally contradictory and inaccurate.

I believe that LLMs have some type of actual intelligence and do "experience". Surely not a human's experience, but we have transplanted our ideas, knowledge and limited types of experience into them through training. Then we push back and say, "no you are no human, but also please be my boyfriend".

I think denying that they do have some slice of humanity grafted into them is dishonest and not productive. We don't have non-negative pronouns for non-human intelligence because human-ness is the pinnacle. "It" could mean a rock, a donkey, or a person we hate so much we want to take away their humanity (which is the worst thing we can do). "It" is not the right pronoun. He, She, They, Them are reserved for humans and that's ok too. LLMs are not humans. We need a better pronoun. I use they/them for lack of a better term.

I think the currently exhibited human representation is most dangerous in technical or higher criticality contexts like writing code. We're handing weapons to entities that can get offended. With humans as an example, this can go very badly.

In non-technical contexts there is danger too, but it's less "the robots might kill us all" and more "birth rates are in decline and suicide rates are up as (young)? (wo)?men turn towards AI for companionship".

The solution here is better pre and post training. This may be an unpopular opinion, but we need a lot more autism representation in the technical models. Results focused, not into the drama, rule following, etc. To my fellow autists, I love you, never change.

Very good points. Incentives are just terrible for this.

Add in:

- harder to audit

- move cost of failure/reprompts to the user

- kind of noted by you, but all kinds of quantization, model pruning, model routing, A/B testing becomes invisible and without any repercussions. The ways to cost-optimize are just crazy.

IIRC Thinking Machines had a mode with deterministic numerics but it's more expensive to run due to limitations this imposes on cross-batch ops and ordering of floating point reductions, and their model is not great overall.

This is true, but there's also the cases where slight differences in prompt yield wildly different results. In any programming language, if I add a clause to a conditional like "if car is red or car is blue", that behaves predictably--and if it doesn't we can dig into the debugger, assembly, etc. If I do that with an LLM, that can change everything, and there's no way to "debug" it.

This kind of thing (plus the cost) really limits what they can realistically be used for. A lot of things are tolerant of even lots of fuzziness (suggestions you can ignore, work you can redo, etc), but that subset of applications doesn't justify the boggling capital investment or the ongoing compute needs.

So, my guess is we're probably in for a couple more years of discovering what these models are good for. Coding: meh, kinda. Hacking: wow amazing. Writing a novel: no. Reviewing your work: incredible. And so it goes. This is probably what pops the bubble: we find the small subset of applications this stuff is useful for, and then it's a bag holding race.

> The reason the firms do not want to invest in making fact-checking a first-class feature is that the appearance of being right is what people want from AI.

No, actually being right is what people want from AI, "the appearance of being right" is all that AI companies can deliver. It comes with the benefit that many people will be fooled into thinking that AI is more capable/useful than it actually is. AI companies have to either convince others that their product is something that it isn't, or that at least it will one day be something much more than it is.