275 comments
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
What would it mean for a vehicle to be "autonomous", but to "not use AI"?
Consider how trivial it would be to write a program that would destroy every computer on the world. Any competent developer could write this in a hour. An LLM in 5 minutes. The trick is actually getting the code onto every computer in the world and running it! Until someone grants it agency and runs it IRL, the artifact has zero impact.
The models can produce whatever scary text they want. But if a human acts on it, connects that with their email or their drone weapon, it's the human who bears total responsibility. We desperately need some legislation to enforce this; otherwise I see a future where almost any accountability can be avoided by AI-washing the problem.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
* number pulled out of my ass
While that is extreme, if there is a high p(doom) it is also reasonable.
We can absolutely do things to control AI and fine or imprison those who can’t control theirs. This is not some inevitable outcome.
We could vaporize these companies tomorrow if we wanted.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.
We should not only regulate AI more than we do corporations, but we should more strongly regulate corporations as well. There are corporations right now that are killing people, using slaves, and bribing governments just to increase the already massive amounts of wealth and power they have. If AI is showing any signs that it is going to act like corporations do we'd better act quickly.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
The legal model to look at is the European Union's General Data Protection Regulation. That regulates what companies can do with data and how they can use it against people. You can develop anything you want in the data mining and analysis area, but the GPDR restricts how companies can use the results.
Focusing on the development process moves political attention away from what modest AI systems can do to people. They don't have to be super intelligent. Just super attentive. Always watching. Current technology is more than sufficient for oppression and control purposes.
This is the near term threat.
There's the threat to jobs, but that's something society has handled before. Some countries better than others.
Also, not sure it makes a difference in relation to AI whether one thinks of AI more like corporate or a non-corporate thing to regulate. The basics might stay the same, e.g., who can do what under what procedures.
When were anti trust, anti cartel, anti monopoly laws last enforced?
The companies and their employees must be held accountable: if the AI companies can't develop AI safely, they must cease their operations. If employees can't work safely, they must stop working.
Not enough attention is given to February 2026. That month, Anthropic changed its scaling policy roughly from A:
1 [To get the weapon I must endanger innocent others.]
2 [It is wrong to endanger innocent others.]
3 [I will not endanger innocent others.]
To roughly this instead:
1 [If I do not get the weapon, someone else will get the weapon.]
2 [I should have the weapon because I am the best.]
3 [To get the weapon I must endanger innocent others.]
4 [I should endanger innocent others because I am the best.]
This is based on the intuition:
[I should harm others to achieve my moral goals.] (act utilitarianism)
Anthropic has no mandate for these goals. I do not give them my consent to harm others on my behalf. I hope that politicians will communicate that these policies are not acceptable.
The employees of anthropic are responsible for what they do regardless of how much they get paid to do it. If they can't work safely, they must stop.
I prefer to call them "the major LLM manufacturers". They're companies like any other, manufacturing and selling complicated software like many others.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
If you think AI can help design new drugs, great! Let's pick some restricted DSL for describing molecules, preparation instructions, hook it up to the right set of lab robots that can pipette and centrifuge and whatever, _restrict its output to that DSL_, and let it iterate. The 'let an AI design drugs' process doesn't require that the agent can also, say, hack a niche German wiki.
For every area where we think these AIs can be _so valuable_ because they'll find solutions that humans wouldn't (or find them faster), then doesn't that value also imply it would be worth it for some humans to do some advanced setup of their specific domain-specific tools so it can be run in a sandbox equipped with only what it needs and not more?
Specifically, bad at search and returning information with references, bad at discovering and debugging package versioning conflicts, etc.
It's a Metcalf thing. The utility of an agent grows in some fn of the tools it owns. Skilled tool use comes from rich training environments.
Either you're being pedantic and missing the entire point, or you're saying that Anthropic lied and made a fake sandbox knowing their agents would need to connect to the internet anyways.
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
I'll be impressed if AI agents find a way to get infected USB drives out into meatspace.
The original quote follows with "AI", but it should be clear by now that you could put any topic here and the main issue is: private conglomerate money trying to control people.
And the remedies are well known: breaking corporate power and power concentrations through 1950s style regulation, as it was deployed against the robber barrons from back then and which resulted in a golden age for America and the world.
Read the full thread on Hacker News →
Related stories
- The Verge · 0 points · 2 days ago
- Can you forget how you feel about Meta?theverge.comThe Verge · 0 points · 9 days ago
- The Verge · 0 points · 4 days ago
- Can John Ternus find Apple’s next big thing?theverge.comThe Verge · 0 points · 9 days ago
- Hacker News · 73 points · 8 days ago
- The Verge · 0 points · 11 days ago