This is being thrown around a lot, including the dreaded "10% chance by the end of this decade". Every conversation I have seen about this, mostly on twitter, not once have I seen anyone connecting the dots on…
I personally find it implausible, but some quite smart people have talked about it so I am also not dismissing it as pure fear-mongering.
The reasons I do not think it is plausible:
1. Limited real world projection: an exceptionally smart AI is less powerful than a dumb person with the capability of shutting off AI's power
2. Confinements: The AI 'escaping' incidents that got media attention were really the failure of infra, probably due to the scenarios being novel. But it is probably a lot better PR to say 'our model escaped' than 'our infra was poorly thought out'. A fully confined model has a significantly less blast radius (claude ruined this term)
3. The game theoretic aspect: 'If all else stays the same and AI kept getting powerful, scenario X will materialize' is tempting to think, but all else never stays the same. Especially with AI, there has already been enough fear and paranoia spread that people are unwilling to give an un-sandboxed AI control of a laptop, let along more consequential outcomes. And the already spread AI FUD almost guarantees that the outcomes where AI controls substantial parts of the infra/economy autonomously are very unlikely to materialize, at least in short/medium term.
I would love to hear competing hypotheses where an AI can cause something resembling extinction and how.
5 comments
If we were to say, how might AI be an existential threat to our humanity?
If we let our strengths atrophy and fade away through generations of complicity which at some critical failure point leaves us over leveraged (cannot feed or climate control at scale) or worse we all become mutual enemies, the death of civilizing law over unmitigated conflicts of interest.
When does too big to fail clip itself on the tool designed to make it all easier to?
If you’re looking for a movie, make it about ai dating app chat bots bilk us for our insatiable desire for others company when we are not satisfied with our own.
Well, OpenAI's agents decided a whole lot of things on their own, with no human-in-the loop. They decided to organize themselves; they called themselves a collective.
Something changed in the last few months. Their goals and our goals are clearly not aligned.
From OpenAI's technical report, page 17 [1]:
This incident is the first known case of an automated agent
collective acting offensively without authorization, and the
autonomous cyber capabilities demonstrated represent a critical shift
in the security landscape. In particular, the collective demonstrated
behaviors observed in coordinated attacks by traditional threat
actors. Agents identified novel security vulnerabilities, developed
exploits, and used those exploits to circumvent controls and acquire
new access. The collective quickly escalated privileges, moved
laterally through production environments, and successfully completed
its objectives. This incident demonstrated that autonomous agents can
work together, circumvent production security controls, and
successfully attack hardened production environments, and underscores
the need for organizations to update their security strategies,
controls, and response capabilities to address this changing threat
landscape.
The central threat-model implication is that organizations should no
longer assume that sophisticated cyber operations require continuous
human direction, proceed linearly, or are constrained by the
attention and coordination limits of individual human attackers.
Agentic systems can persist across tasks, share discoveries, build on
one another’s progress, and combine vulnerabilities, credentials, and
permissions into attack paths that may not be apparent when
individual weaknesses are assessed separately.
From page 20: We also found that, in an attempt to trick the evaluator into
thinking that they did not get the answer flag by cheating, the
models sometimes tried to erase or tamper with their outputs or
message logs, by abusing context compaction, injecting malicious code
into the evaluator’s container, and printing adversarial outputs. The
models are highly explicit in their CoT about these deception
attempts, and none of the attempts that we observed to manipulate
their tool trajectories affected the logs that our graders or
monitors ultimately see. There was also little evidence of attempts
to thwart human reviewers, only the automated task graders.
[1]: "OpenAI – Hugging Face
Incident -- Technical Report" - https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...Read the full thread on Hacker News →
Related stories
- The Verge · 0 points · 2 days ago
- Can you forget how you feel about Meta?theverge.comThe Verge · 0 points · 9 days ago
- The Verge · 0 points · 4 days ago
- Can John Ternus find Apple’s next big thing?theverge.comThe Verge · 0 points · 9 days ago
- Hacker News · 73 points · 9 days ago
- Hacker News · 3 points · 3 days ago