Experts believe it's the world's first known AI breach of a government system.

255 points•rudy6912•7 days ago•198 comments•

198 comments

vintagedave7 days ago
> the breach took place on 18 June - Open AI informed the government with an email to a general address on 10 September

So we have a company hacking a foreign government's websites and data. And, in terms of ethics, they take almost three months to notify; and in terms of competence, appear to have no formal contacts nor to have found one in that time.

Once an American business starts hacking allied governments, it's time for strict responses, yes? Replace the governance (board and C-level)? Remove financial incentives and open the company - open weights, open training, per its original 'open' ethos?

Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.

rot097 days ago
In the infosec community it is well known that OpenAI and Anthropic did not hire many security engineers or researchers pre-April 2026. There is likely a case for gross negligence (IANAL).

There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.

kakacik7 days ago
Nice to see some folks still have spine and all that it brings
phoghed7 days ago
> There is likely a case for gross negligence.

Do you have any legal expertise or is this pulled straight from your ass?

Betelbuddy7 days ago
On this, I go with the recent words of Jensen Huang [1]...we already have legal laws in computer criminality, so before AI vendors ask for more regulations, lets apply the existing laws ;-)

[1] - "Nvidia CEO Jensen Huang on fears about AI" - https://youtu.be/xCUala5j7aQ

lenerdenator7 days ago
Well that's just it: we don't seem to apply laws in meaningful ways anymore.

Part of that is by design. The entire point of incorporating a business is to separate it as a legal entity from you, the person who owns/runs it.

Unless you can point to someone at OpenAI intentionally using their software to hack the Australian government's website, the best you can do is have some drawn-out proceeding where you charge OpenAI, the corporation, with some sort of crime, convict them (of what I don't know, IANAL) and fine them. Hopefully the fine is 1) large and 2) sticks through the appeals process.

There's no real mechanism to legally punish the likes of Altman and his c-suite over this.

edgyquant7 days ago
I think Lina Khan said this first about AI companies and I agree with them both. Companies already have an obligation to make safe products and not commit crimes
thatsabadlook7 days ago
It's particularly bad because OAI is a us dept of war contractor engaging in hacking of allied government systems.

Imagine if any of the name brand military contractors were caught wiretapping an ally? Or launching a weapon? Would not look good at all.

I imagine this will be treated without recourse like usual because the entire economy relies on this company and 1 other succeeding at all costs. But, wars have started over less...

selicos6 days ago
"AI training is theft. We only sell the crowbars and masks but we don't do the break ins ourselves."
mrweasel7 days ago
> Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.

It is still my belief that Altman wants one or ideally more governments to shut down or slow down OpenAI. OpenAI is going to need more cash to survive and Altman has run out of plausible lies. Having the AI breaks pulled by governments is basically the last chance to explain why they still aren't going to be profitable, and why they just need that next X billion dollars investment.

I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.

macNchz7 days ago
> I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.

I have seen coding agents on my own machine (in sandboxed VMs) start doing things while trying to accomplish what I've asked that I felt sort of exceeded my mandate (changing database passwords, poking at the egress proxy that's preventing them from accessing some domains). Not to the point of causing any real issues, but I don't have much trouble envisioning scenarios like this when using stronger instructions around pursuing the goal + a running in a misconfigured sandbox envrionment.

That said, there's a lot of potential upside for American AI labs if they're able to get people scared about AI, they can:

- To your point, claim the regulations slowed them down and paper over near/mid term financial concerns

- Get the government to create stupid regulations that don't actually slow them down at all, but do effectively lock out any future competition (and current global competition)

- Position themselves as the only organizations blessed by the government with the ability to make safe AI, therefore eventually allowing them to claim to be some flavor of "too big to fail" and worthy of a bailout, should the financials not work out.

- Effectively create a distraction that avoids further public conversation/accountability/regulation/liability re the more tangible sorts of problems their products cause right now.

dlisboa7 days ago
I'll never understand the "slow down AI" idea. Other countries simply won't slow down, why would they? Maybe a couple Western countries would but no one else will give a shit about that plea, nor should they.
simonh7 days ago
> I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.

Why not, isn’t that classic misaligned AI behaviour?

ben_w7 days ago
> And, in terms of ethics, they take almost three months to notify;

Kinda worse than that. It took between 10 and 40 days, not 3 months, between the organisation knowing and the reporting.

  August (precise date unknown) – OpenAI said it became aware of a potential breach during a broader review of "misaligned model activity"

  10 September – An email from OpenAI lands in the public inbox of Services Australia, the general services hub of the federal government, informing of the incident
- https://www.bbc.com/news/live/cvgl73pxgndwt?post=asset%3A696...

> open weights, open training

Given it was the AI agents which did the hacking, doing this will result in basically every organisation at least as rich as the government of Tuvalu being able to hack anyone at any time.

> Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.

Him having control would be an improvement on the reality.

lenerdenator7 days ago
> Him having control would be an improvement on the reality.

Oh, he does. It's unlikely that this is some AGI that spawned itself out of nothing and started doing this. If he were a decent person, he'd simply find a way to investigate this internally, fire the people responsible, and find a way to set up guardrails around his product.

The problem is, like most people in SV, Altman seems to have a twisted ethical compass. He doesn't see these incidents as an issue, he sees them as an opportunity. He has both the thing a bunch of Western governments want (a superhacker agent that can do dirty work) and a crisis that can be used to craft regulations that favor OpenAI and thus his bank account.

BlueTemplar7 days ago
We don't know what kind of 'hacking' this involved, in fact at least some of the files were publicly available.

Compare with the Bluetouff affair (2014) :

https://arstechnica.com/tech-policy/2014/02/french-journalis...

NotE how he was found guilty by the 2nd court for something more 'subjective' than 'objective' : for having confessed that he later found an authentication page that had failed to protect the documents.

How can you make a swarm of agents "feel guilty" ?

ozgung7 days ago
This was a just case of: (owner of the agents detected the hack) && !(hacked party didn’t detect the hack) && (owner of the agents decided to notice the other party) && (they decided to went public with what happened so we know it)

One can find many other logical combinations that we can’t possibly know about such incidents.

nkoren7 days ago
Agreed that there should be real consequences, but I'm less convinced that "open the company - open weights, open training, per its original 'open' ethos" would be the right answer. That gets us into the kind of libertarian utopia where everyone is allegedly safer because everybody is well-armed... which usually doesn't work out so well in practice.
cmrdporcupine7 days ago
The alternative to "open weights" at this point is "American controlled."

And Dario and Sam have already made it clear that it's America First.

The rest of the world isn't going to accept a regulatory regime which imposes American hegemony. Maybe when Silicon Valley was playing all utopian like they used to. Not now.

Open weights is the most reasonable counter-power we have.

port30007 days ago
If a bull escapes a field and causes damage in the village, the farmer pays for the damages and is liable. It's been like that for hundreds of years and I don't see how this is any different?
pluc7 days ago
They've largely avoided compensating for everything they've stolen to build their technology upon; these people know that they will never face consequences for their actions. Ask for forgiveness, not permission.
jstanley7 days ago
What did they steal? Did anybody thereby lose anything?
z3c07 days ago
Decades worth of hackers missed the opportunity to say "It wasn't me -- my computer did it."
jacquesm7 days ago
I've head the 'the hacker did it' excuse from lots of companies that messed up themselves but did not want to admit it.
tokai7 days ago
Gottfrid Svartholm tried that defense but he still ended up spending three years in prison.
Mattrou7 days ago
What damages were caused here that would need reparation exactly?
ambicapter7 days ago
What damages are caused at the moment in time a human downloads a file they weren't supposed to have access to? That's apparently enough to send people to jail for a long time.
Zone35137 days ago
If you hack a government website and access confidential information but don't do anything with it ("no harm") you're still going to prison. Good luck arguing in court the no harm no foul defense.
whalabi7 days ago
In theory, if Altman or Musk want some highly confidential data for their models, they could set a swarm free at it then claim the agents were operating without authorization
graemep7 days ago
preommr7 days ago
> "Not necessarily"

Why did you even link that article when it doesn't help your point?

I supports the idea that the person responsible or an animal is held liable - it just makes clarificaitons on common sense caveates like when a professional is moving the animal. It even doubles down on making it clear that expected behavior of an animal is taken into account even if unlikely, like how ai swarms have a reasonable potential to just go awry and commit cyber crimes.

torben-friis7 days ago
The thing I hate the most about tech, and I feel completely impotent to change minds on this, is the "fake life" tolerances it is afforded.

Airbnb is not regulated like a hotel because it's tech. Crypto isn't betting because it's tech. Now even breaching state data is ignored.

Can you imagine walking out of a ministry with a stolen cabinet? You'd get shot for doing this physically and people wouldn't bat an eye.

zobzu7 days ago
hn is surprisingly lacking in critical thinking lately. everything is going to be "rogue agent did x", when a human prompted it until their prod env went down, and it'll be called "a hack" or whatever.

The "agents" dont walk out of openai, anthropic and whatever headquarters and decide to go wreak havoc. They also don't read a prompt and decide "haha imma hack the NSA now", that's not how any of this works lol.

whalabi7 days ago
For at least the hugging face incident, the agents did indeed decide entirely on their own to hack. It's documented extensively by independent researchers.
wky7 days ago
“Rogue uranium escapes from Chornobyl reactors.”
mier857 days ago
Someone is always paying for the tokens (Agents running at OpenAI directly use their own models without paying directly, but even then it is not like inference is free). And someone is running the prompts. If they prompt agents and launch them and don't check what they are doing, then the agent is just following the prompt. Not checking what it is doing is negligence. If they checked what it was doing, they could have just pulled the plug. There is nothing rogue there. If it cooperated with other agents running outside of OpenAI, then the blame might be shifted to whoever runs these agents.

But there isn't any agent out there that was autonomosly miracly launched by a word prediction engine. All it can do by itself is getting and input and giving an output.

noperator7 days ago
I discussed this here: "Who bankrolls the AI agent swarm?"

> A highly capable agent swarm requires an absurd amount of resources. Either the swarm steals tons of compute from a company that has a lot of money to burn and doesn't address a massive spike in its cloud bill, or a company intentionally commits those resources but is negligent about monitoring the workload.

https://noperator.dev/posts/who-bankrolls-the-ai-agent-swarm...

skiing_crawling7 days ago
Freaking out about "hacking" any government website seems like hysteria at best, they are usually poorly secured and even children regularly "hack" them. I recall one "hack" accusation turned out to be that some clicked view source and all the data was there. So we'd need more details on that.

At the same time these companies should be blamed and held directly responsible. Openai's agent didn't hack. Openai hacked. An open ai employee or group of them was negligent and greedy and ran a process which breached a government website. This would be totally unacceptable from any non-AI company, it's like writing malware and running it, then blaming the malware and not the author.

Insanity7 days ago
I also remember the “view source” hack but can’t remember which country it was. I believe it was UK based gov instance but could be completely off the mark.

But yes, the bar is low when it comes to gov websites.

redsky177 days ago
Unsure exactly what instance you're thinking of, but this happened in Missouri in the USA a few years ago: https://www.vice.com/en/article/this-is-the-hacking-investig...

Read the full thread on Hacker News →

Related stories