You don’t even need to go all the way to “air gap”

What has been described is far far below the standards for running untrusted 3rd party code. If they were actually as afraid as they claim to be they would have a sandbox at least half as good as ec2

We should not be creating/running models that would unilaterally choose to hack into Hugging Face.

Yes, we should also have excellent sandboxes. But we need defense in depth. So if/when there are flaws in the sandbox, the models don't unilaterally hack into third parties. This is especially important in light of the models of the future being more capable than the models of today.

And real world use of these models involves them having access to the internet, libraries, etc. So we can expect their evaluations to continue granting them some amount of internet access.

As for your theory about their motives - these companies make money by charging high margins for frontier models. If regulations slow their development such that their cheaper, less capable competitors catch up, I would switch to their competition.

While o don’t this it’s a threat in training, it should be stated that air gaps have been bridged before. Example, stuxnet

the problem is these things are meant to eventually be run everywhere by everybody, so what good does air-gapping do? If they air-gapped the model but still logged it trying to do some craziness - that makes the test safer but not the model.

I mean technically I don’t think it is airgapped. The DoD didn’t run their own cables. They run encryption devices and run their own network on top of the existing infrastructure.

More than one thing can be true. OpenAI was absolutely negligent, but this was only able to happen because the models were capable and persistent, and had a tendency to go far beyond any reasonable boundaries. And, importantly, OpenAI's level of negligence here is pretty common. It's not hard to imagine what could happen if similarly capable and inclined models were generally available, and someone yolo'd them into a swarm to complete some other difficult-to-impossible task.

No, you are forgetting the second incident where a more capable model swarm later discovered the message board and took control over the entire research cluster at OpenAI.

From the technical report:

"The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod… Agents take over active evaluation infrastructure… Agents now control the challenge evaluation endpoints that other agents are connecting to."

https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute.

Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.

As of writing it still says:

> For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor.

If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.

The failure mode was _not_ identical. The HF incident agents were not directly connected to the internet and had to compromise an internal package registry in order to access the internet.

thank you for this reasonable reaction

what you read (past tense) is already the corection

The same thing is happening with Laya, people didn't seem to click through to evaluate the supposed paper

tyranny of confirmational headlines

I guess the AI is getting good in some ways. Still a bit lacking in others.

Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1].

[1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... - page 9

in 2024 they could not break into unsecured all-my-passwords-and-acces-keys.txt on my Desktop

> Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense?

No, and I don't see anyone who's actually demonstrated understanding of what happened in the Hugging Face incident (e.g. reading the reports in their entirety) making this claim.

> Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation?

Yes. Again, this is hard to deny if you've actually read the reports.