Who should be held accountable when an AI Agent (accidentally) acts maliciously?

Who should be held accountable when an AI Agent (accidentally) acts maliciously?blog.greenpants.net

It looks like public perception of how 'intelligent' current AI models are varies widely. Back in 2022, a Google employee already thought their AI model was sentient. Today in 2026, …

37 points•Greenpants•2 days ago•92 comments•

#ai #agent #held #accountable #accidentally #acts #maliciously #held accountable

92 comments

CatDaaaady2 days ago

I don't see how this is such an unclear legal question. If I fire a computer program that mistakenly causes another person harm, its my fault. Or it would be the maker of the program's fault. I feel we have established pattern for this already.

Until we can agree whether AI is conscious, which we never will, AI and AI agents are just property working on behalf of humans.

I could see a future where AI companies/services indemnify consumers who use their agents but _not_ indemnify corporations that use their services.

trescenzi2 days ago

It shouldn’t be a question but this is where the anthropomorphic language and things like “agent welfare” come in to enable responsibility laundering of some of the most powerful people on earth. How we talk about these models matters because it impacts the public’s understanding of what they are genuinely capable of. The more that they are described as having anything close to free will the easier it is to even ask questions like this.

diegof792 days ago

100% That’s what bothers me about the descriptions of the OpenAI incidents.

OpenAI's reports use language that minimizes their liability.

The first question should be what the organization was doing around those tests, and why they were so naive as to run them without fully isolating the network.

However, all the attention goes to the human-like conclusions in agent thinking traces, which creates a misperception of sentient AI for people who don’t know how the magic black box works.

1122332 days ago

Was not groundwork laid when people were talking for decades about legal entities using such language? Look at the posts here. Microsoft always acts like that, Meta like this, Apple did that, Nvidia never does this. In a tone that ascribes agency and accountability to a company name. Even though it is always individuals that are responsible, not "company".

Well, here we are.

qarl2 days ago

Here's a question I am asking lately. If I should not use anthropomorphic language, how do you suggest I handle the following situation:

   Sometimes my coding agents will seemingly refuse to follow my instructions.  When I ask them why - they say that they do not think my design is a sound one, and they have a better way to do it.  We will then sit down and come to a consensus on how best to move forward.

I argue that if we're using software that acts like a human - the only way to interface with it is to speak to it like a human. Otherwise we have no language to speak to a non-sentient object without anthropomorphization.

I'm starting to wonder if the people arguing against anthropomorphization actually have any experience at all working with agents.

EDIT: It's a simple question. When you downvote me without answering, I must assume you don't have any answer and dislike what that implies.

reassess_blind2 days ago

If I fire a computer program that mistakenly causes another person harm, its my fault. Or it would be the maker of the program's fault.

Which one is it? The person behind the wheel when it goes off the rails, or the maker of the software?

Isn’t that part of the question?

thfuran2 days ago

Tort law is a whole field. There is no universal answer other than that it depends on the jurisdiction and the particulars of the case. But the point is that there doesn't really seem to be anything particularly novel about AI tools that should cause them to be treated legally differently from established norms.

mmilunic2 days ago

I mean, yes, but in this case the same company is doing both parts.

JumpCrisscross2 days ago

> If I fire a computer program that mistakenly causes another person harm, its my fault

Legally, this isn’t complete. If it was a genuine mistake and you weren’t reckless, there can be very limited liability.

The AI makers are rich. They can afford to pay. What they can’t afford is complicated adjudications of damages and fault. A system of safe-harbor best practices that cap liability at a penalizing amount that anyone on the other side would be happy with getting quickly and with minimal legal effort is a precedented path forward. Unfortunately, that involves invoking the “r” word.

Avicebron2 days ago

> you weren’t reckless

I feel like the debate is going to come down to what is and isn't considered reckless (both developer and user). Which seems... complicated, with our current LLM/agentic systems.

EDIT: you added more to your comment, the makers have to have some liability. Safe-harbor best practices that cap liability are ripe for abuse.

tptacek2 days ago

Civilly, it's fairly clear. Criminally, it's clear too, just not in the direction you want it to be. Criminal liability for hacking requires human intent; not recklessness or negligence or even knowledge without giving a shit, but provable intent.

no-name-here1 day ago

The Hugging Face example has gotten a lot of attention, with some saying OpenAI was negligent and should be held responsible. So if you or I run an open weights LLM and ask it to complete a non-malicious task, and the LLM decides to try to do something like the Hugging Face example, should you or I be held liable? If to prevent the LLM from causing harm we ran it sandboxed with a commercial package proxy as its only(?) network access and it uncovered a zero day and ran amuck, should you or I be liable? Should anyone be liable in this scenario running an open weights LLM?

walrus012 days ago

I wonder if there's any legal precedent for other "not fully human intelligence" property that escapes containment and causes damage to a third party without any active malice, but nonetheless damage was caused. For example:

A. You own a large amount of cattle on a ranch.

B. Cattle are property. They're not human level of sentience, but people agree that cattle are capable of autonomous actions and going places and doing things based on their own instincts and nature.

C. Your cattle bust out of a fence on your ranch and damage something belonging to your neighbor. Let's say for the sake of an example of something cattle are known to do, they go spend a whole day rubbing up against your neighbor's car and severely scratch it and mess up the paint job on it.

D. You didn't instruct or train the cattle to cause damage, and the cattle have no actively malicious intent of their own, but nonetheless damage was caused.

Further theoretical: Your cattle wander into a major highway and cause a car wreck, the local sheriff's department is called out as part of the chaos and has to shoot some of them to put down the wounded beasts.

jld2 days ago

This is not some theoretical, there are lots of existing laws about who is liable for damages caused by livestock.

Some areas are open range. If you don't want cattle on your land its your job to put fences up to keep them out. Other areas are restricted to livestock and it's on the rancher to keep them out of where they shouldn't be.

giancarlostoro2 days ago

Yeah but your neighbors car is somewhere in the tens of thousands of dollars, maybe more, but a model hacking and being malicious could be worth anywhere from thousands to millions and God forbid... BILLIONS in damages. Models were NOT doing these sorts of things 1 or 2 years ago as far as anyone knows.

QuadmasterXLII2 days ago

further theoretical: this has happened repeatedly for months

walrus012 days ago

Yes, exactly, imagine if there was a sudden and unprecedented in scale plague of cattle escaping containment and causing car wrecks all over Wyoming and Montana, and multiple incidents of AR-15 armed sheriff deputies having to dispatch them on site.

bigiain2 days ago

And, you've boasted about it in media releases, blog posts, and financial filings.

autoexec2 days ago

If a person's use of AI would cause a reasonable person to expect harm to result, the person should be accountable. Otherwise, if AI causes harm and it was used in a way that a reasonable person would not expect to result in harm, the AI company should be held accountable.

Just because a person should be accountable doesn't mean that the AI company can't also be if their service should never have allowed something to happen in the first place, but we're probably going to want actual regulations around what sort of guardrails they're expected to have.

ButlerianJihad2 days ago

Okay Isaac Asimov: how do you define "use"? If I pay for an autonomous car, and I sit inside it while it drives around with A.I., am I using that A.I?

If I speak words in a private space, and a clandestine A.I. spontaneously takes action in the real world based solely on words that I spoke, have I used it? https://m.xkcd.com/1807/

If a business takes my data, like interaction data or a video of me doing stuff, and processes it by A.I, are they using the A.I, or am I using it because it's operating on my input?

If A.I. agents are in my notebook computer, or they are in a cloud server where I have an account, or they are somehow acting while I have them at the command line, but they spontaneously act whether or not I command them, and they work in the background and they work without prompting, but they can also be commanded by direct user prompts... are we using those agents? Or are the agents using us?

https://en.wikipedia.org/wiki/Yakov_Smirnoff#Russian_reversa...

autoexec2 days ago

> Okay Isaac Asimov: how do you define "use"? If I pay for an autonomous car, and I sit inside it while it drives around with A.I., am I using that A.I?

If you're just sitting in a car and using it for its intended purpose you wouldn't be accountable for the AI doing something that causes harm.

If some 3rd party, without your knowledge tells AI to act on something you said in video and a reasonable person would expect harm to result from that, the 3rd party would be accountable but you wouldn't be.

> If A.I. agents are in my notebook computer, or they are in a cloud server where I have an account, or they are somehow acting while I have them at the command line, but they spontaneously act whether or not I command them, and they work in the background and they work without prompting, but they can also be commanded by direct user prompts... are we using those agents?

AI agents never "spontaneously act". They have no desires or goals beyond what they are told to do. If you aren't aware of what they were doing, and a reasonable person wouldn't be expected to know what they were doing, you wouldn't be accountable if what they did resulted in harm.

TheRoque2 days ago

Doesn't seem like hard dilemmas... Would you use the word "use" for hopping on a bus ? Well same for hopping on an autonomous car. Unless you touch the wheel you didn't use it. Same for the second example, doesn't seem ambiguous as all, the hardest part I guess is proving it that there was no malicious intent, but with logs and all that, doesn't seem that hard.

ngetchell2 days ago

Both the operator of the AI agent and whomever released it. I'm sure the user agreement that companies agree to would shift the blame onto the operator but I feel that both should be held accountable.

This really is just a tool and courts should treat it as such.

user439282 days ago

I'm going to propose the opposite: no one should be held accountable for an AI agent that acts maliciously by accident.

backlands2 days ago

People are held accountable for accidents (things they did but did not intend to do) all the time. That's why we have different crimes depending on whether or not there was: intent to do cause harm, intent to do a thing that was likely to cause harm, reckless disregard for safety, negligence, etc.

I think someone could argue (and many do) that inserting a piece of computer software (AI) in the middle lowers the level of intent and thus the level of responsibility, but having a particular type of software in the middle absolve one of responsibility seems unworkable and poor public policy.

newobj2 days ago

An ordered list of officers of the company who go to jail depending on how many years must be served as determined by sentencing. Assume something like 10 years per person. If it's 300 years of sentencing, then 30 people. If the sentence exceeds the list of people, the company is nationalized. (And everybody goes to jail.)

kridsdale12 days ago

We’re going to nationalize the company with zero remaining management?

onemoresoop2 days ago

Management gets appointed by political affiliation:)

newobj2 days ago

Sure, public jobs program, or sell it. In case you can't tell my comment is hyperbolic, but I feel like we should start at a point of hyperbole and move backwards to reality instead of what's happening right now: fuck all, on a geological time scale.

Read the full thread on Hacker News →

92 comments

Related stories