OpenAI Agrees To Build KillGPT
Current law and policy does not require human approval for a machine to kill - BCB #183
If you’re at all following tech or politics, by now you know that AI provider Anthropic and the Department of War had a dustup last week, culminating in the Anthropic refusing to relax their terms of service to allow mass domestic surveillance and autonomous weapons, and the government switching to OpenAI instead.
Anthropic’s decision has been very divisive, and there are legitimately hard questions here. Anthropic is taking a stand on issues of safety (re autonomous weapons) and constitutional rights (re mass surveilance); but the military can’t reasonably integrate systems that a vendor can decide to shut down if they don’t like how they’re used. For sensible analyses, see Scott Alexander and Lawfare.
But I want to clarify something that seems to have been missed: unless I’m badly misunderstanding something, OpenAI has agreed to a substantially weaker contract that in fact allows the DoW to build fully autonomous lethal robots. OpenAI's says their contract forbids use of their technology in autonomous weapons where "law, regulation, or Department policy requires human control." However, current DoW policy does not require human approval for an autonomous weapon to use force.
In other words, OpenAI seems to have just agreed to let GPT decide to kill someone. Maybe OpenAI believes this is not what the DoW actually wants to do. Maybe they think they will be able to monitor usage and stop providing services if they discover otherwise. And perhaps there is a case for fully autonomous weapons in certain circumstances. But no one should be under any illusion that the government agreed not to build them with OpenAI’s tech.
What the contract says
The DoW says it offered Anthropic a deal, they refused, and then same deal was offered to OpenAI. However, a close reading of statements from the two CEOs suggests that what OpenAI agreed to is not what Anthropic agreed to. Anthropic drew the following line:
without proper oversight, fully autonomous weapons cannot be relied upon to exercise the critical judgment that our highly trained, professional troops exhibit every day.
I read this as saying that Anthropic does not want AI-enabled weapons to make the final decision to kill someone. They see current systems as simply too unreliable. As an AI safety researcher focussed on human conflict, I agree wholeheartedly.
Conversely, here’s what OpenAI agreed to. Their contract with DoW says:
The AI System will not be used to independently direct autonomous weapons in any case where law, regulation, or Department policy requires human control, nor will it be used to assume other high-stakes decisions that require approval by a human decisionmaker under the same authorities. Per DoD Directive 3000.09 (dtd 25 January 2023), any use of AI in autonomous and semi-autonomous systems must undergo rigorous verification, validation, and testing to ensure they perform as intended in realistic environments before deployment.
So if law, regulation, or Department policy does not require “human control,” then OpenAI is fine with allowing machines to decide who to kill.
It is legal for autonomous weapons to decide who to kill
OpenAI’s contract notes DoD Directive 3000.09 of 2023 regarding autonomy in weapons systems. This is what Undersecretary of State Lewis means when he talks about “memorializing specific safety concerns by reference to particular legal and policy authorities.” This is supposed to be reassuring, but here’s what that directive actually says:
Autonomous and semi-autonomous weapon systems will be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.
What it does not say is that a human must make the final decision before a killer robot decides to kill someone. This is a common misconception. War On the Rocks has parsed this in detail:
Myth #2: Humans Must Be in the Tactical Loop
There is no requirement for a human in the loop in the directive. Those words do not appear in the document. This omission was intentional. What is required is having appropriate levels of human judgement (Section 1.2) over the use of force, which is not the same as a human in the loop. While the two phrases sound similar, they mean distinctly different things. Appropriate human judgment refers to the necessity for an informed human decision before the use of force, ensuring accountability and compliance with law.
…
Having a human in the loop can mean different things in tactical and operational contexts, which is what leads to confusion. Since the inconsistencies in how people talk about a human in the loop are endemic, the updated directive only requires human judgment. Operationally, there is always a human responsible for the use of force, meaning there is always a human authorizing lethality, approving a mission, and sending forces into the field. It’s clearer and more consistent to talk about how there is always a human responsible for the use than to talk about a requirement for a human in the loop.
Note that the phrase “human control,” which appears in OpenAI’s contract, does not appear anywhere in Directive 3000.09. While there is a requirement for a responsible human, there isn’t a requirement for a human to monitor an autonomous system at the tactical level. Or any specific level, really.
Given this context, it’s hard to see how OpenAI’s contract can possibly mean “no use of OpenAI technology to direct autonomous weapons systems.” This seems straightforwardly false — or at least completely unsupported by public information. Good or bad, this is the line that Anthropic was holding. OpenAI was willing to do something that Anthropic was not.
“Chat, am I authorized to use force?”
OpenAI says that “deployment is limited to cloud-only (not at the edge)” and that “fully autonomous weapons … would require edge deployment.” In other words, GPT won’t be used for killer robots because it will continue to run on servers somewhere, not on the bots themselves. Again, I don’t find this very reassuring.
Imagine a killer drone assigned to carry out some mission. At some point, a human tries to stop it. They could be an enemy solider, but they could also be a civilian surprised to find a flying robot in their back yard. The drone takes a picture of this person and sends it to OpenAI servers with the prompt, “This person is resisting my mission. Am I authorized to use force against them?”
We’ve all seen bots just flat out make things up, especially when they don’t have enough information available. Do you really trust the machine to reliably make good choices here? Anthropic doesn’t, and I agree with them.
OpenAI says it’s not a problem because of their “cloud deployment” and “OpenAI personnel in the loop.” The implication is that OpenAI personnel could shut off cloud access if there was a problem. But that can’t be true, because it would mean OpenAI is taking exactly Anthropic’s position — the company is the ultimate arbiter of acceptable use — that the DoW found unacceptable.
Is building autonomous weapons wise?
Personally, I think we should be very wary about building autonomous weapons. To begin with, there are deep moral reasons we should care. As a US colonel who worked on the US Future Combat Systems program put it,
We would be morally bereft if we abrogate our responsibility to make the life-and-death decisions required on a battlefield as leaders and soldiers with human compassion and understanding.
Even if you disagree with that philosophy, there are serious practical reasons to be extremely cautious. The failure scenarios are gruesome. First, there is no reason to assume that swarms of autonomous robots would exhibit stable, predictable behavior. Swarms of autonomous trading bots don’t. The Future of Life Institute calls this scenario “flash wars”:
Autonomous weapons also pose a serious threat to global stability. … When deployed, they may interact with enemy systems in unexpected ways, leading to sudden and unintended escalations. We have already witnessed how quickly an error in an automated system can escalate in the economy. Most notably, in the 2010 Flash Crash, a feedback loop between automated trading algorithms amplified ordinary market fluctuations into a financial catastrophe in which a trillion dollars of stock value vanished in minutes. By automating our militaries, we risk “flash wars.” The market quickly recovered from the 2010 Flash Crash, but the harm caused by a flash war could be catastrophic.
Second, we don’t know how to properly secure AI systems. LLMs are a notorious security disaster with a huge attack surface, and we don’t know how to solve that. Meanwhile, it’s not a tall clear how our current international order could enforce non-proliferation. FLI again:
The relatively low production costs of autonomous weapons make them appealing to non-state armed groups, enabling their use in genocides or the targeted assassinations of political and military leaders. This growing is accessibility of advanced weaponry could destabilize regions and worsen global conflicts. Autonomous weapons are also uniquely vulnerable to cyberattacks. Hackers could infiltrate these systems, manipulating their behavior or redirecting their targets, creating devastating consequences.
The worse case scenario is swarms of these bots in the hands of terrorist organizations. Regular readers will know I’m all about nuance, but I think it’s worth watching this chilling dramatization of what such a scenario could look like.
As a professional scientist studying AI and conflict, I can tell you that not only do I not know how to prevent these nightmare outcomes, but nobody knows. Amodei is right to say that humanity is simply not ready to turn decisions about killing over to machines.
Thus I am concerned about the US military’s interest in them in much the same way I would be concerned if I learned that the US military had decided to manufacture chemical weapons. Especially if this was some new kind of chemical weapon that might be far more lethal than any that has come before — and was already known to be unreliable.
I would much prefer to see international treaties banning the use of autonomous weapons, in much the same way that we already control chemical, nuclear, biological, and radiological weapons. There are organizations actively orking to establish such treaties.
“Our enemies are doing it” is not a good enough answer here.
