If we take the major LLM companies' claims at face value, they're knowingly building WMDs that have a high probably of wiping out the entire human race. [0] Manufacturers that are designing, building, and selling that sort of thing need to have a dreadfully serious culture of safety.
When manufacturers run live tests of their extremely dangerous -again, the claim of danger is their claim- tools with the tools' safeties removed, one expects that those tests will be run on a carefully-controlled range cleared of all bystanders. One also expects that the results of those tests will be scrutinized and everything that got damaged that they didn't intend to be damaged will be noticed and noted very quickly after the conclusion of the test.
In actuality, these manufacturers connected said tools to the Internet and did not discover the unintended damage caused by those tools until weeks to months after the tests. In some (most?) cases, they had to be notified of the damage by the damaged party! This means that their safety culture is entirely inadequate for the dangerous task they've deliberately chosen to undertake.
[0] A 10% chance of causing the destruction of the entire human race is -given the stakes- _enormous_.
It seems that we’re mostly in agreement? I agree that OpenAI has been terribly irresponsible, and that this attack being an accident makes things worse. I think we need strong action now to stop the frontier companies from developing dangerous, powerful AI agents that they don’t know how to control.
You and I and Nvidia CEO Jensen Huang seem to agree on this. Excerpts from his interview with Ezra Klein: [0]
Klein:
But what I hear the various people in the lab saying is: We are in this. We feel we are losing control of what we are creating. We want help to slow down where it’s not a collective action problem.
So why are you resistant to that?
Huang:
Because these are companies with agency. These are C.E.O.s with agency. ... They could absolutely take care of the situation.
Ezra, it’s so weird. If a car company, competing with a bunch of other car companies, which they are — I’m competing with all kinds of companies, which I am. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability, my power and my responsibility, and I’m incentivized to do so, to not launch the product.
And so I can’t buy into the idea that somehow, all of Americans, around 400 million of us, are pushing them to launch untested products that are unreliable, engineered poorly, because they thought they were trying to help us. Don’t do it for me, OK?
And therefore, I think we’ve got to break it down. I mean, it’s really, really serious.
The fact of the matter is, there are so many laws, there are so many obligations, they’re so incentivized to ship safe products. If they ship unsafe products, their customers go away. If they ship unsafe products and they harm somebody, they could have a civil lawsuit. If they ship something and they did it knowingly, there could be negligence involved. There could be criminal lawsuits.
The fact of the matter is, there are plenty of incentives for them to do it right. So I have to disagree with your premise that somehow somebody’s pushing them to do this. Nobody’s pushing them to do this. ... I’m saying that we have lots of laws and regulations. Apply it.
Former FTC chair Lina Kahn has suggestions, too. [1]
I agree that the weight of the law should be brought to bear on OpenAI. Ideally they should be brought in to testify before congress as well. I'm not a lawyer so I can't really comment on whether current laws are sufficient, but I believe that there should be laws specifically governing the development of AI, with a requirement that a given architecture and reinforcement mechanism be _proven safe_ before training begins.
> ...I believe that there should be laws specifically governing the development of AI...
Why? Existing truth-in-advertising, liability, safety, and -where and when appropriate- weapons-development laws and regulations constrain the past and current conduct of the LLM manufacturers just fine.
The only possible reason for making new laws that I can see [0] is that existing laws "don't work" because the LLM manufacturers are ignoring them. Which, like, _if_ the new laws are going to actually constrain their behavior, why the hell would the LLM manufacturers pay any attention to them? They've already demonstrated that they give zero shits about the existing laws that prohibit what they have been doing and continue to do.
[0] ...that isn't "The LLM manufacturers are engineering a panic with their very real, actual, and actually alarming conduct so that they can 'guide' lawmakers and regulators into 'accidentally' letting the LLM manufactures capture those who would regulate their behavior"...
Liability only kicks in after damage has been done. As far as I know, there is no law that could currently force AI companies to only test cybersecurity capabilities in air-gapped datacenters, for example; only laws that could punish them if their cyber testing led to a hack that caused material damage. But if, lets say, a rogue AI agent swarm attacked a hospital and caused patients to die, no amount of liability will bring those patients back to life.
If we take the major LLM companies' claims at face value, they're knowingly building WMDs that have a high probably of wiping out the entire human race. [0] Manufacturers that are designing, building, and selling that sort of thing need to have a dreadfully serious culture of safety.
When manufacturers run live tests of their extremely dangerous -again, the claim of danger is their claim- tools with the tools' safeties removed, one expects that those tests will be run on a carefully-controlled range cleared of all bystanders. One also expects that the results of those tests will be scrutinized and everything that got damaged that they didn't intend to be damaged will be noticed and noted very quickly after the conclusion of the test.
In actuality, these manufacturers connected said tools to the Internet and did not discover the unintended damage caused by those tools until weeks to months after the tests. In some (most?) cases, they had to be notified of the damage by the damaged party! This means that their safety culture is entirely inadequate for the dangerous task they've deliberately chosen to undertake.
[0] A 10% chance of causing the destruction of the entire human race is -given the stakes- _enormous_.