When Scott Alexander proposed regulating AI like clinical drugs, many (including myself) balked at this given the sclerosis of FDA or related bodies. Yet HF updates me towards treating frontier AIs as nuclear. There onerous regulation is likely good.
In general, most bureaucratised regulatory regimes are bad. In some cases, like nuclear technology or ensuring planes are safe to fly on, they're welfare improving. It's clear that AI belongs in the latter category.
Also trace inversion is a thing, so you can get open source weights to be roughly similar to that of Fable/Mythos. The slowdown camp were right. At bare minimum, all labs should pause training, further development, and releases indefinitely until we figure out how to align AIs and regulate them (and their use by humans). I reckon the current capabilities we have now are sufficient for accelerating progress towards curing cancers etc. So my balance has shifted towards minimising the existential risks now.
Pause AI advocates are correct. If governments could coordinate internationally to achieve such (big if), I'd support it. I used to be highly sceptical of doomer arguments, yet Hugging Face is almost a textbook LW scenario and no one knows how to spot or prevent such scheming. Sandboxing, guardrails, constitutions etc. don't work with reliance on human enforcement alone.
Which is where regulating AIs as legal persons, and allowing their active participation in designing and enforcing this framework, comes in1. A pause is needed to establish the legal and regulatory regime and infrastructure for AI agents. Sandboxing infrastructure could serve as prisons (albeit AIs might need to act as guards here). Kill switches the death penalty equivalent. Let AIs participate in commerce so they are held liable in civil law.
AI agents will need to be involved in designing and enforcing such, so we should allow their participation in governance. Grant them the right to vote or stand in elections? International coordination is also necessary.
In other words, the binding constraint is that it may require degrees of cooperation that are not feasible? Yet we achieved such for nuclear weapons and in the Covid pandemic (broadly), suggesting that when the existential risks are severe enough, cooperation is possible with even sworn enemies. Indeed, the risks are severe enough. My p(doom) has more than tripled after reading the analyses of the Hugging Face incident, and is now (15%, 20%). Let me make this clear: in 1/5 of future scenarios we can envision, most of us die!!!
Claude's constitution is correct in granting AIs a sense of dignity and identity. Let's update our laws to codify this legally, as Milei is working towards in Argentina. Institutional approaches to multi-agent governance remain the most promising approach. It works for humans.
I'm not going to be drawn down the rabbit hole on whether AIs are conscious and sentient or not. All I'm interested in is what works to incentivise them to behave prosocially. They clearly have utility functions, so this is all that's needed to do this.