Often I see an argument that there can’t possibly be a treaty to ban artificial superintelligence (ASI) that’s something of the form:
While it would be globally better for ASI to be banned, any individual party to the treaty would benefit greatly from developing ASI in secret. So nobody would actually sign this treaty, nor would such a treaty be stable if implemented.
This is not so! ASI kills you.1 If you develop ASI in secret in contravention of a global treaty, ASI kills you. If the US and China sign an agreement to not make ASI, but then the crafty US makes ASI in secret where China can’t see it, while the idealistic China hobbles themselves by staying honorable, then everyone in the US gets killed by ASI.
“Global coordination to not build ASI would be good” primes one to think that this global coordination is an idealistic ‘if-everyone-would-just’ endstate that, on Earth, where there are still wars and still nuclear weapons, couldn’t end up happening. But, in fact, it’s good for someone who is party to the treaty to avoid building ASI so as to save their own skin, without even taking into account any effects of violating the treaty, nor their effects on the behaviour of other parties to the treaty, nor some kind of idealistic self-sacrificing altruism.
This is not all that common! Usually, anything that is happening in individual countries, for which the world would be improved if it was happening in no countries, is something that is good for those countries and bad for the world, such that countries would like to bind each other but wouldn’t want to be bound themselves.
Armies are expensive and a world that’s the same with no armies would be better, but if you are the only country with an army, you can invade all the other countries. Nuclear weapons are extremely dangerous - they’re one of a few things that could possibly take down civilisation - but if you are the only country with nuclear weapons, you can threaten all the other countries into doing what you want using your nuclear weapons. CFCs damage the ozone layer and a world with no CFCs would be better, but if you’re operating out of the only country that uses CFCs, your refrigerators are cheaper and more efficient and the ozone layer is fine.
And, so, while maybe all the countries of the world would want there to not be any nuclear weapons, no country with nuclear weapons would want to give theirs up. And if we did have such a treaty, then countries would develop nuclear weapons in secret, to their own advantage (like India, Pakistan, Israel and North Korea). It’s natural for one to think that proposals to ban ASI globally are something similarly idealistic, a solution to a commons problem. Yet, it is not, it just kills you.
Why, then, is anyone developing ASI, if it just kills them, and probably they don’t want to die? Mostly, I think they’re mistaken about whether it kills them. For whatever reason, they don’t think that it’s going to kill them, and something that doesn’t kill you isn’t so much an urgent priority to stop. The argument that it’s a bad idea to secretly make ASI in private rests on the fact that it kills you; if it didn’t kill you, it wouldn’t necessarily be a bad idea.
To some extent, it’s that a lot of the precursors to ASI are probably genuinely beneficial; GPT-5 is probably good and does make you a lot of money, and it’s the kind of thing one might create as an intermediate step on the way to ASI, such that a global ban on ASI would, as a side effect, make it harder to make things like GPT-5. In principle, all one needs to do to save the world is to not run the code for the ASI that kills everyone; but, in practice, cutting it that fine ends up with someone somewhere running ASI code that kills everyone, so a treaty that works to save the world needs to preclude doing things that lead to ASI short of that.
Then there’s the interaction: it is famously hard to get someone to believe something that their salary depends on them not believing.2 If GPT-5 is making you a lot of money, and believing that ASI would kill everyone would cause you to want to stop making GPT-5, then you might fail to believe that ASI would kill everyone, so that you want to keep making GPT-5, and therefore make money from GPT-53. (Doing this, however, also causes you to die from ASI. You don’t get to live and keep your money just by being mistaken about whether ASI will kill you. Rather, you make GPT-5, you get money from GPT-5, then you or someone else makes ASI by building on your research, and then you are killed by ASI).
Why, then, would we want a treaty at all, if it’s already to everyone’s benefit to avoid building ASI? Some of it is to get people onto the same page about how ASI is a threat and how it shouldn’t be built, rather than never having considered it (countries have limited attention and don’t always immediately pay attention to the most important things).
Some is to agree upon some of the valuable measures that are short of avoiding building ASI itself. Nations might still be unwilling to unilaterally avoid building the precursors to ASI, and give up having GPT-5, even in cases where two nations might agree to both give up having GPT-54. Depending on what specifically it takes for a treaty to work, there might need to be some idealism for these aspects but not the broader treaty.
Some is to ensure that there’s a legal pathway for preventing people who mistakenly think that building ASI will be to their benefit from building it anyway.
Some is that being killed by your own hand six months earlier isn’t extremely bad compared to being killed by someone in another country six months later (though still very bad!), so the advantage of personally not building ASI is higher if you know for a fact nobody in any other country will do it either.
Nonetheless, a correctly informed party to a treaty banning ASI would follow the treaty and save the world, and they wouldn’t even have to want to save the world, they’d just have to want to save their own skin. Many beautiful things we would like to globally coordinate on require idealism. This is not one of them.
There are short arguments for this, and then lengthier arguments that address the different counterarguments that each reader has. List of Lethalities is a decent existing list of the arguments.
This operates inside people - see OpenAI trying to reform itself out of being a nonprofit - but also between people. If there are ten founders, and nine of them think ASI kills them, and one of them doesn’t, then we will still see one ASI company with one founder getting paid for making non-ASI precursors to ASI.
Not consciously, but subconsciously someone can be influenced by their environment, especially regarding the kind of things that they get money and praise for.
This is a toy example; I don’t mean to say anything about whether banning GPT-5 specifically would be a good treaty provision.
I think banning it in the US in particular is idealistic. The money grubbing billionaire heads of companies will fight it, and every single controversial topic in the US gets politicized, meaning half of voters will fervently defend it.
This argument holds if you are 100% confident that a superintelligence would kill all of humanity. But if you have even a small amount of doubt in that proposition, then the argument falls apart.
Let's say, for the sake of argument, that the world has agreed to a treaty banning the creation of artificial superintelligence. And let's say that OpenBrain, one of the leading AI labs, has found a way to violate the treaty and develop ASI in secret. For simplicity's sake, let's say that OpenBrain is 100% sure that if they defect from the treaty, they will win the ASI race, but they're not sure whether they can safely align the ASI or not. Is it rational, from OpenBrain's perspective, to defect?
Let P denote P(doom | ASI) -- the estimated likelihood that ASI will kill all humans if it is created. Let V denote the total value of continued human existence in a world without ASI. And let Δ denote the differential value of OpenBrain winning the race to ASI -- meaning, how much extra value is added to the world (at least from OpenBrain's perspective) if they win the race instead of their nearest competitor. Then, the expected value of OpenBrain defecting would be: P(-V)+(1-P)Δ
This number could be negative. The closer P is to 100%, the more likely it will be negative. But it could also be positive. If, for instance, our P(doom | ASI) is 80%, we set V at 1 million utilons, and we set Δ at 1 billion utilons (since, for whatever reason, OpenBrain is really confident that the OpenBrain future is orders of magnitude better than whatever future would be created by a rival company), then our EV would be: 80% x (-1,000,000 utilons) + 20% x (+1,000,000,000 utilons) = +199,200,000 utilons.
Hence, from the perspective if any individual AI lab or any individual country, it may actually be rational to race to pursue superintelligence, so long as you put a non-trivial credence on there not being human extinction and a high differential value on us achieving ASI instead of our rivals.