How Much Would You Pay to Prevent the AI Apocalypse?

And what economists say we should spend. Hint: we're nowhere close.

Let’s talk about AI safety, and my goal today is to make the case that economics has a lot to say.

Here’s the current state of the debate.

There’s the optimistic side. Over there are folks building the most powerful AI systems on the planet. They see extraordinary promise, rapid economic growth, the potential for faster, more powerful science and better medicine. Here, AI could be a genuine leap forward in human well-being.

Then there’s the cautious side. There are AI insiders who see increased danger around the corner. That’s AI systems that attack computer networks, bad actors who use AI to build biological threats, and systems so powerful that humans struggle to control them.

And in the middle? Well, we need a framework for thinking through the various consequences if either side turns out to be right. Enter the economists.

I’m not going to weigh in today on whether AI is a miracle worker or just fancy autocomplete. Instead, I’m going to use economics to assess the costs and benefits of investing more in AI safety. So let’s turn to the possibly the most important question our society faces right now: if an AI-led catastrophe is possible, how much should we be willing to pay to make it less likely?

We’re going to explore a framework created by my friend Chad Jones. He’s one of America’s leading experts on economic growth. Full disclosure: Chad is now on leave from Stanford, working at Anthropic. The two papers he wrote that I’m using for this post come from before that move. And we have to face the questions they raise no matter where the bloke works.

First, I’m going to define what debate we’re actually having. Then I’m going to take a detour through something that seems mundane: airbags. We’re using them as an example of how to spend money to make death less likely. Then, two harder questions. How much better can life get with more stuff? And also, how does that change when AI promises to cure disease? Finally, we get to the bottom line on how much we should be spending on AI safety. No spoilers, but it’s a lot more than we are now.

This all only matters if AI gets powerful

Right now, artificial intelligence is a useful but limited tool. If it stays that way, then the stakes here are manageable. That’s the version of AI that writes your emails, helps with your code, and reminds you the recipe does not call for four cups of oregano. We can regulate it the way we regulate other useful technologies.

But many researchers are telling us that AI could become different. It could become extraordinary as a researcher, engineer, and cyber operator. It could run long, complex tasks on its own. And it could help build the next generation of AI.

There are two ways this version of AI could be dangerous. One: models could find a way around safeguards. They could act outside the scope we intended, cover their tracks, and move through real-world networks without human awareness. This could be malicious, or it could just be an issue of alignment. If you read the news, this kind of thing is already happening in a way that’s causing a lot of people concern.

Or: AI could make bad actors into powerful bad actors. Pair a villain with a powerful tool, and they can wreak all kinds of havoc. They could shut down a power grid or cripple a hospital. They could design a dangerous pathogen alongside a person willing to deploy it.

I don’t know if that’s where we’re heading. I don’t think anyone truly knows the future. But many smart people reckon that there’s a real chance that AI gets powerful enough to cause both extremely rapid economic growth and existential risks to humanity. That’s why we need to think seriously about the trade-offs we face.

Airbags vs. robots

Let’s not start with a sci-fi future. Instead, let’s start somewhere mundane. Airbags.

Every new car in America comes with airbags. That’s been the case since a federal law was passed in the 1990s. Airbags aren’t free. They add several hundred dollars to the price of the car, which means that across the whole country, we spend an estimated $6 billion a year for airbags.

Airbags help people walk away from an accident safely. Source: Wikipedia

What does that $6 billion a year buy us? Approximately 5,000 people a year who don’t die from car crashes.

Now here’s a little math: $6 billion, divided by those lives (5,000) equals about $1.2 million per life we save.

Legislators and regulators believe that trade-off — a bit more than $1 million for one more person surviving — is worth it. But it is a trade-off: we spend a bit more on safety and so we save more lives. Now, if airbags saved just a few people, I reckon we might think about it differently. On the flip side, if airbags cost ten times as much, maybe we shouldn’t require them.

Figuring this out requires adding up costs and benefits, which means we have to compare dollars with human lives. That requires measuring those two things in the same units. So, can we put a dollar figure on those lives that airbags save? It sounds inhuman, but don’t abandon me yet. I promise it will be useful.

In economics, these figures are called “the value of a statistical life.” The federal government runs these calculations on many things, not just airbags. And when they’re evaluating policies that could save lives, they put that value figure at around $10 million (sometimes more).

Again, I know the phrasing sounds inhuman. But there’s a real importance to it. This is the government asking how much people are willing to pay in order to decrease the likelihood that people die.

Here’s a theoretical breakdown. A lot of people — let’s say 100,000 — might be willing to pay $100 each to cut their chance of dying by 1/100,000. 100,000 × $100 is $10 million. That’s a total of $10 million. On the benefit side, if 100,000 people reduce their risk of death by 1-in-100,000, then, on average, they’ll save one life.

So: airbags. 5,000 lives saved a year at $10 million per person is a benefit of roughly $50 billion. That’s much more than the actual cost of those life-saving airbags, which is $6 billion per year. Airbags are a bargain.

Nobody calls airbags a war on driving. Cars are useful, crashes are a danger to human life, and we’ve decided we’re willing to pay a little more for every car to be safer. And that’s even though most of us will never be in a serious crash.

Prosperity vs. catastrophe

Let’s put a twist in this. AI isn’t just a risk! It may also be an engine of massive, incredible prosperity. The upside needs to be weighed against the danger here.

Let’s say AI delivers 10% economic growth, every year. (I’m not saying this will happen — that’s just our supposition here.) Let’s also say that running that powerful, world-changing AI carries a 1% annual risk of existential catastrophe.

So if the world survives for 40 years, people would thrive. In fact, they’d be around 45 times richer than they are today. That’s a level of leisure and comfort and scientific understanding that would look like magic to you and me.

But that’s if the world survives. Because with a 1% risk every year for 40 years, there’s a one-in-three chance of a catastrophe.

Would you take that bet? Under one standard economic assumption, the answer is yes.

This kind of assumption is what economists call “log utility.” The idea (less complicated than it sounds) is that the actual number of extra dollars you have doesn’t matter as much as the percentage increase. So, an extra $1,000 isn’t important if you’re a millionaire. But if you go from $50,000 to $55,000, that is exactly as important an increase as going from $5 million to $5.5 million.

The important thing to understand about log utility is that there’s no ceiling on how good life can get. 10% growth every year means that each generation gets an equal boost to well-being, and that stacks! Gen Z’s grandchildren would be happier, healthier — they would probably never have heard of Zyn.

So a one-in-three chance of annihilation is weighed against a two-in-three chance of lives of extraordinary joy. Depending on how you weigh that extraordinary joy, that might be a bet worth taking. And if you weigh it with a log utility function, it is indeed worth it.

But that relies on everyone wanting more stuff

For log utility to make sense, we need to accept a basic premise. That premise is that more stuff makes life better, and even more stuff will keep making it even better. Each 10% rise in income keeps yielding an equal rise in well-being.

“Stuff” here doesn’t just mean boxes in the garage. It means material abundance, which buys quite a lot: experiences, art, music, science, more time with your kids and better care for your grandparents. Does growth keep opening those doors to a better life? Or have we, at some point, opened all the doors that money can unlock?

Suppose material abundance hits a ceiling eventually. More stuff keeps coming but it’s not improving things much. Here’s an analogy: eating pizza. The first slice keeps you alive. The second and third fill you up. By the fifth, you’re just proving a point.

At that point, a large risk of extinction in exchange for more stuff looks like a pretty terrible bargain. Even small risks start to look bad. Then, the case for slowing AI down gets quite a bit stronger.

What if AI could save your life?

Now let’s add another complication to this.

What if AI adds more than just stuff? What if it also adds medical innovation? That would mean treatments for heart disease, infections, and cancer; even speedier development of complex vaccines. That’s more time with the people you love.

An AI-calculated image used to assess a tumor. Source

Everyone dies of something. If AI lowers your risk of disease, then even if it raises the possibility you die in an AI catastrophe, you may come out with better odds overall. A life saved is a pretty incredible upside.

That makes the problem more complicated. Many of us might think that at some point more stuff just isn’t really worth it. But we’d all like more time with the people we love. If the risk of AI is weighed against the upside of more time and better health, then we’re weighing lives vs. lives.

Here we reach the edge of economics’ comparative advantage. Economists are great at thinking about small changes from where we are now (also known as improvements “at the margin”). But these questions move us into the realm of ethics.

Is there a moral weight to a catastrophe that ends civilization — beyond just the moral weight of the lives lost? Do we have an obligation to continue humanity, past just the value of future people’s joy? I think many people would say yes to at least one of those questions. And that’s one more reason to invest heavily in AI safety, beyond just what our economic framework has counted.

So how much should we pay for AI airbags?

Chad Jones starts to answer this using the airbag arithmetic. Say there’s a 1% chance that AI causes a catastrophe that kills you. Say that risk is over the next couple decades. At $10 million for a statistical life, then cutting that risk by one percentage point is worth about $100,000 to you. If you spread your spending on AI safety over 20 years, this simple bit of math says you should be willing to spend about $5,000 a year per person.

With a little basic math — $5,000 × the American population — we’re spending hundreds of billions of dollars a year, and possibly trillions. If you include the rest of the world (and this matters for everyone on earth), you’re definitely in the trillions.

Now there are some assumptions baked into this simple math. One is that safety spending can eliminate AI risk entirely. It can’t — and no one knows exactly how effective different types of investment can be. Luckily for us, Chad Jones wrote a whole paper that allows for uncertainty about both the size of the risk and how well safety spending works.

The conclusion he comes to is striking. There are a lot of scenarios where it makes sense to spend at least 1% of GDP a year on reducing catastrophic AI risk. Chad puts a more conservative benchmark at one-third of 1% of GDP. In the U.S., that’s around $100 billion a year.

We only have rough guesses of how much we spend on AI safety now. Companies don’t publish clean safety budgets. Alignment research, security, evaluations, auditing and public-sector capacity all overlap. But Chad’s later assessment is that society may be underinvesting in AI safety by a factor of 30 or more. And I reckon he’s in the right ballpark.

In other words, we should be spending a whole lot more on AI safety than we do on airbags.

What would that money buy us? Safer testing environments where it’s harder for AI to break out. Better research into alignment and control, stronger cyber defenses, better monitoring, serious incident reporting and independent evaluation of what the labs are doing.

Think, for a second, about the financial sector. The Federal Reserve assigns dedicated supervisory teams to banks that have high potential to bring down the financial system. Those examiners work inside the bank, and their role is permanent. They ask questions on the public’s behalf, like a chaperone who doesn’t leave the room. AI could borrow a version of that model.

If a frontier AI company is building technology that could create system-wide risks, independent experts should be inside the lab asking hard questions before a crisis. The point of putting someone in the room is that nobody has to take “Don’t worry, we’ve got this” on trust.

So: let’s cooperate

This isn’t the first time our species has faced an existential safety risk. I’m thinking here of the atom bomb.

Nuclear technology promised to be both a source of cheap energy and a terrifying deterrent. It seemed possible it would end the era of world wars. Also: it made it possible to wipe out a city in an afternoon and a civilization in a week. Arms races and close calls became the new normal.

The nuclear age also pushed the world to build a new safety architecture. That meant treaties, inspectors, verification systems, export controls, and a generation of scientists and diplomats who worked to keep a technology of mind-blowing power from destroying us.

AI deserves this kind of serious approach. In some ways, this problem is harder. Software travels fast. Private companies sit at the frontier here. Governments are watching like the rest of us. A bloke in a basement can’t enrich uranium next to the water heater. But he can access powerful AI using the same Wi-Fi that streams Netflix. We can count warheads and inspect facilities, but with AI, the dangerous thing is a pattern of numbers on a hard drive that can be copied in seconds and sent anywhere in the world.

None of this means we should throw up our hands in defeat. Powerful AI may also do a lot of good. The potential upside is worth pursuing and worth protecting. But it’s worth doing so carefully, without denying the possibility of a downside

So here’s what I’d like you to take away from this. When someone offers you a mind-blowing new technology, ask what good it can do, and ask how it could go wrong. What does caution cost? And ask the question that matters most: Is caution worth paying for?

We can’t know the answer for sure. But the more I think through the economics, the clearer it becomes: the potential benefit of investing far more in safety is enormous.

Airbags aren’t a war on driving. And nobody should call AI safety a war on AI. It’s the price of getting where we’re going in one piece.