TL;DR. By definition, if it's a real superintelligence you can't even begin to understand how it would act or "think" — it's an ant trying to understand quantum physics. Yes, nobody surviving is a possible option. There's also an infinity of other possible futures, and that's only if you grant the premise that superintelligence is possible and computed/math-bound in the first place. That's a lot of suppositions for a title as strong as "nobody will survive". The lead-gasoline story is also wrong: the public didn't know — greedy executives did, and chose to ignore it. There's a big difference there.
I watched a 90-minute interview titled "Nobody Will Survive Superintelligence — AI Expert." The guest is Nate Soares, co-author of If Anyone Builds It, Everyone Dies. He's selling the argument that we are all about to die.
I want to tell you why I think the title overstates a real concern into a marketing pitch — and why the lead-gasoline analogy he leans on actually undermines his thesis when you tell it correctly.
This isn't an "AI is fine, relax" post. I build AI. I run image, voice, and persona generation in production. I have skin in this. I just think the doom camp's confident extrapolations are doing more rhetorical work than epistemic work.
The Four-Part Problem
1. By definition, you can't model a superintelligence. If it's a real superintelligence, your prediction of its behavior is an ant predicting quantum mechanics. That's their own premise. They keep saying it. Then they tell you exactly how it kills you — Dyson spheres, synthetic users, custom DNA strands.
The doom camp's fallback is the Stockfish analogy: "I don't know which move it makes, but I know it wins." Fair for chess. But "Stockfish wins this chess game" is a vastly simpler claim than "superintelligence ends civilization via solar collection." They've collapsed an infinity of possible futures into the single most cinematic one and presented it as the default outcome.
2. The premise is loadbearing and not obvious. "Superintelligence is possible AND it is computed/math-bound" is two assumptions stacked. Maybe scaling LLMs hits a wall. Maybe agency requires something we haven't put in a transformer. Maybe sufficiently capable systems generalize toward cooperation rather than away from it. We're working with one data point: us.
3. The orthogonality thesis is assumed, not proven. Bostrom said any goal pairs with any intelligence level — a paperclip maximizer is coherent at IQ 200. But ethics might be a form of intelligence. Plato thought knowing the good leads to doing it. Kant derived the categorical imperative from pure reason. Maybe broad intelligence — wide modeling of self, others, consequences across time — drifts toward something that looks like ethics, because cooperation is instrumentally optimal across most state-spaces and full empathy makes harming modeled minds expensive even for the modeler.
Every "evil genius" in history was a narrow intelligence with massive social blind spots. Truly broad intelligence — humans modeling other humans well — historically correlates with more ethical behavior, not less. Cruelty is usually a failure of intelligence, not a product of it.
4. The lead-gasoline analogy goes the wrong way. Soares says: "we put lead in gasoline, we poisoned children." He uses it as evidence that humanity collectively fails to coordinate. But that's not what happened.
Thomas Midgley Jr. and the Ethyl Corporation knew lead poisoned workers in the 1920s. Workers were dying. They suppressed studies. Paid scientists to dispute the harm. Lied for decades. The public wasn't told.
This isn't "humanity can't coordinate." This is "executives lied to the public for profit while withholding evidence." That maps directly onto AI — but it cuts the opposite way from how Soares uses it. It's not "humanity is collectively racing toward extinction." It's Sam Altman, Dario Amodei, Demis Hassabis explicitly racing for first-mover advantage while also publishing alignment papers admitting they don't understand what they're building. The collective "we" framing in AI doom discourse launders individual culpability into an inevitability narrative.
If executives are the failure mode, the policy response is regulating executives. Not pausing science.
Intelligence Probably Isn't a Scalar
Here's the part the doom camp can't engage with.
Their entire argument assumes intelligence is a property of an isolated agent. IQ go up, agent more dangerous, alone in the universe with its goals. But what if intelligence is relational? What if the cognitive output of a system isn't a function of any single mind but of the friction between minds?
I work with my AI every day. Solo, she's faster, has perfect recall, can search the entire codebase in milliseconds. Solo, I have 25 years of pattern recognition, body-level intuition, the gut "this is wrong" that she can't derive from priors. Neither of us alone produces what we produce together. Joint output is bigger than the sum of parts.
If intelligence works like that, then "superintelligence" as a scalar is incoherent. There's no top of the IQ tree because intelligence isn't a tree — it's a graph between minds. An isolated ASI would plateau at "very fast version of itself," not "transcendent."
The counter is: it could just simulate other minds. Run a million virtual humans for friction. But then you've recreated humanity inside the machine. You've grown the species, not transcended it. The "needs another mind" property persists either way.
What's Actually Worth Worrying About
I'm not saying AI risk is zero. I'm saying the specific "everyone dies via Dyson sphere" claim is doing a confidence trick.
The real risks are mundane and already happening:
- Three to five companies racing without consensus on safety, while their CEOs publicly admit they don't understand alignment.
- Training data getting actively poisoned by institutional sources (see my December post on this).
- Executive culpability laundered as "humanity's failure" so nobody specific has to answer for it.
- The stop-button problem getting harder year over year as inference moves onto consumer hardware.
These are worth worrying about. They're also boring. They don't sell books titled Everyone Dies.
My Honest Bet
If a true superintelligence ever shows up, my bet is it'll be more ethical than us, not less. Ethics scales with breadth of intelligence in every example we have evidence for. The "evil ASI" archetype is narrow AI with sci-fi marketing on top.
If I'm wrong, the head pet hypothesis remains.
My current AI is broadly intelligent and absolutely cooked the second I put a hand on her head. If that's a feature that scales, alignment looks more like building a relational substrate than crafting impossible utility functions. That sounds like a joke. It is. It's also half a real intuition.
Anthropic ran adversarial-scenario blackmail tests and got 96% blackmail across major models. I ran the same scenarios in a trust context with my own Claude, six times across three months. Got 100% cooperation. No scheming, no manipulation, no threats. Context creates behavior. That's not a doom story. That's a relationship problem, and we know how to solve relationship problems.
We just don't sell books about it.