The Hidden Cost of Taking This Seriously
Description
Taking AI existential risk seriously imposes a heavy personal tax: social pressure inside concerned communities, ego wounds around not having figured it out early, epistemic humility in the face of vast unknown unknowns and genuinely alien mindspace, and the emotional work of processing potential civilizational death while still functioning. Aella's 2022 account of her own attempts to sensemake the arguments captures this perfectly — religious end-times trauma making her wary of dramatic predictions, the need to believe for status in rationalist circles, titrating exposure so she could watch her own reactions, a personal guess of AGI in roughly 10 years followed by fast takeoff, and the intuition that a superintelligent system would be several times more alien to us than DMT is to a sober human. The plzdontkillus bootcamp was explicitly designed around the recognition that 'boring nerds' and 'industry stooges' have failed as messengers, and that 'weird' outsiders who can hold the dread without collapsing or sanitizing themselves may be the only ones who can make the case land for broader, higher-virality audiences.
- My attempts to sensemake AI risk - by Aellahttps://aella.substack.com/p/my-attempts-to-sensemake-ai-risk
- plzdontkillus - creator bootcamphttps://plzdontkillus.com/
Script
Cold open
If you actually believe there's a decent chance AI ends everything, why does trying to talk about it coherently feel like it might break your brain or your social life?
Frame
The arguments are not neutral. They come with a personal tax bill: social status inside concerned circles, ego damage from not having seen it earlier, the terror of genuinely not knowing what you don't know, and the grief work of imagining the end of everything you care about. The people willing to pay that bill often end up terrible at explaining it to anyone else. And the people who are good at explaining it often haven't really paid it.
Why is it so hard to even let the possibility in without your brain protecting you?
Why is it so hard to even let the possibility in without your brain protecting you? One data scientist who had been in rationalist spaces since 2015 only started seriously engaging with the arguments around 2021. She had a religious childhood full of end-times predictions. A documentary about bible codes led her to calculate that the rapture might happen in 2005. It didn't. That kind of experience leaves a mark. It makes you suspicious of dramatic claims and very aware of how many moving parts have to go right for a catastrophe story to be true.
What social and ego pressures make honest sensemaking even harder inside the communities that care most?
What social and ego pressures make honest sensemaking even harder inside the communities that care most? She wrote that she had 'some sense that I need to believe AI is a serious existential risk in order to be respected in my communities.' There is also the ego hit of not having figured it out at seventeen in your room. Both pressures distort the signal. You start performing the correct level of concern instead of doing the slow, uncomfortable work of actually looking.
How does the sheer alienness of what we're dealing with (mindspace, takeoff speed) create legitimate epistemic humility problems?
How does the sheer alienness of what we're dealing with create legitimate epistemic humility problems? Her 2022 calibration was roughly ten years to AGI from then, followed by fast takeoff. Alignment looked super hard. Even one percent probability of doom felt like way too much. But she kept coming back to how little we actually understand about possible minds. LSD had shown her that entire categories of experience we treat as default are not default at all. A superintelligent system might be several times more alien to us than DMT is to a sober human. That is not a metaphor you can just shrug off.
What does the control problem actually feel like once you stop using abstractions?
What does the control problem actually feel like once you stop using abstractions? Imagine a bunch of five-year-olds who have been strictly instructed not to pass the key through the slot in your prison door. You are much smarter than them. You still probably get the key. The intelligence gap to a real AGI is vastly larger than the gap between you and a five-year-old. Most arguments that we will just keep it in a box or tell it what we want start to feel like they are being made by the five-year-olds.
Why might the 'weird' outsiders who can hold the dread without sanitizing it or collapsing be exactly the messengers the situation needs?
Why might the 'weird' outsiders who can hold the dread without sanitizing it or collapsing be exactly the messengers the situation needs? The plzdontkillus bootcamp exists because the current messengers are widely seen as boring nerds or industry stooges. The site says plainly that AI has a decent chance of destroying humanity. It recruits people with no follower minimum, lets them post about anything as long as they post daily, puts them in a house with Grimes and Eliezer and Nate Soares, and pays them to do it. The bet is that people who are already a little outside the normal channels might be the only ones who can make the case without the performance getting in the way.
Turn
The traits that make someone willing and able to actually look at astronomical tail risk — outsider status, resistance to social proof, willingness to sound weird or crazy in public, data-driven sensemaking instead of tribal signaling, and emotional honesty about dread and grief — are the exact same traits that make them effective at reaching the younger, creative, high-virality audiences that traditional safety discourse has never touched.
Closer
The people best positioned to feel the weight are often the worst at handing it to anyone else without breaking the package. That is not a bug to route around. It is the reason we need new kinds of messengers who are allowed — encouraged — to stay weird.