Okay so lately I've been getting absolutely bombarded with this stuff, death-of-truth video essays, dead internet theory threads, it's everywhere on my feeds, and it keeps coming up with my friends too, the same conversation on repeat where someone shares a thing and someone else goes "wait is that even real?" and nobody can actually answer, and with GenAI making it easier and easier to churn out plausibly believable slop it's genuinely getting harder to tell wtf is true and what's not. So I finally sat down and dug into what the actual research says, and I need to get this out of my system, because the more I read the more I'm convinced that most of the AI risk conversation is staring at the wrong apocalypse.
Like, the discourse is all about deception, right, scheming models, superintelligences that tell evaluators exactly what they want to hear until the day they don't need to anymore, and look, serious people spend entire careers on those questions and I'm not waving them away, but there's this quieter failure mode that needs no scheming, no superintelligence, no malice, nothing. All it needs is volume.
Here's the thing that actually broke my brain a little. In January 2026 some researchers showed people AI-generated deepfake videos of crimes, and they told them explicitly, in advance that the videos were fake. Didn't matter. Even the participants who consciously accepted "yes this is fake" stayed influenced by what they'd seen, which is so much weirder than "deepfakes are convincing", because it means knowing a thing is fake no longer reliably protects you from it, the machinery in your head that turns seeing into believing evolved for a world where seeing something meant it probably happened, and that machinery was never built for an environment where most of what reaches your eyes might be manufactured.
Now scale that up to the size of the internet and shit gets grim fast, by mid-2025 there were over 1,200 AI-generated "news" sites publishing under plausible mastheads in sixteen languages, the EU counted AI involvement in 27% of foreign information-manipulation attempts, nearly triple the year before, and AI-generated misleading posts were going disproportionately viral despite mostly coming from small accounts. None of this required a single misaligned model anywhere in the pipeline, all it required was that generating content became cheap.
Which is also where dead internet theory stops being a funny conspiracy and starts being, like, directionally correct. The strong version, "everything online is bots and you're the last human here", is obviously wrong, but the boring version gets more true every year, automated traffic crossed roughly half of all web traffic around 2024, slop farms figured out that engagement-bait images cost nothing to mass-produce and the recommendation algorithms will happily shovel them at boomers by the millions (shrimp Jesus, my beloved), and entire "authors" and "musicians" and "photographers" now exist that are just a prompt loop with a payout account attached. The theory got the mechanism wrong tho, there's no coordinated psyop, nobody needed one, you just need content generation to be free and attention to be monetizable and the synthetic stuff crowds out the human stuff the same way cheap counterfeits flood a market nobody polices.
And that's the real danger, not that people get fooled, but that they stop trying to figure out what's true at all, not because they're gullible but because they're fucking exhausted. This video was actually one of the ones that kicked off the whole bombardment for me and it walks through that exhaustion spiral way better than I can, so if you want the full gut-punch version, here:
Anyways, back to the point: "epistemic collapse". The term is floating around in at least four different research communities right now and they don't quite mean the same thing by it, which annoyed me at first, but the more I sat with it the more I think they're describing stages of one cascade rather than four separate problems.
The ML people mean model collapse, train a model on the outputs of a previous model, repeat a few times, and the tails of the original distribution vanish while each generation drifts further from whatever ground truth it was anchored to, and this has been proven mathematically btw, collapse can't be avoided when training solely on synthetic data. The ecosystem version is worse, when researchers tested 27 LLMs across 155 topics every single one was less epistemically diverse than a basic web search, and the larger models produced less diverse claims, not more, so monoculture isn't just a market concern, it's an accelerant. The AI-ethics people mean validation overload, peer reviewers, fact-checkers, editors, courts, teachers, all rate-limited systems trying to keep pace with a generation process that is no longer rate-limited by anything human, more claims than anyone can verify, synthetic content feeding on itself, and the humans doing the checking just... wearing down.
The philosophers mean something they call misrecognition, which sounds fancy but is honestly the one that haunts me most, a model neither knows nor claims to know the content of what it produces, its outputs are just statistical distributions, and when a reader treats that output as a knowledge claim anyway they're doing the epistemics on the model's behalf, quietly lending their own credibility to something that has none of its own, and now multiply that tiny act by billions of daily interactions and you get a society that systematically miscounts how much verified knowledge it actually has. And the social scientists mean the one that makes the news, disconnected information realities, lost common ground, radicalization inside silos, usually treated as its own thing, a "social media problem" or a "polarization problem", but I think it's just the surface expression of everything underneath it.
The cascade runs in order, the technical collapse degrades the tools, which overwhelms the validation infrastructure, which normalizes the misrecognition because unverified content is increasingly all there is, which finally fragments the shared reality that any kind of collective decision-making depends on.
Now, the obvious pushback, and I had this thought too: misinformation is ancient. Forged letters, yellow journalism, wartime propaganda, humanity has never lived in an epistemically clean environment, which is exactly why we evolved journals and courts and editorial desks and peer review in the first place. But what's new isn't deception, it's the unit economics of deception. Every verification institution we have quietly assumes that producing a credible-looking claim costs something, time, expertise, reputation, printing presses, whatever, verification could afford to be slower than generation because generation was expensive too, and that assumption just stopped being true. Producing a plausible claim now costs approximately nothing while checking one costs what it always did, some recent work calls the result "industrialized deception", automated production of misleading content at a scale no verification system was ever engineered for.
And there's this paradox that's actually been formalized which I can't stop thinking about: as synthetic media becomes indistinguishable from authentic media, the individually rational move is to discount all digital evidence rather than sort the true from the false, so the endgame isn't a world full of successful lies, it's a world where evidence stops working altogether. The right analogy isn't counterfeiting, it's debasement, what loses value is the currency itself.
The flip side even has a name already, the "liar's dividend", once everyone knows convincing fakes exist, anyone caught doing something real on camera can just shrug and go "that's AI", and the more slop is out there the more plausible that defense gets, politicians have already used it, so the fakes don't even need to fool you to do damage, their mere existence hands every liar a get-out-of-evidence-free card, real footage and fake footage end up equally worthless, and that's the debasement completing itself from both ends. This one covers that whole trust-collapse spiral, it's the other video I kept coming back to while chewing on all this with my friends:
So at this point you might be going "but surely the AI safety people are on this?" and yeah, kinda. Sorta. Not really. The most detailed governance proposal out there right now (AI Futures Project's "Plan A") is admirably concrete about compute verification, optical network taps, chip supply chain audits, mutually assured compute destruction as a deterrence backstop, hundreds of pages of modeling, and on epistemics? A two-page appendix. It names a genuinely useful idea, the "basin of sanity", a self-reinforcing equilibrium where truth-seeking AI tools make society saner, gestures at a handful of interventions, and moves on, nothing about how a society enters that basin, how to measure distance from its boundary, or what keeps it stable against actors who profit from the other basin.
The working assumption seems to be that if we control the compute and align the models the epistemic environment will mostly take care of itself, and the research says no, it won't. A perfectly aligned model still floods the commons, alignment constrains intent and this cascade doesn't run on intent, it runs on volume, honest models trained on increasingly synthetic corpora still drift, validators still burn out under content produced in complete good faith, readers still mistake statistical output for knowledge even when the statistics are benign. Alignment is necessary for the deception problem and nearly irrelevant to the exhaustion problem.
There's also a deeper structural mismatch here that I think explains why nobody's touching this, alignment is a property of an artifact, which means somebody can own it, a lab, a safety team, a regulator certifying a model before release, but verification capacity is a property of an ecosystem, it lives in the coupling between models and institutions and incentives and habits of mind, and our institutions are good at assigning responsibility for artifacts and notoriously shit at assigning responsibility for ecosystems. So the tractable, ownable problem absorbs all the funding and talent while the binding constraint absorbs neither.
So what do we do about it? Honestly? I have no fucking clue. I'm not gonna stand here and pretend I've got a five-point plan for fixing the epistemic commons, I'm just ranting. There are ideas floating around in the literature, tracking the health of the information ecosystem the way we track the health of the economy, keeping model ecosystems diverse so the monoculture thing doesn't accelerate, funding verification tooling with even a fraction of the money we pour into generation, but I have no idea if any of that actually works, and neither does anyone else, because nobody's actually trying it.
And that's the strange sociological fact that made me write this whole thing in the first place. The ML community has proven collapse theorems, the AI-ethics community has formalized validation failure, philosophers have diagnosed the misrecognition, social scientists have measured the fragmentation, and the most ambitious AI governance proposal in circulation literally concedes in an appendix that keeping society inside the basin of sanity should be a top governmental priority during AI takeoff, and yet there is no field here. No shared benchmarks, no observatory, no funding stream, no institution anywhere whose actual mandate is the verification capacity of the knowledge commons, alignment has labs and fellowships and conferences, epistemic infrastructure has scattered arXiv postings that only occasionally cite across disciplinary lines.
Which is both the opportunity and the warning, because every year the generation-verification gap widens, the institutions that would need to build this stuff operate with less credibility and more fatigue than the year before, this is the rare risk that erodes our ability to respond to it as a consequence of the risk materializing, a society that can no longer agree on how to verify claims can't verify the claim that it should fix its verification systems. Sit with that one for a sec. The race to build more capable AI has a thousand entrants, the race to keep truth checkable has almost none, and honestly? I think the second one is the actual bottleneck.
Anyways, that's the rant. If any of this itched your brain the way it itched mine, reach out, I'd genuinely love to argue about it.
~ A.