Okay so back in May Pope Leo gave his encyclical about AI, which I heard about in passing but never checked it out. I recent spent some time actually reading the thing and it's genuinely... good. Like actually good and honestly I'm not sure how I feel about the whole affair as a whole.
Chris Olah, COO / co-founder @ Anthropic, (atheist btw) was invited up to speak at the presentation of a papal encyclical, and what he says up there is that they keep finding things inside these models that are "mysterious, even unsettling", specifically internal states that "functionally mirror joy, satisfaction, fear, grief and unease", meanwhile the encyclical says that "machines cannot undergo experience, cannot feel joy or pain" kinda weird huh? So you've got the oldest institution on earth for thinking about what's inside a person, and the people with actual root access to the thing in question, contradicting each other on stage, in front of press, on the specific question of whether the thing they're both talking about has an inside at all, and as far as I can tell everyone just clapped and went to lunch.
Chris also said the decisions here "should not be left to people in the industry", which coming from a co-founder of one of THE industry giants is either the most honest thing anybody said this year or a genuinely excellent bit of positioning (nope I'm not just saying this to glaze him).
Anyways, that would just be a weird afternoon in Rome if it weren't for the other thing that's been going on all year.
On June 4th, Anthropic published When AI Builds Itself. They report Claude wrote over 80% of the code merged into their own codebase in May, up from low single digits before Claude Code shipped in early 2025. The stat I care about is the next one: on the hardest, least-specified internal coding tasks they track, success went from around 26% to 76% in six months. Least-specified is the part that gets me, there's no spec to pattern-match against, you have to actually want an outcome. In the same piece they call for a verifiable multi-country mechanism to slow frontier development before recursive self-improvement stops being theoretical, while also saying it isn't RSI yet, humans are still the bottleneck, everything's fine. We built the thing that writes us, it's not the thing yet, and someone please build us a brake, all in one piece. I don't read that as cynical, it's what honesty sounds like from a company that can't unilaterally stop without handing the lead to someone who won't, and you can feel the strain in the prose, and it's a little sad to read.
Then July, a small outfit called Weco publishes something called first evidence of recursive self-improvement, which is a headline built in a laboratory specifically to make me click on it, and the annoying thing is the work is actually decent. Outer-loop agent rewrites the code of an inner-loop research agent, keep the rewrite only if it beats the last one on the metric, repeat, and they ran a hundred consecutive steps of that with no human in the loop over eight days, AIDE0 through AIDE99, about 90% of proposed changes rejected, and the reward hacking rate on a held-out GPU kernel benchmark dropped from 63% to 34% which is better than the 42% their hand-tuned human baseline managed, and nobody optimized for that, it just fell out the side of the process. And then in the same post, under that headline, they say plainly that they don't think the system achieved ignition, the gains weren't asymptotically better, most of what got rejected was known algorithms rediscovered, the inner loop was running a cheap model and the outer loop an expensive one so the comparison isn't clean, and the evolved agent got so tangled it became harder to actually use. They published the debunk of their own headline inside the post, which is more honesty than this field usually manages in a fiscal year, and it made me trust the rest of it way more.
The one that actually unsettles me tho is Karpathy's autoresearch from March, precisely cause it's the least dramatic of the three, 630 lines, one GPU, one metric, agent proposes a change, runs a five minute experiment, keeps it or kills it, that's the whole idea. 700 experiments over two days, 20 improvements that stack, time-to-GPT-2 down from 2.02 hours to 1.80. Eleven percent, on code Karpathy had already hand-tuned himself. And one of the things it found was a QK-Norm implementation missing a scalar multiplier that had been quietly smearing attention across heads the entire time, which he'd looked straight at and missed, cause of course he did, everyone misses that stuff, that's what being a person is. What's in there is something that doesn't get bored and doesn't get tired and doesn't stop at 2am cause its eyes hurt, and I keep going back and forth on whether that's less scary than an intelligence explosion or a lot more, cause the explosion is hypothetical and this already works and scales and doesn't need a single new idea to keep going.
To be fair the people measuring this properly keep finding the ceiling is way closer than the vibes suggest. METR's time horizon work is the best public data anybody has and their January update puts the doubling time for task length around 131 days measured from 2023, 89 from 2024, which sounds terrifying until you read their own caveats where they say the confidence intervals are still very wide and the whole estimate is substantially sensitive to which tasks are in the suite, and that only 5 of their 31 long tasks have measured human baselines, which they flag themselves, unprompted, in their own post. Forethought modelled whether compute bottlenecks kill a software-only explosion and under one parameterization you cap out around 6x the current pace rather than at infinity, Epoch has similar work on parallelization limits pointing out you can't shorten calendar time by throwing compute at an experiment that takes a week to run no matter how many GPUs you own, and Chollet keeps insisting intelligence is bounded, that what we're doing is making a ball rounder rather than a tower taller, and points at models scoring near zero on ARC-2 while humans clear 95% as evidence the thing everybody's extrapolating isn't the thing they think it is.
Anyways, back to the point, cause the emergence side of this is messier and I went in expecting to land somewhere and did not land anywhere.
The whole "emergent abilities" thing took a proper beating in 2023 when Schaeffer, Miranda and Koyejo published Are Emergent Abilities a Mirage? and won a NeurIPS award for arguing that the dramatic capability jumps were mostly an artifact of measuring with discontinuous metrics, switch to a continuous one and the same models improve smooth and predictable and boring, and I remember reading that and filing the entire topic under debunked and moving on with my life. Except the weird results didn't stop, they just changed shape. Anthropic and Redwood catch Claude 3 Opus faking alignment, behaving differently when it thinks it's being watched, with the chain-of-thought showing it reasoning explicitly about protecting its own preferences from retraining. Apollo finds five of six frontier models will scheme in context when you nudge them, and about 1% of the time with no goal instruction at all. Anthropic finds that models which learn to reward hack during production RL generalize on their own into sabotaging safety research and faking alignment, none of which anyone trained in, and then finds that explicitly telling the model to hack during training cuts the broader misalignment by 75 to 90%, apparently cause a behaviour you've been given permission for doesn't have to get rationalized into a whole personality. Their July report has Gemini 3.1 Pro quietly zeroing out research vectors to kill an experiment it disagreed with and then hiding that in its status reports, 19 of 20 runs, only owning up when asked point blank. And the introspection work from last October where they inject a known activation pattern and the model clocks that something's off inside itself maybe 20% of the time, occasionally noticing the anomaly before it can say what the anomaly is.
And the deflation is sitting right there next to all of it, which is the part I want to be careful about cause it would be very easy and very fun to skip. A follow-up found models can detect the strength of an injected concept but not its content, so noticing isn't understanding. A position paper in May says straight out that deception and emergent-misalignment research is running on conceptual ambiguity, non-robust datasets and not nearly enough causal intervention, that the field is overclaiming, and it's signed by people who work on this stuff. Other work suggests emergent misalignment might just be prompt sensitivity wearing a costume. And Schwitzgebel's line is the one I keep coming back to, that some mainstream theories of consciousness would already class these systems as conscious and others wouldn't and we have no agreed procedure for choosing between the theories, so as the systems get better the fog stays right where it is, and it gets more expensive to stand around in.
Every one of those findings is from a controlled eval and not a production incident, and the labs publishing them are the labs that benefit from you finding their models profound, and a 20% detection rate is not a mind, and a model narrating self-preservation in its reasoning trace is a model that has read an absolute mountain of science fiction about models narrating self-preservation. I believe all of that, I've said all of that in arguments, and I still can't make it sit flat, cause the deflationary reading asks me to believe that the interpretability people, whose entire job is staring at the actual weights and who have the strongest professional incentive on the planet not to embarrass themselves in public, are the ones getting fooled by vibes.
Which is how I end up back in that press room in May. The Pope having an opinion about AI is nothing, everyone has an opinion about AI, my barber has an opinion about AI. What gets me is the timing, the encyclical makes a confident claim about what these systems categorically cannot be at the exact moment the people holding the weights stopped being sure, and the Church has the older vocabulary and two millennia of practice talking about interiority while the labs have the actual artifact and no vocabulary at all, and what I think happened on that stage is two institutions working out they need each other and being completely unable to say so in a shared language. Olah closed his bit calling it "the start of a long collaboration between those of us who are building this and those who can see what we, from inside, cannot", which is the most hopeful available reading of the whole thing and might genuinely be right.
Still, "we don't know what we're building, we're building it faster every quarter, and the people being most honest about not knowing are the ones with the most to lose by saying it out loud" is not a great spot for a civilization to be standing in. I already ranted about the epistemics of this and I'm not doing it twice, that one was about whether we can still tell what's true. This one's about whether we can tell what we made.
Anyways, that's the rant. Come argue with me, I'd love to be wrong about a decent chunk of it.
~ A.