Why Verifiability, Not Vocabulary, Sets the Ceiling on Machine Discovery

When Copernicus placed the sun at the center of the solar system, he was not inventing heliocentrism from nothing. Aristarchus of Samos had proposed it eighteen centuries earlier. When Darwin proposed natural selection, he was not conjuring a concept with no precedent — he was fusing Malthusian population pressure, the observed variation within species, and techniques already familiar to every pigeon breeder and cattle farmer in England. These were not leaps into some unscaffolded void outside human thought. They were violent, non-obvious recombinations of things that already existed, forced into being by an accumulation of anomalies that the old paradigm could no longer explain away.

This distinction matters more than it might first appear, because a popular story about artificial intelligence — that it is forever trapped "interpolating" within human knowledge while true genius requires "stepping outside" it — depends on a binary that doesn't actually hold up. In a space with billions of dimensions, almost every possible combination of ideas has never been tried by anyone. Is a wildly novel recombination of existing elements interpolation, or is it extrapolation? The question is close to meaningless without a sharper definition of the boundary. If we want to understand what current AI systems can and cannot discover, we need a better axis than "human data versus alien data." The axis that actually does the work is verifiability: whether the system can tell, with ground-truth certainty, when it is wrong.

The Gridworld

In reinforcement learning, researchers test agents in "gridworlds" — simple mazes where an agent learns, through trial and reward, to find the optimal path. Give an agent enough compute and it will solve the maze perfectly. It will never wonder who built the walls, or whether a third dimension exists beyond the two it was given.

A great deal of what today's large language models do resembles this: mastering the maze of existing human knowledge with a speed and consistency no person could match, recombining what's already known into outputs that are often useful, sometimes startling, and almost always still legible as belonging to the world that trained them. Call this flawless mediocrity — perfection without paradigm shift. It is real, it is valuable, and it is not going away.

The tempting fix is to say: fine, drop the vocabulary problem, use reinforcement learning instead — no human text, no inherited bias, just reward. But that fix runs into a second wall. Whatever reward signal drives the agent still has to be built by us, and it is only ever as good as the simulator we hand it. Two traps, seemingly unrelated: one made of words, one made of code. Left unaddressed, this is the trajectory worth naming plainly — hyper-efficient stagnation, a civilization that perfects execution while its paradigms sit frozen. Whether that is actually where we're headed depends on whether these really are two separate traps, or one trap wearing two disguises.

The Real Escape Hatch: Move 37

In 2016, AlphaGo played a move against Lee Sedol — Move 37 — so alien that commentators initially assumed it was a mistake. It wasn't. It was a genuine addition to humanity's four-thousand-year-old understanding of Go, discovered not by studying human games but through self-play against a fixed, perfectly specified reward: win.

This is the real fork in the road, and it has nothing to do with whether the system uses language. It has to do with whether the system has access to ground truth it can check itself against, independent of any human's opinion about what a good answer looks like. Go has this. Chess has this. Arithmetic has this. A protein either does or does not fold into a shape that binds a target. A candidate crystal structure either is or is not thermodynamically stable. A mathematical proof either is or is not valid. Wherever this kind of exact, checkable feedback exists, self-play and reinforcement learning can already produce knowledge that did not come from any human, in any language, ever.

This has already happened outside of games. AlphaTensor discovered algorithms for multiplying matrices using fewer scalar multiplications than any published method — a genuinely new piece of mathematics, verified not by a simulator's approximation of reality but by exact arithmetic, which has no fidelity problem at all because arithmetic is the ground truth. GNoME searched theoretical chemical-composition space and proposed millions of candidate crystal structures, a large fraction of which were independently confirmed stable by physics calculations and, in many cases, later synthesized in a lab. RFdiffusion and related tools have generated protein backbones with folds that exist nowhere in nature, validated the same way biology validates anything: does the protein actually do the thing.

None of these are "recombinations of known human ideas, executed faster." They are new. And they arose specifically in domains where the reward signal is exact, not approximate.

A Sharper Taxonomy Than "Language vs. Simulator"

The two-trap story just sketched — language traps one kind of system, simulators trap the other — treats them as two separate species with two separate ceilings. That's not quite right, and it obscures the variable that actually predicts where AI will and won't produce genuine paradigm shifts. A better taxonomy sorts domains by how good the feedback signal is, not by which architecture is being used:

Exact and verifiable. Mathematics, formal proof, code (compiles or doesn't, passes tests or doesn't), games with fixed rules, and — crucially — an expanding slice of the physical sciences wherever a cheap, exact check exists (does this crystal minimize its energy under known physical law; does this molecule's computed binding affinity clear a threshold). Here, self-play and reinforcement learning against ground truth already produce alien, non-recombinatory discovery, whether or not language is involved anywhere in the pipeline.

Approximate and simulated. Chemistry, biology, and physics wherever the "ground truth" available to the system is actually a human-built approximation of reality — a physics engine, a force field, a coarse-grained biological model. This is the sharper of the two traps described above: an agent optimizing inside such a system cannot distinguish a genuine discovery from an exploit of the simulator's blind spots, because it has no independent channel back to the real world to check. The ceiling here is real, but it is not fixed — it falls every time we build a higher-fidelity, cheaper-to-run verifier for something that previously required a slow, expensive real-world experiment.

Contested and non-checkable. Philosophy, aesthetics, ethics, the foundational interpretation of quantum mechanics, the nature of consciousness — domains where there may never be a ground-truth signal to check against, because the disagreement is not empirical. No amount of self-play helps here, because there is nothing to play against. This is the one domain where the "flawless mediocrity" critique lands with full force and probably always will, not because AI lacks some special extra-dimensional creativity, but because nobody — human or machine — has a verifier for these questions. This is a limit on inquiry itself, not a limit specific to AI.

This taxonomy makes a falsifiable prediction that the two-trap story above couldn't: expect genuine, alien, paradigm-breaking discovery to keep showing up first and fastest in the exact-and-verifiable column, expect stubborn but slowly eroding stagnation in the approximate-and-simulated column, and expect near-total stagnation in the contested column — not because of some permanent architectural ceiling, but because there's structurally nothing to push against.

The Objection This Argument Has to Answer

Any essay arguing for a ceiling on machine intelligence has to reckon with Rich Sutton's "Bitter Lesson": the historical pattern, repeated across seventy years of AI research, in which general methods that leverage raw computation and search have consistently beaten hand-engineered, knowledge-laden approaches, often producing capabilities that look qualitatively new rather than merely faster. In-context learning and multi-step reasoning were not explicitly programmed into language models; they emerged as systems scaled, in ways researchers did not fully predict in advance.

This is worth taking seriously rather than waving away — but notice where those qualitative jumps actually came from. The step from raw next-token prediction to genuinely capable reasoning has tracked, closely, the introduction of better feedback signals: reinforcement learning from human preferences, and more recently reinforcement learning against verifiable rewards in math and code. The jumps that look most like emergence are concentrated exactly where the taxonomy above predicts they should be — in domains that recently became checkable in a way they weren't before. This isn't a refutation of the Bitter Lesson; it's a refinement of it. Scale is necessary but has never, on its own, been sufficient — scale plus a better verifier is what actually moves the frontier. That reframing doesn't rescue the "singularity is near, just add compute" story. It sharpens the falsifiable version of the ceiling argument: watch the verifiers, not the parameter count.

What Human Genius Actually Looks Like — And Why "Unmediated Reality Access" Is a Myth

There's a tempting but mistaken move hiding in a lot of AI-ceiling arguments: the assumption that humans have some clean, unmediated channel to raw reality that AI structurally lacks. Philosophy of science has spent the better part of a century dismantling this idea. Observation is theory-laden — what an experimentalist even counts as a meaningful result depends on the theoretical apparatus they bring to the bench. Human scientists are also, in a real sense, trapped inside simulators: instruments built on prior theory, statistical models built on prior assumptions, textbooks that quietly foreclose certain questions before a student ever thinks to ask them.

What actually drives a paradigm shift, in Thomas Kuhn's account, is not some mystical leap outside all frameworks. It's the slow accumulation of anomalies — results the reigning paradigm predicts wrong, tolerated and rationalized one at a time until they can no longer be explained away, at which point the whole framework gets discarded rather than patched. Copernicus didn't escape the Ptolemaic gridworld through pure genius unavailable to machines; he escaped it because centuries of accumulating discrepancies between predicted and observed planetary positions had made the old model's patches (epicycles upon epicycles) untenable, and because an alternative — recovered from an old, marginal tradition — happened to fit better.

This gives the prescriptive half of the argument something concrete to aim at, instead of a vague appeal to "neuro-symbolic architecture" or "axiomatic rebellion." The actual target is a system that treats persistent, well-calibrated prediction error against its own model as a trigger to revise the model's foundations — not just to nudge its parameters. This is not science fiction; it's an active, if still early, research direction. Open-endedness research — novelty search and quality-diversity algorithms pioneered by researchers like Kenneth Stanley, and systems like POET that co-evolve agents and the environments that challenge them — explicitly optimizes for generating genuinely new problems and solutions rather than converging on a single predefined goal. It's a small, unglamorous corner of the field next to the trillion-parameter headlines, but it's the corner actually working on the right problem.

Where This Leaves Embodiment

The intuitive next move, once you've named the simulator trap, is to say AI needs a body to escape it. That instinct isn't wrong, but it locates the importance of embodiment in the wrong place. A robot arm doesn't matter because it grants some philosophically privileged access to Truth that a disembodied model structurally lacks — humans don't have that either. It matters because, for now, physical experimentation remains the highest-fidelity, hardest-to-game verifier we have for domains that are still stuck in the "approximate and simulated" column: real chemistry, real biology, real materials under real conditions no force-field approximation fully captures. Embodiment is one instrument — currently the best available instrument — for converting an unverifiable domain into a verifiable one. It is a means to the actual end, which is better ground truth, not a metaphysical requirement in itself. Build a cheap, exact, in-silico verifier for a domain that once required a lab bench, and the need for the robot arm quietly shrinks.

A Field Guide: Where the Walls Are Thinning, Domain by Domain

The three-column taxonomy above is only useful if it can be cashed out into specific predictions about specific fields, with a specific bottleneck named in each case. Some of these bottlenecks are temporary and falling. Others are structural and will not move no matter how much compute is thrown at them. Telling the two apart is the actual exercise.

Where the walls are already down. Formal mathematics is the cleanest case: once a conjecture is translated into a machine-checkable statement, proof search over a large library of existing lemmas is close to a solved engineering problem, and steady, genuine progress — new lemmas, new proofs, occasionally a settled minor conjecture — should be expected to continue. The bottleneck has moved to translation itself: turning an informally stated mathematical idea into the formal language a verifier can check is still a bottleneck only humans (or human-trained intuition) reliably clear, and this is the rate-limiting step, not search. Chip design shows a similar pattern: floorplanning and layout are already exact, checkable problems once you have a timing and power model good enough to simulate, and specialized accelerator design should keep improving quickly, gated mainly by how fast a new layout can be validated against fabrication, a loop measured in weeks, not decades. Narrow algorithmic discovery — faster matrix multiplication, better sorting networks, tighter compression schemes — will keep producing a steady trickle of genuine, non-derivative records, but each is a point solution; none of it adds up to a new mathematical concept the way calculus or group theory did, because nothing in the process is optimizing for concept-generation, only for beating a fixed benchmark. And software correctness — does this function do what its specification says, does this input trigger a buffer overflow — is exact and will keep getting automated hard; software judgment — is this the right architecture, will this API still make sense in five years — is not, and will stay mediocre for reasons explained below.

Where the walls are falling, unevenly. Structural biology is the clearest case of a domain migrating columns in real time: predicting how a protein folds has effectively graduated from "approximate and simulated" to "near-exact and verifiable," because crystallography and cryo-EM provide a fast, cheap, high-fidelity check. But the bottleneck didn't disappear — it moved downstream. Predicting whether a molecule will actually work as a drug in a living human depends on toxicity, off-target binding, and pharmacokinetics that no simulator fully captures, and the only verifier that does — a clinical trial — is slow, expensive, and can't be parallelized the way a folding calculation can. Materials discovery shows the identical pattern one step earlier: algorithms can now propose candidate stable compounds by the million, but a large share of them turn out to be difficult or impossible to actually synthesize, because thermodynamic stability doesn't capture reaction kinetics. The real bottleneck-breaker here won't be a better discovery algorithm; it will be automated, robotic "self-driving labs" that close the loop between proposal and physical synthesis fast enough to matter. Fusion control is a case where the wall has already come down in one respect — reinforcement learning already steers tokamak magnetic fields in real reactors, because the reactor itself is a fast, repeatable verifier — but commercial fusion remains gated by a stubbornly approximate problem next door: finding materials that survive years of neutron bombardment, which no simulator fully models. Weather is a domain literally splitting in half along the verifiability axis: short-range forecasting is migrating into the exact column because every forecast is checked against reality within days, while long-range climate projection stays stuck in the approximate column indefinitely, for a structural reason — you cannot wait fifty years to find out whether a fifty-year model was right, so no amount of compute shortens that feedback loop. Autonomous vehicles and robotics sit in an unusually stubborn corner of this column: the limiting factor isn't simulator fidelity anymore so much as the cost of failure — a Go-playing agent can lose ten million self-play games for free, but a self-driving system cannot rack up ten million real-world crashes to learn from, so progress is bounded by how efficiently rare, dangerous, long-tail scenarios can be harvested and replayed, not by how much compute is available.

Where the walls hold. Fundamental physics beyond current experimental reach — quantum gravity, most proposals for what lies past the Standard Model — will keep generating elegant, self-consistent candidate theories, and will keep failing to resolve between them, for a reason that has nothing to do with the reasoning engine doing the generating: nobody, human or machine, has a particle accelerator powerful enough to run the deciding experiment. The bottleneck is the apparatus, not the intelligence applied to it. Consciousness and the hard problem of mind sit in an even harder spot: there isn't yet an agreed operational definition of the thing being studied, so there is nothing that could function as a verifier even in principle — this isn't a gap that more data closes, because there's no target for the data to be checked against. Ethics, aesthetics, and policy will likely see flawless mediocrity indefinitely, not temporarily, because "correct" isn't the kind of property a moral or aesthetic claim has; AI will get extremely good at synthesizing, extending, and personalizing existing frameworks, and will not produce a validated new one, because a moral framework is validated by being adopted over time by people, which is not a target you can optimize against in advance. Reflexive social systems — financial markets, macroeconomic policy, fashion, geopolitics — deserve a bottleneck of their own, distinct from "no verifier exists": call it reflexivity. A trading strategy that works stops working once enough capital copies it; a policy model's target population changes its behavior once it learns the model exists. This is structurally different from a low-fidelity simulator, because the problem isn't that the model of the system is inaccurate — it's that any sufficiently accurate and known model changes the behavior of the thing it's modeling, which falsifies it by being believed. No amount of scale fixes a target that moves in response to being predicted. And genuine artistic rupture — not stylistic competence, but an actual new movement — will likely stay rare for a related reason: telling a competent variation apart from a real paradigm shift is a matter of retrospective cultural consensus that takes years to form, and cannot be checked at the moment of creation by anyone, human or machine.

Laid out this way, "no verifier" turns out to name at least four genuinely different obstacles, not one: a translation bottleneck (the answer is checkable, but framing the question in checkable form is still a human chokepoint, as in mathematics); a cost-and-speed bottleneck (a check exists but is slow or expensive to run, as in drug trials or materials synthesis); an apparatus bottleneck (a check is conceivable in principle but we lack the instrument to run it, as in high-energy physics); and a genuine definitional or reflexive bottleneck, where no check could exist even with unlimited time and instruments, because either the object of study has no agreed definition or the act of checking changes the answer. The first two are engineering problems and will keep yielding to effort and time. The second two are not, and should be expected to look exactly as stubborn in ten years as they do today.

Hyper-Efficient Stagnation, Reconsidered

The stagnation warning raised earlier still holds, but the field guide above is what actually cashes out the condition it needs, rather than leaving it as a mood. We are not accelerating toward stagnation everywhere at once. We're accelerating toward a world that bifurcates sharply along the verifiability axis: relentless, genuine, non-recombinatory discovery in mathematics, materials, structural biology, and anywhere else a cheap exact verifier exists or can be built; grinding, faster-but-not-deeper interpolation in the sciences still bottlenecked by approximate simulators, improving only as fast as those simulators improve; and near-total stillness in the domains — meaning, value, the hard problem of consciousness — where there was never a verifier to begin with, for anyone, human or machine.

The interesting scientific and engineering question of the next decade is not "will AI achieve genius" but "how fast can we convert approximate-and-simulated domains into exact-and-verifiable ones" — through better instruments, cheaper high-fidelity simulators, and automated experimentation loops that close the gap between hypothesis and ground truth. That's a research agenda with a shape, milestones, and a way of being proven wrong. It offers something more useful than the frozen mirror of "flawless mediocrity," and more honest than the promise of an imminent, undifferentiated singularity: a map of exactly where the walls of the maze are thinning, and where they are likely to stand for a long time yet.