The Obligation of the Impossible: Going Beyond the Complexity Ceiling

The era of agentic software development has arrived with a striking, juicy promise: multi-file codebase awareness, autonomous background agent execution, and a drastic increase in code output. According to its adepts, engineering teams should no longer reason line-by-line; they should direct autonomous workflows that refactor, debug, and implement full features across dozens of scattered files simultaneously. Yet, as the volume of generated code explodes, we are hitting a massive, unyielding wall. That wall belongs to one particular way of working — a human steering, reviewing, and re-prompting an agent one file at a time. It says nothing about what happens when the human step is removed altogether.

Qian et al.'s ChatDev assembles a fully virtual software company: LLM agents playing CEO, CTO, programmer, reviewer, and tester carry a one-line product brief through design, coding, testing, and documentation with no human writing a line of the resulting code. The roles that used to require a room full of engineers now run inside a single orchestration loop, and a working, runnable artifact comes out the other end.

The industrial case is sharper still. In May 2025, Google DeepMind's AlphaEvolve, an evolutionary coding agent built on Gemini, discovered a way to multiply two 4×4 complex-valued matrices using only 48 scalar multiplications — improving a bound set by Strassen's algorithm in 1969 and left untouched for 56 years. The same system now runs in production inside Google's own infrastructure, autonomously rewriting the scheduling logic that allocates compute across the fleet and continuously recovering, on average, 0.7% of Google's worldwide compute resources. No engineer hand-derived either result; an agent searched, verified, and shipped.

These are not isolated party tricks. A virtual company of agents can carry a one-line brief to working software before a human team finishes its first stand-up; an evolutionary search can out-do a fifty-six-year-old mathematical bound overnight and then go run a hyperscaler's own infrastructure.

The ceiling on what software can become was never physical. The question is no longer whether machines can build ambitious systems without a human watching every line. It is what we choose to point them at.

01 Where Do We Go From Here?

With this, the question of what to build is just as important as ever. While we see many re-implementation or code modernization projects, there is no real glory in using this profound computational unlock simply to rebuild the exact same systems we already possess. Using autonomous multi-agent orchestration layers merely to clear out routine IT backlogs, refactor legacy boilerplate, or optimize familiar business logic slightly faster is likely an immense waste of a generational paradigm shift, not to mention the human costs involved.

The true dignity and challenge of this new epoch lie in our willingness to look past the familiar.

Before we can push beyond this boundary, we must first confront a vital ideological question: what is the right challenge for this generational paradigm shift, and how do we best deploy this newfound power?

• Correctness. Given the encouraging results of the last year or two, we see more than a glimmer of hope for better correctness outcomes achieved not through manual code reviews or endless cycles of automated regression testing, but by anchoring our systems to stronger mathematical verification layers of formal reasoning that can prove architectural integrity properties before a single line is executed in production. This progress gives us the ability to go further.
• Complexity. Useful complexity is unlocked by consciously turning our cognitive energy away from routine boilerplate maintenance and focusing entirely on bigger, more complex, and more ambitious projects. By establishing rigorous, machine-checked guardrails, we gain the structural safety net required to expand our engineering horizons, liberating our creative ambition to design infrastructure that was previously deemed fundamentally unbuildable.

02 The Monuments of Unfathomable Code

To understand why we must leap beyond the traditional frontier, we must first look at the sheer scale of the software monuments we have collectively built thus far. The most complex software systems ever constructed are no longer just applications; they are sprawling, artificial digital continents.

Consider the modern infrastructure that anchors our global economy:

• The Monolithic Sovereign: Google's central repository stands as one of the largest single codebases in human history, housing an estimated 2 billion lines of code across a single monorepo. It requires tens of thousands of active developers executing millions of commits annually just to keep the ecosystem up-to-date.
• The Operating Theater: A modern operating system, or the core software suite running an enterprise cloud platform like AWS, easily spans 50 to 100 million lines of code.
• The Sovereign Vehicles: A modern premium connected vehicle or a commercial jetliner running advanced fly-by-wire and autonomous assistance systems routinely relies on 100 million lines of code distributed across dozens of interlocking Electronic Control Units (ECUs).

The financial and temporal costs of constructing these monuments under the legacy paradigm are staggering. To build a highly complex, enterprise-grade system from scratch, companies face a baseline investment starting at hundreds of thousands to tens of millions of dollars, with development timelines routinely bleeding across 12 to 18+ months before a single stable version is deployed. When adjusted for scale, building and validating safety-critical systems historically required hundreds of millions of dollars and multi-year orchestration cycles.

At this tier of complexity, the software becomes essentially unmanageable by human cognitive power alone. The system experiences severe architectural drift, where a localized, minor change to a data serialization method on one edge of the codebase can trigger a cascading, catastrophic tail-latency failure on the other.

We have built towers so high that no single architect can see the foundation while standing on the roof.

03 The Metric of the Leap: Effort Compression

When we compare this legacy model to an autonomous, agentic approach, the metric that matters most is effort compression—the radical reduction of human labor hours required to manifest identical semantic outcomes. In traditional systems engineering, code is written entirely by hand. A developer manually drafts data models, handles edge-case error checking, sets up network routing tables, and builds testing frameworks line-by-line.

When this process is shifted to an autonomous loop, the human is elevated to writing high-level specifications. While that flow is not without many caveats (see a recent blog post on under-specification), the agentic system handles the verbose implementation details, the compilation fixes, and the validation cycles autonomously.

This data proves that the compression of effort is not marginal; it is non-linear.

By stripping away the unglamorous mechanical execution of typing, merge conflicts, and compiler warfare, we should free human talent to operate strictly at the tier of macro system architecture. This, however, implies larger, more complex systems.

04 The Mythical Agent-Month: Breaking the Glass Monolith

Brooks's fifty-year-old warning that adding manpower to a late project makes it later has a new incarnation in the agentic era: when code generation is treated as free, infinite labor, the result is not faster delivery but scope creep, structural bloat, and a codebase that turns opaque exactly when it needs to stay legible.

Brooks's Law and the Tar Pit

In 1975, Fred Brooks gave the software industry its most enduring axiom: "Adding manpower to a late software project makes it later." This law—The Mythical Man-Month—exposed a fundamental bottleneck in human organization. Software development is not a task that can be partitioned into isolated, assembly-line components. It is an exercise in complex, interconnected logic.

When you add a human developer to a team, you don't just gain their coding throughput; you introduce a devastating communication penalty. In a team of n engineers, the internal communication channels grow quadratically; eventually, the team crosses a critical threshold where every ounce of human energy is consumed by meetings, alignment, and resolving merge conflicts, rather than writing code. The project collapses under its own organizational weight into what Brooks famously termed the "Tar Pit."

The Mythical Agent-Month

Today, we are watching the industry fall headfirst into a modern, automated incarnation of this exact trap: The Mythical Agent-Month.

The prevailing optimization thesis of the current AI epoch is cargo-cult thinking. Engineering teams are treating large language models simply as hyper-fast, zero-cost human programmers. If a human team is bogged down by a backlog, the modern prescription is to spin up autonomous multi-agent orchestration layers—deploying swarms of digital engineers to generate code, patch vulnerabilities, and execute refactoring tasks at unprecedented velocity.

But as Wes McKinney (creator of pandas and Apache Arrow) observes, dropping the marginal cost of code generation close to zero completely destroys the structural friction that used to keep systems cohesive. When writing code is hard, engineers are forced to exercise discipline, taste, and intentional constraint. When code generation is free, the barrier to saying "yes" vanishes.

The result is a devastating wave of automated scope creep and structural bloat. Instead of managing human communication overhead, the enterprise architecture is suddenly drowning in inter-module dependency complexity. The autonomous agents throw "one more million tokens" at a problem to patch an edge-case bug, inadvertently generating thousands of lines of hidden, brittle logic. Beyond a critical mass—typically around 100,000 lines of code—the system transcends human comprehension.

The codebase enters the Agentic Tar Pit: a state where the AI agents begin to suffocate under the weight of the very code they generated, creating intertwined, opaque dependencies that become impossible for a human to audit, guide, or structurally verify.

The Glass Monolith

To visualize why traditional, hand-coded architectures hit an unyielding ceiling under this paradigm, imagine a machine constructed entirely out of interlocking glass rods and gears.

When the machine is small (Phase 1), it possesses structural clarity. A master engineer can look through the glass, trace the exact path of mechanical energy, and predict how turning a dial on the far left affects a lever on the far right. This is software under 50,000 lines of code.

But as the system scales to millions of interlocking pieces (Phase 2), the medium itself betrays us. The sheer volume of overlapping components causes light to refract, bounce, and scatter. The interior of the machine transforms into a visually opaque, dark gray mass. The structural translucency is completely lost.

When a hidden internal gear cracks due to a rare edge-case input, the human engineer sees the outer framework shatter, but finding the original point of fracture becomes an act of forensic guesswork. The team is no longer engineering; they are managing a chaotic black box. Hiring more humans or spinning up more agentic coding loops to patch the glass monolith doesn't make it clearer; it simply adds more overlapping components that block the remaining light.

The Way Through the Glass

Yet this is not a dead end. A companion piece on post-textual software engineering argues that the Glass Monolith's opacity is fatal only if we keep debugging it the old way — human eyes tracing gears through glass.

Replace that expectation with machine-checked invariants, behavioral telemetry, and agent-to-agent debugging loops, and opacity stops being a failure mode and becomes an accepted property of the internal execution plane — provably safe at the boundary even when no human can see through it. On that architecture, there is real hope of eventually approaching the complexity and scale of the monuments surveyed earlier: Google's two-billion-line monorepo, the fifty-to-hundred-million-line operating platforms and vehicles — sustained not by engineers holding the whole system in their heads, but by proofs that never require them to.

05 The Moral Imperative

The transition to trained, verifiable software foundations is not merely a technical or economic necessity; it carries a profound moral obligation to expand the boundaries of what humanity can build. As agentic AI drastically compresses the labor hours required to write standard code, the tech industry stands at a critical ethical crossroads.

The Subtractive Impulse

The prevailing corporate impulse is sadly cargo-cult and subtractive in nature: to treat AI tools merely as human-replacement engines designed to shrink engineering departments, eliminate mid-level developer roles, and squeeze margin out of existing software paradigms. This subtractive mindset represents a severe failure of imagination, using history's greatest computational unlock to downsize human potential rather than amplify it.

The Additive Alternative

True ethical leadership demands an additive framework—one that shifts human effort rather than eliminating it. When AI systems absorb the unglamorous mechanical execution of typing syntax, tracking boilerplate, and fighting compilers, the human developer is not rendered obsolete.

Instead, their cognitive bandwidth is liberated.

A New Tier of Engineering

By elevating the human role from manual code production to high-level system architecture, combined with correctness tooling such as elements of formal verification, we unlock a massive net expansion of industrial capability. Building planetary-scale energy orchestrators, real-time closed-loop therapeutics, and polymorphic hardware networks requires more human oversight, systemic design, and philosophical guardrails—not less.

This additive paradigm flips the script on technological unemployment. Instead of hollowing out the profession, it creates a new tier of highly sophisticated, intellectually fulfilling engineering roles tasked with directing autonomous multi-agent knowledge factories — repositories where humans and agents co-produce software at scale, the framing Syme develops in "Understanding Repositories as Human/Agent Knowledge Factories" (2026) — and safeguarding absolute mathematical invariants.

In Search of Dignity

There is no dignity in using a revolutionary technological leap to build the exact same systems slightly faster or with fewer people.

In the fullness of time, the true validation of the AI epoch will be measured by the net-new value we introduce to the world and the unprecedented systems we bring into existence. We are morally obligated to utilize this profound explosion in velocity to scale our ambitions, expand the global engineering footprint, and confidently march beyond the complexity ceiling to solve the impossible.

06 Crossing the Frontier: From Coding to Training

The ultimate lesson of both the Mythical Man-Month and the Mythical Agent-Month is that human-readable, procedurally scripted text files are a fundamentally unscalable medium for high-complexity systems. An earlier blog post, From Code Complete to Code Missing, pushes this same premise to its logical extreme: source code was never a computational requirement, only a concession to the limits of the human brain — and lifting that concession doesn't just fix legibility at scale, it removes the abstraction tax that has been throttling runtime performance all along. When the glass monolith turns opaque, we must stop trying to manually map the gears.

Ultimately, we must cross the frontier and shift our paradigm entirely from coding to training.

Software 2.0: From Instructions to Weights

In the traditional frontier, software is built deterministically. A human or an agent writes explicit, line-by-line conditional instructions. Beyond the frontier, we abandon procedural instructions altogether. Software transitions into an intent-driven, probabilistic medium optimized through continuous training loops. This is the structural reality behind Andrej Karpathy's framing of Software 2.0—the shift from human-authored strings to data-driven matrices of weights compiled directly via optimization layers.

Blondel and Roulet's The Elements of Differentiable Programming provides the formal scaffolding for this transition, showing how gradient-based optimization supersedes procedural instruction as the fundamental computational primitive.

The System Architect

Instead of instructing how a system must execute a task across millions of lines of text, the human engineer operates as a System Architect. The developer's role is elevated to defining the macro-objectives, the evaluation functions, and the absolute mathematical invariants that the system is forbidden to violate — a fundamental restructuring of requirements engineering from functional specification to objective specification, a transition the software-engineering research community had already begun documenting by 2019.

The system then throws massive compute at training optimal execution pathways autonomously. The software behaves like a biological organism, continuously morphing its internal state, data layouts, and memory profiles to match real-time telemetry and changing hardware platforms — the same intuition Kevin Kelly explored three decades earlier in Out of Control, his study of emergent, self-organizing systems that behave more like biology than engineering.

The Formal Safety Net

To ensure safety in this black-box paradigm, we throw away manual, human-in-the-loop code reviews. Instead, we anchor the fluid system to machine-checked formal verification layers (such as the Lean 4 theorem prover). The software can mutate and optimize its execution graphs freely, but the verification layer continuously generates mathematical proof that the system's operational choices never breach our predefined structural boundaries.

07 The Frontier Pushed: Domains of Emergent Logic

We don't have to wait for the distant future to see this frontier being shattered; we are actively witnessing it across industries where manual, deterministic programming has hit an unyielding complexity wall.

Interactive Media and Game Development

For decades, video games were the ultimate manifestation of rigid, human-designed determinism. A game designer sat down and manually drafted branching dialogue trees, coded predictable state-machine behaviors for Non-Player Characters (NPCs), and carefully hand-balanced every single character attribute and environment variable.

Now, studios are subverting this paradigm by using AI to train gameplay and simulate dynamics rather than designing them manually.

Instead of following predictable, hard-coded routines, NPCs are trained via reinforcement learning loops to reason about the state of the world in real time. Modern AAA and indie studios are increasingly utilizing advanced orchestration stacks like Inworld AI to replace hard-coded dialogue scripts with "Memory-First" AI systems. Rather than forcing a writer to draft millions of permutations for an open-world sandbox, developers configure a character's identity, core motivations, and world lore into the model. The NPC then dynamically generates context-aware, low-latency responses based on the player's full session history.

According to an industry overview, studios are compressing world-building and balancing timelines dramatically. For instance, Riot Games reported running up to 4 million simulated matches per week via automated AI agents to test design variations and mathematically optimize gameplay balancing—a scale of regression testing completely impossible to execute by hand.

The same shift is visible at the frontier of racing AI. Sony AI's Gran Turismo Sophy project trained a deep reinforcement-learning agent to race against (and beat) world-champion human drivers in Gran Turismo 7. Published in Nature by Wurman et al. (2022), the result was not a scripted bot that memorized the racing line; it was an agent that had learned, through millions of simulated laps, to discover novel braking points and slipstream tactics that human drivers had not consciously articulated.

The training pipeline ran on a distributed cluster; the agent never read a line of hand-authored "how to drive" code; and the resulting policy exhibited emergent racecraft—drafting, defensive blocking, late-braking passes—that the human designers had to reverse-engineer from the agent's behavior rather than specify up front.

Electronic Arts' SEED group has documented the same pattern on the testing side, where reinforcement-learning agents now play AAA titles thousands of hours per week to surface progression-blocking bugs and balance regressions that no human QA team could reach on a reasonable budget.

Gudmundsson et al. at King (the studio behind Candy Crush) have published complementary work showing that a trained playtesting agent can predict human-perceived difficulty levels across thousands of levels far more accurately than the studio's prior heuristic models, letting designers retire brittle, hand-tuned difficulty curves in favor of agent-informed ones.

High-Precision Medical Imaging

In healthcare diagnostics, the frontier is shifting from hand-crafted digital filters to deep, probabilistic image synthesis. Historically, computer vision software for MRI or CT scans relied on rigid mathematical algorithms manually tuned by engineers to detect edges, isolate organs, or highlight anomalies. If a patient shifted slightly, or if hardware variables drifted, the underlying code struggled to adapt.

Today, medical imaging systems deploy promptable, concept-driven segmentation networks that do not operate on fixed, procedural rules. Instead, they act as fluid networks that adapt dynamically to noisy sensory inputs, reconstructing clear diagnostic data and outlining complex anatomical structures with an algorithmic accuracy that hand-coded software simply cannot replicate.

In the medical domain, manually annotating pixel-by-pixel boundaries for tumors or retinal layers used to consume senior radiologists' time for days on end. As documented in modern enterprise reviews, platforms like Scale AI, Encord, and V7 are leveraging model-assisted labeling (MAL) and universal mask classification architectures to turn raw, imperfect imagery into precise, diagnostic training sets automatically.

The shift is not incremental—it is categorical. The open-source MedSAM model, a medical-specialized segmentation-anything architecture, has been fine-tuned on over 1.5 million medical image-mask pairs spanning CT, MRI, ultrasound, and dermoscopy. A single text-prompted query ("segment the liver") returns a pixel-accurate boundary in seconds, a task that previously required a trained radiologist minutes-to-hours per slice across a typical volumetric scan.

The trained model adapts to organ shape, scanner vendor, and patient anatomy without any hand-coded conditional logic for each modality—behavior that would have required thousands of lines of bespoke image-processing code in the Software 1.0 era.

Autonomous Flight Control Systems

Nowhere is the limit of human programming more apparent than in aerospace. Modern unmanned aerial vehicles (UAVs) and next-generation flight control systems must operate in hyper-dynamic, unpredictable atmospheric environments. Writing deterministic, rule-based code to account for every conceivable wind shear, mechanical anomaly, or obstacle variable requires millions of lines of interwoven conditional statements that become impossible to test reliably.

The frontier is being pushed by moving toward trained neuro-flight controllers. The software learns optimal control laws through millions of accelerated simulations, executing micro-adjustments to flight surfaces at sub-millisecond rates. The transition from manual PID (Proportional-Integral-Derivative) loop tuning to deep reinforcement learning (DRL) is a core focus of modern drone aviation.

Manual control tuning is notorious for failing when a physical drone encounters hardware defects mid-flight or extreme, real-time wind deltas. Deploying deep reinforcement learning agents within ultra-realistic virtual simulators allows autonomous flight software to execute hundreds of thousands of trial-and-error configurations safely. The resulting trained "neuro-flight" models consistently outperform traditional human-scripted controllers, maintaining stability during simulated in-flight hardware degradation by adjusting control surfaces at an unprecedented rate.

A concrete demonstration of this crossover is Gran Turismo Sophy's hardware analogue: Scaramuzza's robotics group at the University of Zurich trained a reinforcement-learning controller entirely in simulation, deployed it on a physical quadrotor, and outpaced three champion-level human pilots on a real indoor course (published in Nature, 2023). The trained policy executed 100+ Hz control updates—well above the marginal stability envelope that a human-tuned PID controller could sustain. The result was not a one-off: at the 2025 A2RL Drone Championship in Abu Dhabi, a monocular-vision racer built by TU Delft's MAVLab beat three human world-champion pilots in open competition, reaching speeds of 100 km/h (Bahnam et al., 2026).

The controller recovered from mid-air rotor-failure scenarios that would have triggered an immediate crash under any hand-authored failsafe. The training pipeline never encountered those exact failure modes; it generalized from adjacent simulation experiences, a behavior no explicit if-then fault tree could have produced at this scale.

08 Beyond Software 3.0: The Additive Imperative

Software 3.0: Prompts as the Programming Language

The evolution of this paradigm took a significant conceptual leap when Andrej Karpathy formalized the next structural milestone at the Sequoia Ascent summit: Software 3.0. In this framing, the relationship between code and compute is entirely inverted.

If Software 1.0 was characterized by human-authored instructions and Software 2.0 was defined by data-curated weight binaries, Software 3.0 treats massive, frontier foundation models as a fixed physical processor. The model weights become the static CPU, the context window acts as the volatile RAM, and natural language prompting emerges as the programming language itself. Entire traditional application architectures—composed of complex databases, brittle API integrations, and human-written frontend scaffolding—collapse into a single block of declarative intent executed inside the context window.

The Subtractive Trap

Yet, even as Software 3.0 accurately captures the decline of traditional application architectures, its underlying economic thesis remains fundamentally subtractive. The core optimization driver of the Software 3.0 narrative is efficiency through reduction: minimizing human engineering overhead, shrinking the surface area of imperative codebases, and compressing traditional multi-layered web applications down to a single foundation model call.

It is a philosophy centered around doing what we already do, but with less friction, fewer files, and fewer engineers. It looks at our existing digital infrastructure and asks how much of it we can safely dissolve into the context window.

The core thesis of this essay rejects that subtractive boundary. We must pivot from a mindset of cost reduction to a framework of radical expansion. The true mandate of this technological epoch is not to use history's greatest computational unlock to optimize the familiar or clean up routine corporate backlogs.

Clearing out legacy boilerplate, optimizing standard e-commerce funnels, or building slightly faster internal tools is, frankly, boring. What is currently possible under our existing engineering paradigms represents a solved cognitive frontier. There is no historical glory, nor any genuine industrial progress, in simply automating the mundane and declaring victory.

The Additive Mandate

The true purpose of this tooling is additive: leveraging unprecedented generation velocity, trained behavioral dynamics, and machine-checked safety layers to engineer systems that were previously deemed structurally impossible. We must use this paradigm shift to turn our focus outward toward infinite complexity. When human talent is liberated from the mechanics of text generation, our baseline expectation for software must scale exponentially.

This additive framework unlocks domains of emergent logic that no human team could ever hope to code up by hand. We gain the capability to construct planetary-scale energy orchestrators that balance millions of intermittent green-grid nodes in real time. We can design polymorphic network routing fabrics that dynamically mutate their own machine code to bypass hardware failures at sub-millisecond latencies. We can build closed-loop, combinatorial molecular drug infusion controllers that safely navigate non-linear human biochemistry to treat critically ill patients based on real-time biometric feedback streams.

These are not marginal optimizations of existing applications; they are entirely new classes of infrastructure that could never survive the limitations of the legacy frontier. They require an architecture where code behaves like a biological organism—continuously morphing its internal state and memory profiles to match real-time telemetry—while remaining anchored to machine-checked formal verification layers that guarantee safety boundaries.

Imagination Is the Bottleneck

By shifting our engineering culture from coding to training, we stop acting as mechanics for an increasingly fragile and bloated digital infrastructure. We finally unlock the structural safety net required to expand our engineering horizons. The cognitive bandwidth of the human mind is no longer the hard ceiling of the digital age. The frontier has moved; as a result, our main remaining bottleneck is the scope of our own imagination.

We must stop rebuilding what we had in the past and begin constructing the impossible architectures of the future.

09 Grand Challenges for Unscriptable Software

To explicitly anchor this mandate, we must focus on specific engineering targets where manual coding or scripting is entirely useless, and where global funding syndicates are actively deploying capital to foster breakthroughs.

The "Unscriptable" Frontier

The "unscriptable" frontier is therefore not a slogan but a rephrasing of a familiar theorem: certain functions are unreachable from within the imperative medium, however diligent the author. Training, in this light, is a route around the wall that script-writing cannot breach—not unlike the way constructive mathematics casts non-computable truths as the limits of a particular formal language rather than the absence of underlying reality.

Challenge Problems

The three that follow were not chosen for shock value. Each clears the same bar: the underlying control problem is combinatorial enough that no human-authored rule set could cover it safely, and a funding agency — DARPA, NSF, the EIC, ARPA-H — is already writing grants against exactly this thesis rather than treating it as speculative. They span infrastructure, hardware, and medicine on purpose, to show that the unscriptable frontier isn't confined to any one industry.

Challenge 1: The Autonomous Heterogeneous Micro-Grid Orchestrator

Challenge 2: Sub-Millisecond Polymorphic Circuit Network Routing

Challenge 3: Real-Time Combinatorial Molecular Drug Infusion Control

None of these three problems yields to more code, and none of them is exotic — each is simply the next rung up from what agentic tools already handle today. What they share isn't the domain; it's the shape of the failure. Past a certain scale, the correct behavior can be trained but never fully written down, and the only way to trust it is to verify the boundary rather than read the interior. That is the same thread this essay has been pulling since the glass monolith first turned opaque — worth naming plainly before closing it out.

10 Conclusions

The glass monolith gave us two phases: small and transparent, or large and dark. We accepted that as a law of nature.

It wasn't. Opacity was a property of one medium — imperative text piled on imperative text until light could no longer pass through. Swap that medium for training, and swap the audit-by-reading for audit-by-proof, and the interior can stay dark without the system becoming unsafe. What holds a system together at scale was never whether a human could trace every gear; it was whether the invariants at the boundary still held.

None of this is hypothetical anymore. A virtual company of agents has already shipped working software with no human writing a line of it. An evolutionary search has already broken a fifty-six-year-old mathematical bound and gone on to run a hyperscaler's own data centers. A monocular-vision drone has already beaten three human world champions — not in a lab, in an open tournament.

The ceiling was never physical. It was staffing, and it was legibility, and both are dissolving faster than the industry's cargo-cult instincts can keep up.

That is the real unlock, and it is also why this essay insists on the word unscriptable rather than merely difficult. Not the velocity, not the effort compression, not the fact that an agent can rewrite a thousand lines in seconds — those were always going to arrive. What matters is that some of what we are chasing was never reachable by scripting at all: a boundary that predates agentic AI by ninety years, drawn the moment Turing showed certain functions have no algorithm. Training does not defy that limit; it routes around it, and stays auditable anyway — not by a person reading code, but by a proof that never sleeps.

This does not license a subtractive reading, where the same systems get built by fewer people at the same level of ambition. The obligation this essay opened with was moral, not just technical: use the compression in labor to expand what gets attempted, not to shrink who attempts it. That is the difference between an additive epoch and a subtractive one, and it is a choice — nothing about the technology forces one over the other.

Most of what passes for Software 2.0 in industry today is itself cargo-cult: more compute and more data thrown at a problem without the objective-function rigor Karpathy's framing actually demands — the same reflex this essay diagnosed as the Mythical Agent-Month. The road from there to a genuinely additive Software 3.0 is long, and it does not complete itself just because the tools got faster. But that road is where the real glory begins — not in trading one cargo-cult for a shinier one, but in finally building what the trade was supposed to be for.

Systems that balance a million green-grid nodes per millisecond, that titrate a drug infusion against live biometric telemetry, that reroute a polymorphic hardware fabric around failures before they propagate, that approach the two-billion-line scale of Google's own monorepo without a single engineer holding it in their head — these were never beyond our imagination.

They were beyond our medium. The medium has changed.

What's left is not a technical constraint. Our sole remaining bottleneck is the courage to honor that additive obligation — to point this much power at something worth building.

References

• Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., Xu, J., Li, D., Liu, Z., & Sun, M. (2024). "ChatDev: Communicative Agents for Software Development." ACL 2024. Available at: arxiv.org/abs/2307.07924
• Google DeepMind (2025). "AlphaEvolve: A Gemini-Powered Coding Agent for Designing Advanced Algorithms." Available at: deepmind.google

• Karpathy, A. (2017). "Software 2.0." Medium. Available at: karpathy.medium.com
• Karpathy, A. (2026). "Software 3.0." Sequoia Ascent Summit. Available at: karpathy.bearblog.dev
• Blondel, M., & Roulet, V. (2024). "The Elements of Differentiable Programming." arXiv preprint arXiv:2403.14606. Available at: arxiv.org/abs/2403.14606
• McKinney, W. (2026). "The Mythical Agent-Month." Wes McKinney's Personal Blog. Available at: wesmckinney.com
• Syme, D. (2026). "Understanding Repositories as Human/Agent Knowledge Factories." Available at: dsyme.net
• Kelly, K. (1994). Out of Control: The New Biology of Machines, Social Systems, and the Economic World. Addison-Wesley.

• Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, S., et al. (2022). "Outracing champion Gran Turismo drivers with deep reinforcement learning." Nature, 602(7896), 223–228. Available at: arxiv.org/abs/2202.04616
• Sony AI & Polyphony Digital (2022). "Gran Turismo Sophy Project Technology Overview." Available at: gran-turismo.com
• Gillberg, J., Bergdahl, J., Sestini, A., Eakins, A., & Gisslén, L. (2023). "Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games." Electronic Arts Research. Available at: ea.com/seed
• Electronic Arts SEED (2023). "CoG 2023: Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing." EA SEED Blog. Available at: ea.com/seed
• Electronic Arts SEED (2018). "Imitation Learning with Concurrent Actions in 3D Games." Available at: ea.com/seed
• Andelkovic, A., & Freyr, S. (2016). "How King Uses AI in 'Candy Crush'." Game Developers Conference (GDC) Vault. Available at: gdcvault.com
• Gudmundsson, S. F., et al. (2017). "Crushing Candy Crush: Predicting Human Difficulty Levels in Mobile Games Using Playtesting AI." KTH DiVA Consortium. Available at: kth.diva-portal.org
• InfoQ (2020). "How King Uses AI to Test Candy Crush Saga." Available at: infoq.com

• Razzaghi, P., Tabrizian, A., et al. (2024). "A Survey on Reinforcement Learning in Aviation Applications." Engineering Applications of Artificial Intelligence. Available at: arxiv.org/abs/2211.02147
• Kaufmann, E., Bauersfeld, L., et al. (2023). "Champion-level drone racing using deep reinforcement learning." Nature, 620(7976), 982–987. Available at: nature.com
• University of Zurich (2023). "Drone Racing: Autonomous Drone Beats Human Champions." UZH News. Available at: news.uzh.ch
• Bahnam, S. A., Ferede, R., Blaha, T. M., Lang, A. E., Lucassen, E., Missinne, Q., Verraest, A. E. C., De Wagter, C., & de Croon, G. C. H. E. (2026). "MonoRace: Winning Champion-Level Drone Racing with Robust Monocular AI." arXiv:2601.15222. Available at: arxiv.org/abs/2601.15222

• Ma, J., He, Y., Li, F., Han, L., et al. (2024). "Segment anything in medical images." Nature Communications, 15, 654. Available at: nature.com
• Wang, B., et al. (2024). "MedSAM: Segment Anything in Medical Images." GitHub repository. Available at: github.com/bowang-lab/MedSAM