$_ stdout

LLMs + Security = Trouble

The industry's default answer to insecure AI-generated code is more AI: an LLM checker hunting for bugs an LLM wrote, or an LLM attacker probing for the ones it missed. It's a pragmatic move, and it catches real bugs. It also shares a structural weakness with the thing it's checking.

This post argues that fighting fire with fire only ever secures the common case, walks through what the empirical literature actually shows about that gap, and traces a different path: instead of detecting and repairing vulnerabilities after the fact, enforce security constraints during generation itself.

BL Dr. Ben Livshits July 28, 2026 · 8 commits

In February 2026, Anthropic's own research team reported finding more than 500 high-severity vulnerabilities in open source software using Claude Opus 4.6 — "out of the box," without task-specific tooling, custom scaffolding, or specialized prompting. Some of the targets were codebases that had been fuzzed for years, accumulating millions of CPU-hours of prior testing. In GhostScript, Claude found a bounds-checking gap by reading Git commit history and noticing a similar check missing on a different code path. In CGIF, it caught a buffer overflow by reasoning about the LZW compression algorithm conceptually — recognizing that decompressed output could exceed the size of its input, not by pattern-matching a known bug signature.

Eight months earlier, an autonomous system called XBOW became the first AI to reach #1 on HackerOne's US leaderboard — ahead of every human researcher on the platform, having submitted more than a thousand reports in a few months, 54 of them critical. LLM-based tools are no longer marginally competent at finding vulnerabilities. In specific, well-documented ways, they're now better at it than most of the humans whose job this used to be.

That success is exactly why the field's default move — training or prompting an LLM to check the code another LLM just wrote — looks so appealing. If the technology is this good at finding bugs, why not point it at your own output? This post, adapted from a recent arXiv paper, argues that the appeal is real but the guarantee it seems to offer isn't: a probabilistic checker inherits the same blind spots as the probabilistic generator it's grading, and the gap between the two only shows up on the bugs that matter most.

01 The AI Security Triangle

Setting aside prompt injection and directly malicious models — real problems, but a different post — the pragmatic starting point for AI-assisted code security is a pattern that keeps recurring across the field, regardless of which vendor or paper you're reading. Call it the AI security triangle: three roles, played by some combination of LLMs, SLMs, and traditional tooling.

In production, these roles rarely stay cleanly separated. Saul, Wang, Sen, and Wagner's SCGAgent is a representative case: generator and checker become tightly interleaved inside a single agentic loop, where the same underlying model proposes a fix, evaluates whether the fix holds, and revises again. That interleaving is often a genuine improvement — SCGAgent lifts the rate of secure-and-functional samples from 61% to 76% on a Claude Sonnet 3.7 base. But it also means the triangle's three corners increasingly draw from the same trained distribution, the same training data, the same blind spots.

The AI security triangle: a generator produces code, a checker analyzes it, and an attacker probes it for exploits, forming a closed loop with a dashed repair feedback path back to the generator. All three roles are frequently implemented with the same family of large language models, which means they tend to share the same blind spots.
// Figure 1. Three roles, one shared failure mode: when generator, checker, and attacker are all drawn from the same model family, they tend to miss the same things.

None of this makes the triangle useless — it's the pragmatic default for a reason, and it catches a large share of ordinary bugs cheaply. The question this post is actually interested in is what happens at the edges: the rare, adversarially relevant vulnerabilities that a well-resourced, patient attacker goes looking for first, and that a checker optimizing for coverage under a fixed budget is structurally unlikely to reach.

02 Why Probabilistic Defense Doesn't Scale

Three separate arguments explain why "fighting fire with fire" tends to plateau well short of the vulnerabilities that matter most — not because the checkers are badly built, but because of what they're structurally up against.

State-Space Explosion

Large codebases have enormous state spaces and control-flow complexity, and every analysis technique — a traditional static analyzer, a fuzzing campaign, a test suite synthesized by an LLM, or an "AI red team" generating candidate exploits — runs into some version of path explosion. This is especially acute for exploitability specifically, because many real exploits correspond to exceptional execution paths that are intentionally rare in normal operation or testing: error-handling branches, corner-case parsing logic, unusual privilege boundaries, race windows requiring precise timing.

Defenders who run LLM-based detection tools or deploy an automated attacker are typically optimizing for coverage under a fixed budget — time, compute, tool calls, rate limits — which means the checker necessarily misses a large fraction of the exceptional behaviors where high-value bugs hide. Worse, a defender's own attacker is constrained to stay safe, reproducible, and non-disruptive: it cannot freely DoS a production deployment or take destructive actions the way a real adversary can. Real adversaries face no such constraint. They can focus narrowly on one target, iterate indefinitely, and exploit partial signal leaked by crashes, logs, or timing differences that a well-behaved defensive tool is built to ignore.

Absence of Evidence Is Not Evidence of Absence

This asymmetry means that even when an AI-based attacker succeeds in finding some issues, its failure to find a given exploit provides weak evidence of that exploit's absence. "Not found" often just means "not reached." Deploying an automated attacker doesn't eliminate the underlying risk — it merely changes which slice of the vulnerability space gets explored, and a different attacker, human or AI, can still succeed by searching a different region or investing more effort into exactly the rare paths that defensive testing is least likely to hit.

An attacker with a more powerful model, more powerful hardware, or insight that guides exploration of the codebase will often have the upper hand, and the sheer volume of code LLMs now produce only widens that opportunity.

Lack of Forward Security

AI-generated code may have very weak forward security: code produced a year ago may not withstand aggressive security hammering by today's top-line models, and code produced today may not withstand exploitation by the models in use a year from now. Not everything about this arms race is symmetric or public, either — a defender relying on publicly available models to identify vulnerabilities may be no match for an attacker running a privately developed model on more powerful custom hardware. The checker you deployed to grade last quarter's code is not the checker your adversary will be facing next quarter.

// the valley of probabilistic reasoning

He and Vechev showed that prefix tuning can raise the ratio of secure code produced by CodeGen-2.7B from 59% to 92% — a real, measured gain, and one of the strongest probabilistic steering results in the literature. But steering a model's distribution toward safer output is not the same as constraining what it's allowed to produce, and as later evaluations show, that kind of gain routinely comes bundled with a hit to functional correctness the raw security number doesn't disclose.

03 What the Evidence Actually Shows

None of the three arguments above are hypothetical. A growing body of empirical evaluation over the past two years puts numbers on exactly the gap they predict.

Agents Fix Bugs by Introducing New Ones

Zhao and colleagues' SUSVIBES benchmark evaluates coding agents on 200 real-world feature-request tasks drawn from open source projects — the kind of multi-file, project-level work "vibe coding" actually involves, rather than isolated function completion. Their headline result is stark: SWE-Agent running on Claude 4 Sonnet solved 61% of tasks functionally, but only 10.5% of those solutions were secure. Over 20% of the functionally correct solutions introduced brand-new vulnerabilities that weren't present in the original codebase at all. Simple mitigations don't close the gap either — the authors found that adding a generic security reminder to the prompt was largely ineffective.

Chong, Yao, and Neamtiu's earlier comparison of LLM-generated and human-written solutions across LeetCode and algorithmic tasks found the same pattern from a different angle: LLM-generated code contained 11.2% more security issues on LeetCode problems and 7.1% more on general algorithm tasks than human-written equivalents, frequently due to a plain lack of defensive programming — missing bounds checks, missing overflow protection.

One Static Analyzer Isn't Enough

Dai, Xu, and Tao's evaluation of four state-of-the-art secure code generation techniques found a genuinely disturbing trend: many of them significantly degrade the base model's functional performance — sometimes by more than 50% — while delivering only marginal security improvement in return. Part of the reason those improvements looked larger than they were is a measurement artifact the authors name directly: security overestimation from relying on a single static analyzer. Using three different analyzers and two LLMs for detection instead of one, they found that CodeQL alone misses more than 20% of the vulnerabilities present in generated code that a broader panel catches.

Put together, these results describe the same failure mode from three separate angles: a checker or a steering technique that looks like it's working, measured on the axis it was built to optimize, while the axis that actually matters — real functional correctness, cross-validated security — tells a less flattering story.

04 Beyond Probabilistic Guarantees: Neurosymbolic Reasoning

Neurosymbolic code generation is the most direct response to that gap: a hybrid approach that integrates the statistical pattern-matching of LLMs with the rigorous, logical structure of symbolic computation and programming-language theory, aiming to bridge the semantic gap inherent in standard probabilistic text prediction.

Astrogator

Councilman and colleagues' Astrogator targets Ansible, the infrastructure-as-code language widely used for system administration on critical systems. Its core innovation is a Formal Query Language (FQL) — precise enough for formal verification, but high-level and natural enough for a user to review and confirm it actually matches their intent. Astrogator translates both a user's formal query and the LLM-generated code into a novel State Calculus that tracks system-level changes like file creation or package installation, then uses a symbolic interpreter to compare the two behaviors and confirm they match.

A Knowledge Base maps high-level concepts — "an Apache server" — to platform-specific implementation detail, reducing how much specialized systems knowledge the user needs to bring. Evaluated across 1,260 programs spanning 21 tasks, Astrogator verified correct code in 83% of cases and correctly identified incorrect code in 92%.

ARC

Bayless and colleagues' ARC takes a two-stage approach: it uses LLMs, with optional human guidance, to formalize natural-language policies, then performs inference-time autoformalization to validate whether new natural-language statements are logically consistent with those policies. Where correctness is paramount, the framework runs multiple redundant formalization passes and cross-checks them for semantic equivalence, producing auditable logical artifacts that substantiate its verdicts rather than a bare accept/reject label. Crucially, ARC is designed to be conservative rather than confident: when it isn't certain a translation is correct, it flags the case as ambiguous instead of risking a false approval. That design choice pays off in the numbers — ARC exceeds 99% soundness, with a near-zero false-positive rate on logical validity.

Both systems show genuine strides toward stronger-than-probabilistic guarantees for LLM-generated code. But they share a catch this post keeps returning to from different angles: unless the entire verification pipeline is fully automated end to end, a developer still has to get involved — validating specifications, resolving ambiguities, adjudicating the significant percentage of cases that fail automatic verification without being definitively broken or correct.

Given the developer's propensity to "accept all changes" by default in the paradigm of vibe coding, this is not a match made in heaven.

The human-in-the-loop step isn't a minor UX wrinkle on top of an otherwise-solved problem — it's a likely point of weakness in exactly the workflow neurosymbolic verification is meant to secure. A verification gate that a rushed developer habitually clicks through offers little more assurance than no gate at all.

05 Security by Construction, but How?

If detecting and repairing vulnerabilities after generation inherits the generator's blind spots, and verifying them after the fact runs into the human-in-the-loop problem, the remaining option is to change where the guarantee gets enforced: not after the code exists, but while it's being written.

The obstacle is mechanical, and it's specific to how most code-generating models actually decode. During autoregressive decoding — one token, left to right — a security property can't be evaluated against the entire program, because only a partial program is available at any given step. If the partially generated code hasn't sanitized untrusted user input yet, that doesn't mean the finished program won't; the model simply hasn't written that part. Autoregressive generation makes it structurally impossible to evaluate — or enforce — a whole-program property until the whole program already exists, at which point you're back to detect-and-repair.

Autoregressive versus diffusion-style code generation. Autoregressive decoding: only a partial program is visible at any point during generation, so a constraint checker cannot yet confirm whether untrusted input will eventually be sanitized. Diffusion decoding: the whole program is visible and revised together throughout generation, so security constraints can be checked and enforced against the complete structure at every step.
// Figure 2. Non-autoregressive decoding generates the entire program together, which is what makes a whole-program security constraint enforceable during generation rather than after it.

Diffusion Models See the Whole Program

Non-autoregressive decoding changes that equation. Diffusion-style code models — which have drawn growing attention for their speed and their ability to exploit parallelism more effectively than autoregressive generation — produce and iteratively refine the entire program together, rather than committing to tokens one at a time in a fixed order. Because the whole draft is visible at every refinement step, a security constraint can be evaluated and enforced over the complete program throughout generation, not just checked once at the end.

That structural difference is what makes diffusion-style generation a natural fit for modular, hierarchical security enforcement — checking and constraining code at the level of individual AST nodes as the whole tree comes into focus together, instead of waiting for a linear stream to finish.

Constrained Decoding Already Works

The mechanism doesn't have to wait for diffusion models to mature — constrained decoding already demonstrates the underlying idea on today's infrastructure. Fu, Baker, Ding, and Chen's CodeGuard+ evaluation on CodeGen-2.7B is the clearest evidence so far. Unconstrained generation there scores 66.66% Pass@1 but only 43.17% Secure-Pass@1 — correct and secure, together. SVEN, the prior state-of-the-art prefix-tuning defense, trades functional correctness away to buy security: Pass@1 drops to 55.60%, for a Secure-Pass@1 of only 47.48% — barely ahead of doing nothing.

Constrained Beam Sampling, by contrast, hits 76.00% on both metrics at once — matching the unconstrained baseline's functional rate while substantially beating it on security, something no amount of prefix-tuning managed. The authors report that constrained decoding outperforms unconstrained GPT-4 on secure-and-correct code, using open-weight models many times smaller.

This blog covered that result, and the broader mechanics of grammar-constrained decoding, in more depth in a follow-up post — including where the simple version of "just constrain the grammar" breaks down on its own, and the state-of-the-art fixes for each gap.

The Small World Hypothesis for Security Fixes

Constraining generation at the level of an entire program sounds like it demands global reasoning about the whole codebase. It usually doesn't. Wang, Wang, Feng, Sun, and Jajodia's PatchDB — a dataset of roughly 12,000 real-world security patches — has a plethora of examples confirming that most vulnerability fixes, as opposed to the bugs they fix, can be made locally, contained within a single function call. This local-fix nature holds broadly for memory vulnerabilities like buffer overruns and double frees, and it holds for cross-site scripting and path traversal attacks too — an inserted bounds check here, a sanitization call there.

That locality is the structural opportunity diffusion-style generation is positioned to exploit. Because most security fixes are small and localized rather than diffuse across a codebase, the enforcement mechanism doesn't need to reason about the whole system at once — it needs hard, non-probabilistic constraints wired into the right AST nodes: modules, functions, loops, conditionals, array accesses. A proactive bounds check inserted at the point of an array access, or mandatory sanitization at the point untrusted data reaches an HTML sink, is exactly the kind of local, structural guarantee that a whole-program-visible decoding process can enforce directly, rather than hoping a downstream checker catches its absence later.

Paradigm Guarantee Human in the Loop? Fits Vibe Coding?
Fighting fire with fireProbabilistic, best-effortNoYes, but blind to the long tail
Neurosymbolic verificationFormal, if fully automatedYes — resolves ambiguous specsNot without one
Security by constructionHard, non-probabilisticNo, once constraints are specifiedYes — enforced silently

06 The Wider Field

The three paradigms above are the paper's own argument, not the whole literature. The underlying survey this post adapts runs considerably wider — Zhu and colleagues alone catalogue over 100 works on LLMs and software security. Two threads from that broader survey are worth pulling out on their own: systems still trying to catch what slips through detection, and one attempt to define security as a systems problem rather than a model problem.

Detecting What Slips Through

Hajipour and colleagues' HexaCoder takes a data-synthesis approach to the same gap this post has been describing: it uses a security oracle to generate matched pairs of vulnerable and repaired code, then fine-tunes on the result. The payoff is substantial — HexaCoder reduced vulnerable code instances by up to 85% across multiple benchmarks while preserving functional correctness, evidence that better training data narrows the gap detection alone can't close.

Two more systems show the same pattern applied to specific domains. Wei and colleagues' SmartAuditFlow decomposes smart-contract auditing into adaptive plan-execute stages instead of a fixed checklist, and in the process surfaced 13 additional CVEs that existing tools had missed. Zhu and colleagues' VulInstruct goes a layer deeper, extracting the implicit security specifications behind historical vulnerabilities — the assumptions about how code should behave that make a deviation count as a bug — and reached a 45.0% F1-score, a 32.7% improvement over the strongest baseline, while independently discovering a previously unknown production vulnerability, CVE-2025-56538.

Security as a Systems Problem

Christodorescu and colleagues take the widest lens of anyone in the survey: rather than asking whether any single model can be hardened, their systems-security framing treats an AI agent as an untrusted component inside a larger Trusted Computing Base, the same way operating-systems security has always treated any single process. Grounded in eleven real-world case studies of agentic attacks, the paper's throughline matches this post's own: hardening the model is necessary but not sufficient, because the security boundary that actually matters sits at the architecture level, not inside the weights.

On the generation side specifically, Zeng and colleagues' TreeDiff is the clearest working example of the diffusion argument from earlier in this post: instead of masking tokens at random, it corrupts training data along Abstract Syntax Tree boundaries, so the model learns to respect a program's structure rather than its raw token sequence. The result was a 13.3% relative improvement on HumanEval+ over standard random-masking diffusion — a small, concrete data point for the same claim the security-by-construction section makes about why AST-aware, whole-program-visible generation is the more natural place to enforce a security constraint.

07 Conclusions

The AI security triangle — generator, checker, attacker — is a genuinely useful pragmatic default, and nothing in this post is an argument against running it. It catches real bugs cheaply, and the evidence surveyed above shows agentic pipelines improving fast. But every corner of that triangle is typically drawn from the same probabilistic family, trained on overlapping data, and optimized under the same coverage budget — which means it reliably catches the common case and just as reliably misses the long tail an adversary is specifically looking for.

In order to create security guarantees, we need to have hard, non-probabilistic constraints that limit what code is allowed to be produced.

Neurosymbolic verification gets closer to that bar than detection alone, but only when the human reviewing its edge cases is actually paying attention — a condition "vibe coding" workflows are specifically bad at guaranteeing. The more durable fix — enforcing constraints during generation — doesn't remove any corner of the triangle; it changes when the constraint gets applied. A bounds check enforced during generation, at the AST node where an array access happens, doesn't need a checker to find its absence later, because its absence was never a reachable output in the first place.

That's a narrower promise than "secure code," and it should be read as one: constrained, secure-by-construction generation only guarantees the properties someone explicitly encoded as a constraint — a sanitizer here, a bounds check there. It won't invent a security property nobody thought to specify. But for the properties it does cover, the guarantee is structural rather than statistical, and that distinction is exactly the one the rest of this post has been arguing is worth the trouble.

References

On LLM-Discovered Vulnerabilities
On the Paper This Post Adapts
On the Prevalence of Vulnerabilities in AI-Generated Code
On Probabilistic Steering and Constrained Decoding
On Neurosymbolic Verification
On the Wider Literature
On Security-Fix Locality