The Bitter Lesson and the SOC Analyst, Part 1

In a 2025 interview, Simon Orcsik, who leads a SOC team, described his floor as a room full of "ticket monkeys" with no slack in the day: "you don't have time to train, you don't have time to drill." That floor is where AI is now being sold as the fix, with agents promised to work the queue the analysts cannot keep up with. Orcsik expects companies to take that promise as a reason to stop investing in their people, betting that the machines will do the work instead. He thinks the bet will fail: "they're not going to be right about that either."

The security vendors marketing AI analysts and the "autonomous SOC" say the analyst's queue will empty. Orcsik says the floor will not. Neither is right: the machines will take much of the queue, but not the judgment calls, and the people who make those calls still have to learn that judgment somewhere.

For three decades, defense assumed the attacker was slower than the defender's cycle of analyzing a threat, writing a rule and shipping a patch, because every step on the attacker's side took skilled human hours. That assumption has stopped holding. The Bitter Lesson is the cleanest way I know to ask whether the human stays at the center of the loop, because it predicts which parts of a human-built system will disappear first, replaced by AI. This post tests that prediction on the security operations center: which of its jobs, from writing detection rules to triaging alerts to deciding what an incident means, AI takes over, and which stay with people.

The lesson is bitter because it says our expertise was a stand-in for compute we did not yet have. It does not say that expertise was worthless, or that compute can replace all of it.

I start with the detection rules, then turn to the analysts who work the alert queue, and then to the vendors who claim their products can replace those analysts. Figure 1 is the argument in one frame.

01 The Push to Take the Human Out of the Loop

Cybersecurity has an old asymmetry: the attacker needs one exploitable defect, and the defender has to find, fix and watch every one of them across a surface that keeps growing. A common military rule of thumb says an offense needs roughly a three-to-one numerical edge to dislodge an entrenched defense. In digital conflict the ratio inverts. The attacker has to find only one flaw, and once it is found, using it against one more target costs almost nothing. The defender has to close all of them, and that work stays labor-intensive, centralized and reactive.

Reactive Defense Worked While Signatures Shipped Faster Than Attacks Spread

Reactive defense worked because of what I will call the signature window: the buffer between a zero-day's first use and the wide deployment of a rule that catches it. Defenders extracted the artifact, whether a binary, a script or a command-and-control address, reverse-engineered it, wrote indicators of compromise, encoded them in Snort or YARA rules, shared them through industry channels and pushed them to endpoints and firewalls. A window of weeks was fine, because a human attacker needed months to build a tailored kit, and a ninety-day patch cycle could keep up. Figure 2 shows the signature window twice: against a human attacker, and against one running at machine speed.

Machine-Speed Attackers Shrink the Signature Window Toward Nothing

Autonomous agents change that arithmetic. They can scan public attack surfaces, read source or decompiled binaries for memory-corruption bugs, test payloads in local sandboxes, and rewrite their evasion in response to what the defense does. A campaign that once needed a state's budget and months of effort can be run by thousands of parallel reasoning agents billed at utility rates. The window the defender relied on shrinks toward nothing.

What swarms of these agents have already done to real targets, and what happens when nobody is steering them, is in Agentic Swarms. A full agentic intrusion, from foothold to ransom note, is in The Algorithmic Extortionist.

The Signature Window Has Closed, but the Backlog of Old Flaws Has Not

One project has taken on the job of assembling the exploitation numbers. The Zero Day Clock is a public scoreboard for exploitation, built from nine public feeds: four KEV catalogs (CISA's, ENISA's EUVD, CIRCL's and VulnCheck's), Shadowserver's honeypot sensors, Microsoft's advisory feed, CVE records, NVD and EPSS. It is a campaign as much as an instrument, since it ends in ten demands and a list of signatories, so I use its numbers and keep its framing at arm's length.

Its timeline opens with Anderson's 2001 argument that insecurity is an incentive problem before it is a technical one, and carries on through Log4Shell in 2021, exploited within hours of disclosure.

Three recent entries bear on this section:

• Unit 42 found, as the clock records it, that 80% of public exploits appear before the CVE, on average 23 days ahead of it, with 14% arriving before any patch existed.
• VulnCheck counted 28.3% of exploited vulnerabilities weaponized within a day of disclosure.
• Anthropic's exploit-speed test turned 18 Firefox security patches into 14 crashing proofs of concept and eight working exploits, the first inside an hour, and 21 Windows kernel bugs into eight escalation chains at about $2,000 each.

On the same Windows set, an earlier model produced no complete chain at all, so the capability arrived between two releases. Figure 3 lays that history out.

The window closed; the backlog did not. In the week to 1 September 2026, Shadowserver's sensors recorded attacks against 920 known vulnerabilities whose median publication date was September 2022, and the project reports that 83% of every vulnerability its sensors have ever recorded was attacked again within the previous 28 days. Attackers now move within hours of a disclosure, yet flaws that have been public for years are still unpatched. Whether that flood justifies patching everything faster, and why the disclosure clock itself stopped holding, are the subjects of Patchmageddon? Or Not So Much and AI & The Death of the 90-Day Embargo.

One of the Zero Day Clock's ten demands cuts against this post: it treats human-in-the-loop regulation as handcuffs, because human oversight adds latency to a fight measured in minutes. This post asks that question from the analyst's side, and nothing here settles it.

Human Attention Is the One Part of the SOC That Cannot Scale

The flood lands on the one component that cannot scale, which is human attention. A modern enterprise produces billions of log lines a day across endpoint agents, identity providers, cloud control planes, firewalls and container orchestrators. SIEM correlation rules thin that into thousands of alerts, and most of them are benign anomalies, misconfigurations or noise. What reaches the analyst is alert fatigue. Figure 4 follows that funnel.

The funnel narrows billions of events to thousands of alerts, but it still ends at one person, so every alert that the rules let through costs that person time. And the three failures fanning out from the desk, burnout, fading vigilance and buried threats, are limits of human attention rather than of tooling, which is why another rule or another dashboard does not fix them.

The human cost is documented, and the older evidence is the more solid. Since Mackworth's clock experiments in the 1940s, the vigilance literature has shown detection performance falling off within about 20 to 30 minutes of continuous monitoring. Analysts copying hashes, cross-checking reputation engines and filling in tickets for hundreds of alerts a shift are being asked to do the one thing people do badly. Real positives get buried under false alarms. I come back to what analysts say about their own work in Part 2, after weighing the case for and against handing the queue to machines.

Overloaded analysts missing real attacks in a flood of false alarms is why leadership is reaching for full automation: SOAR playbooks, generative copilots and end-to-end agentic triage, with the autonomous SOC as the stated goal. It looks like the obvious conclusion. Whether it is the right one depends on whether defense is the kind of problem that scale solves, and Sutton is the place to start.

02 The Bitter Lesson Applied to Defense

Sutton's essay, published in 2019, summarizes seventy years of AI history in one observation.

Researchers repeatedly build systems that mirror how people think about a problem, with domain knowledge and heuristics embedded by hand. These win early, because human insight prunes the search space. In the long run they lose to general methods that exploit compute: search and learning.

His examples are computer chess and Go, where search and learning beat encoded human strategy; speech recognition, where statistical methods beat phonetic knowledge; and vision, where handcrafted edge detectors and SIFT features gave way to convolutional networks.

It is bitter because researchers want their intuition to matter. Security has the same instinct, with a thirty-year track record behind it. We hand-wrote regular expressions, modeled explicit protocol state machines, defined atomic behavioral heuristics, and built decision trees that copy how a senior analyst takes an attack apart. Every classic detection product, from IDS signatures to spam filters to endpoint heuristics, is an externalization of somebody's expertise.

Foundation models trained on huge corpora of code, system logs and technical discussion are now beating that stack on tasks it was built for. Figure 5 compares them across the dimensions that matter.

In Misuse Detection, a General Model Beats Every Guardrail Tested

The cleanest test I know of is in prompt-misuse detection, and it is narrower than the lesson's reputation. Mariaccia et al. benchmark specialized guardrails, products that sit in front of a model and screen incoming prompts for jailbreaks and misuse, against general frontier models that are simply prompted to flag harmful inputs, across jailbreak and misuse datasets, scored with a metric they call BELLS. Their title is the claim: the bitter lesson of misuse detection is that a general model, used zero-shot, beats every specialized guardrail tested; those guardrails are the amber bars at the bottom of Figure 6.

The margin varies: the weakest of the six general models tested, Mistral Large at 0.825, edges NeMo by only 0.005. GPT-4 scored 0.926 at a 1.5% false-positive rate. NVIDIA's NeMo Guardrails, which is built on GPT-3.5, scored 0.820 at 18.9%, and Meta's Prompt Guard scored 0.646. Figure 6 sets the numbers out.

Obfuscated payloads and abusive prompts are semantic phenomena. A rule written against syntax fails the moment the adversary changes the syntax and keeps the meaning, and a model that reads the whole context is checking the meaning.

Each Detection Rule Buys a Short Lead Before Attackers Route Around It

Detection engineering shows the loop in slow motion. Attackers abuse living-off-the-land binaries, tools that ship with Windows and are signed by Microsoft, so running them raises no alarm. certutil.exe exists to manage certificates, but it will also download a file from a URL and decode base64, which makes it a ready-made payload fetcher. rundll32.exe exists to run functions inside DLLs, so it will just as readily run an attacker's DLL under a trusted process name.

When attackers turned to these binaries, defenders wrote detections for specific flag sequences and parent-child process pairs. Attackers reached the same effect through other APIs, renamed binaries, or injected threads straight into legitimate processes. Each new rule bought a short lead and the cycle repeated.

Sutton's answer is to stop the cycle rather than win the next round of it. Feed wider, raw streams of endpoint telemetry to a model that reasons across the graph of operations and flags deviation from the enterprise's own baseline. Nobody has to anticipate how an attacker chains PowerShell, WMI and RPC.

If the lesson carries over from prompts to endpoint telemetry, the analyst's traditional jobs, writing signatures, authoring rules and adjudicating alerts, are structurally obsolete, and the question becomes which parts of the analyst's job a model cannot take over. The limit of that move, and what building security into generation instead would take, is the argument in LLMs + Security = Trouble.

03 The Strongest Case for Autonomy

If the lesson holds everywhere in cyber operations, AI ends up running all of defense. The strongest version of that argument deserves a fair statement before I argue against it. It rests on four pillars, and I think each is partly right:

1. Speed: machines investigate and contain in seconds what takes an analyst most of an hour.
2. Precedent: a machine has already found and patched flaws in software it had never seen.
3. Removing bug classes: translating and proving code could eliminate whole classes of vulnerability before runtime.
4. Economics: software scales at a predictable price, and people do not.

The first pillar is speed. Humans investigate sequentially. Decoding a base64 command line, reading a memory dump, correlating proxy logs and querying threat intelligence takes an experienced Tier 2 or Tier 3 analyst, the specialists above the first-line triage queue, a good part of an hour. During an automated lateral intrusion that latency is fatal. A cluster of inference agents handles many alert trees concurrently, deobfuscates scripts and maps behavior to MITRE ATT&CK in seconds, and its ten-millionth log line gets the same attention as its first.

Field evidence points the same way, with caveats. In Microsoft's randomized trials of its Copilot for Security, security novices were 26% faster and 44% more accurate on multiple-choice questions, and professionals 22% faster and 7% more accurate, on lab tasks, though the authors are Microsoft researchers evaluating Microsoft's product. Mandiant puts global median dwell time, how long an intruder goes undetected, at 416 days in 2011 and 10 days in 2023, then 11 in 2024, so the trend is real but not monotone.

Sub-second containment is the last step, and proponents argue it needs the human approval cycle removed.

The second pillar is precedent: in 2016 DARPA's Cyber Grand Challenge, the first all-machine capture-the-flag, had systems find vulnerabilities in unknown binaries, prove them exploitable and patch the binaries while preserving service; ForAllSecure's Mayhem won. What it solved was what the next section calls the orchestration problem. The cognition problem it exposed is what a decade of frontier-model progress has since narrowed: models now read source semantics and synthesize patches.

The third pillar is the most ambitious. In the reactive model, systems sit in latent failure and defenders patch on a cycle that leaves weeks of exposure. If models can translate memory-unsafe C and C++ into Rust and generate proofs that rule out whole vulnerability classes, the battlefield shrinks before runtime. At that point, the argument goes, nobody needs to triage attacks on defects that no longer exist.

Figure 7 puts the first three pillars together: the lifecycle the argument implies, and reactive patching beside the provable immunity it would replace.

I like the third pillar more than the others, and I have written about the translation side in Automating C to Safe Rust Translation. The enforcement side of it, and why shallow wrappers are not invariants, is in Enforcing Security Invariants in the Post-Zero-Day Era.

The fourth pillar is economics. ISC2's 2025 study of 16,029 practitioners found that almost all of them, 95%, need at least one skill they cannot find, and 59% call that shortfall critical or significant. AI tops the list of skills needed, at 41%. The people who would do this work are scarce, and getting scarcer.

Personnel is the largest line in most security budgets, and software offers near-infinite marginal scale at a predictable price. Replacing Tier 1 and Tier 2 with agents is, to a CFO, simply efficient.

04 Why Autonomy Stops Short

The autonomy argument assumes that security is a closed-system optimization problem like chess, Go or image classification. It is not. Security is an open-world game against opponents who can often read the defender's code, probe it as a black box when they cannot, and change the rules on purpose. Sutton's examples all came from domains where the rules stood still.

That does not put whole jobs out of AI's reach. Telemetry, analysis, response and enrichment each have a part that scales with compute and a part that does not. Four limits mark where the line falls, and I set out each one as a pair: where AI holds, and where it stops.

Learned Detectors Break Because the Adversary Controls the Data They See

A learned detector assumes training and deployment data come from the same distribution. Security breaks that assumption on purpose.

Figure 8 shows both failures.

A defense stack made only of models therefore needs someone watching it, who questions the model's conclusions and applies judgment when the AI calls a suspicious sequence benign. What enforcement outside the model looks like, and which frameworks actually enforce it, is in Runtime Guardrails, Revisited.

AI Handles Orchestration but Falls Short on Cognition

The second limit separates two kinds of problem, a split Jason Wright of Idaho State University draws in his look back at the Cyber Grand Challenge, accepted at IEEE Security & Privacy.

Wright's reading of the decade since is that language models have narrowed the gap on source code, where names, types and structure remain, while binary-only reasoning and exploitation are still open. I would add unfamiliar protocols and custom hardware to that list, though I have not seen a controlled comparison to cite for them.

Valid Requests Can Still Be Attacks

Sundaramurthy and colleagues' fieldwork in SOCs shows that investigations lean on organizational knowledge held in long-serving analysts' heads, not only on the written runbook.

Practitioners describe the current version. In Thurner and colleagues' interview study of 25 practitioners across 17 organizations, SOC manager M6 describes an LLM that "kept flagging our nightly backup jobs as suspicious," because it had "no understanding of our environment or normal processes." The same knowledge is what walks out the door when people do: a 2026 job-design paper notes that when burned-out practitioners leave, institutional knowledge departs with them.

Business-logic flaws are where this bites hardest. Skipping a payment step by editing a parameter, changing an ID to read someone else's invoice, or moving pennies across millions of synthetic accounts: each is a perfectly valid request. Figure 9 puts one beside the exploit a detector was built to catch.

In the log, account 8842 requests /account/invoice?id=8841 and gets a 200 OK at an ordinary response time. The session is valid, the token is signed, and the response is a well-formed invoice. Whether that is an attack depends on facts outside the request.

Past Data Cannot Explain a Novel Attack or Its Motive

Security is also an intelligence discipline. Behind the tooling are people with economic, espionage or geopolitical goals.

Threat hunting stays human-led for that reason: the analyst forms a hypothesis from context and directs the tools to find the evidence.

Practitioners put the limit in the same place. In the interview study, analyst A4 says a model "can puzzle well, but in the end does not know what the motive is." Another participant says APT groups use tools no one has seen, so there is "no data that the LLMs could use." Novelty and motive are where there is nothing to scale over.

05 The Analyst, Measured

The people who work in a SOC have been telling vendors, year after year, what the job is like while the machines arrive. Three survey series give the best record I know of: (1) Tines and (2) Swimlane, both automation vendors, and (3) the annual SANS SOC Survey. I read them as questions rather than as answers: how much burnout the numbers show, whether adoption changed the load, what the newest survey says automation costs those professionals who are still learning the job, and what leaders see that analysts do not.

Figure 10 lists who ran and paid for each study behind the analyst evidence, what it measures, and on what sample. This section rests on the three survey series, two of them run by automation vendors with a stake in the answer. Part 2 turns to academic work: the unsponsored forum study, interview study and query log in The Voice of the Analyst, and a preliminary job-posting paper in The Oversight Paradox.

Tines and Swimlane sell automation, so their surveys frame manual work as the problem their products solve. SANS does not sell it, but its 2025 edition was sponsored by Elastic, it does not measure burnout, and I did not see random sampling described in either SANS edition, so its percentages are indicative. Surveys repeated year over year are valuable in themselves, but none of the samples are matched from year to year, so differences between editions of the same survey are suggestive, not trends, and the results so far are less than conclusive.

Most Analysts Report Burnout, Few Feel It Often

The burnout number quoted most often in this space comes from Tines, an automation vendor that has run an annual SOC survey since late 2021. Its 2022 edition, a Pollfish survey of 468 US analysts, found 71% reporting some level of burnout, and that figure is still being repeated, including in Tines's own 2025 summary. It is a dated snapshot.

The 2023 edition surveyed 900 professionals, leaders included, across the US and Europe and found 63%. The 2026 edition, 1,813 respondents surveyed by Sapio Research with a margin of error of 2.3%, reports that 76% experience emotional exhaustion, reduced motivation or mental fatigue frequently or occasionally. Only 26% say frequently. Figure 11 sets the three editions quoted here side by side, marking the measures whose wording changed.

The editions do not form a clean trend, and I would not read one into them. Each changed who was asked: analysts only in 2022, then leaders and Europe from 2023, then a global sample twice the size. The 2026 question asks about exhaustion "frequently or occasionally," not "some level of burnout."

Workload Keeps Rising, and Burnout Tracks Staffing

The load holds across all three editions. The share saying workloads had risen went from 60% to 81% and stayed at 81%. Intent to switch jobs eased from 64% to 55% in 2023, and the 2026 report does not ask it.

On manual work, 64% of analysts spent over half their time on tedious manual tasks in 2022 and 25% did in 2023, which Tines read as progress. The 2026 report measures something else, an average of 44% of time on manual or repetitive work that could be automated, so I cannot say whether it fell further.

Satisfaction and burnout coexisted. Job satisfaction in that edition rose from 88% the year before to 99%, and 96% felt fairly paid, yet 63% reported burnout. Tines's chief executive, Eoin Hinchy, summed up the first half of that result in his foreword to the report:

"Overall job satisfaction in the SOC remains high — security practitioners love the work they do."

Burnout tracked staffing. Of respondents who felt understaffed, 79% were burned out, against 47% of those who felt they had the right amount of staff. Analysts like the work; what wears them down is doing it short-handed.

Most SOCs Use AI, but Only 36% Have Built It Into a Defined Workflow

Only the 2026 edition speaks to adoption, and its headline numbers disagree.

Among teams that use 75 to 99 security tools daily, 47% say they frequently experience burnout, against 26% overall, and the report warns that "simply bolting on additional tools to existing workflows can contribute to burnout." Heavy workloads are the leading cause at 39%, ahead of incident-response stress and repetitive tasks at 26% each. Kinsella's foreword concedes that AI adoption "hasn't yet translated into meaningful reductions in manual work." Tool sprawl is a quieter cousin of the oversight problem Part 2 takes up: more machinery means more for the same few people to supervise.

SANS runs the longer record on that question. Its SOC Survey, written each year by Christopher Crowley, reached its tenth edition in 2026 with 444 qualified responses from people working in monitoring or security operations, plus a parallel instrument for 69 CISOs and senior leaders. The 2025 edition, sponsored by Elastic, found that about 40% of respondents used AI or ML tools that were not part of a defined SOC workflow, and that 42% of SOCs ran them out of the box with no customization. It also put AI and ML at the bottom of its satisfaction rankings.

SANS fellow Seth Misner, in that report's commentary, worried that "leadership may see AI as a shortcut to fill staffing gaps, instead of investing in the talent and thoughtful integration of AI needed for substantive SOC improvement." By 2026, 79% of respondents used AI or ML tools, up from about 69% in 2025, but only 36% had built them into a defined SOC workflow, up from 29.5%.

Crowley's reading in the 2026 report is that analysts reach for AI "individually, often without organizational structure around how they are used, validated, or governed." The report puts its sharpest line plainly:

The biggest AI risk in the SOC is not a bad tool. It is an analyst who trusts one.

Its closing section adds that "the risk is not that AI performs badly. The risk is that no one will know when it does." I would call that the same oversight problem, stated by a practitioner survey before the argument has been made. Figure 12 puts the two editions of the integration question side by side.

AI Frees Up Analysts' Time but Narrows the Path Into the Job

The third survey asks a question the other two miss. Swimlane's 2026 report, The New SOC Career Ladder, polled 500 security professionals and leaders in the US and UK who use AI. Capacity is rising: greater capacity is one of the two biggest effects for 47%, 43% spend less time on repetitive work, 35% have more time for complex threats, and 88% say AI has improved their job satisfaction.

The cost turns up in the same survey, and the respondents report it against their own interests: 24% say AI has limited their ability to build skills, and 47% expect a steeper path into the profession, split between 37% expecting higher requirements for entry-level roles and 10% expecting fewer openings for juniors. Asked what the new roles look like, 41% predict work centered on validating and orchestrating AI, and only 1% expect the career path to stay as it was. Figure 13 sets the gain and the cost side by side.

The report's own executives read it the way this post does. "The issue is not whether analysts should trust AI," says Swimlane's CISO, Mike Lyborg. "It is whether they have enough information to decide when to trust it." Its CEO, Cody Cornell, supplies the mechanism: "The challenge is that routine work has also been one of the ways analysts learn the fundamentals."

The respondents still trust themselves, at least on paper: 92% say they could identify an incorrect or incomplete AI recommendation, and 48% would fall back on their own judgment when one conflicts with the evidence. Swimlane sells security automation, so it has an interest in the capacity half of that answer; the apprenticeship half is not a number a vendor needs to publish.

The same sample is more optimistic on a different question: 86% of Swimlane's 2026 respondents feel positive about AI's effect on their careers. That optimism is about whether the job will exist; the skills cost in Swimlane's numbers is about who will be able to do it.

Leaders Say Management Attends to Staffing; Practitioners Say It Does Not

The surveys suggest that management mostly does not notice the skills cost. The cyber leaders module in the 2026 SANS survey, new this year, adds something Tines does not.

Practitioners rank meaningful work first for the third year running, ahead of career progression, training and pay, which has fallen to fourth. Tines's 2026 report shows a similar leader-practitioner gap: leaders rank automation that cuts manual work first for retention, at 40%, while practitioners rank work-life balance first, at 38%. Figure 14 sets out the SANS side: the gap in perceived management attention, and what practitioners say retains staff.

The SANS findings explain why analysts' workload keeps rising even as AI use spreads. AI has entered the SOC faster than the workflow around it has adapted, and the people who would validate its output are the ones management is least attentive to. The interview study at the center of Part 2 adds the same caution in the practitioners' own words: its CISOs ask who is accountable when a model errs, and its analysts doubt the models get past Tier 1 work.