$_ stdout
~/posts/bitter-lesson-part-2.md · on main · 29 min read · part 2
/**
* @topic#bitterlesson #soc #aisecurity #humanfactors #cyberdefense
* @statusdraft
*/

The Bitter Lesson and the SOC Analyst, Part 2

This is Part 2 of a two-part post. Part 1 found that Sutton's Bitter Lesson is already retiring signatures and rules, that the case for handing the loop to the machine is stronger than its critics admit, and that autonomy still stops short where the opponent or the context draws the line. Its surveys showed the load on the analyst never lifted.

Part 2 tests the human judgment case against the people doing the work and the people selling to replace them: 892 coded practitioner posts, 25 interviews across 17 organizations, 3,090 logged analyst queries, and the claims of a 123-vendor market. Neither full autonomy nor human-led defense holds up on their own. Scale takes the computable half of the job, and judgment stays human, but judgment is learned in the work the machine is taking over, so the division ultimately holds only if the apprenticeship behind it is rebuilt on purpose.

BL Dr. Ben Livshits October 29, 2026 · 108 commits
// start with part 1

This is Part 2 of a two-part post. Part 1 covers why reactive defense worked, what the Bitter Lesson predicts for detection, the strongest case for autonomy, where it stops short, and what the surveys measure about the analysts who work the queue. Part 2 picks up from there: what analysts say about their own work, what the vendors are selling, and why the division of labor holds only if the apprenticeship behind it is rebuilt.

Figure 1, the same image that opens Part 1, is the argument of both parts in one frame.

Cover illustration for the post, summarizing its argument: hand-written signatures and rules retiring as compute-scaled methods take over, the resulting alert queue landing on one analyst, and the judgment and accountability that remain human.
// Figure 1. Both parts in one image: hand-written rules giving way to scale, the queue that lands on one analyst, and the judgment that stays human.

06 The Voice of the Analyst

A survey measures a population, and it is written by whoever sells the fix. It cannot say what practitioners write when nobody is asking, what they say when someone does, or what they say out loud on podcasts and panels. For that I use the two studies that have nothing to sell — (1) a coded analysis of 892 practitioner posts on Reddit's security forums and (2) 25 interviews with SOC staff across 17 organizations — and, separately, the practitioners talking to each other.

Online, Practitioners Praise AI's Speed and Distrust Its Reliability

The first is a SOUPS 2026 analysis of 892 posts that practitioners wrote on Reddit's security forums between December 2022 and September 2025, hand-coded across tools, use cases, opinions and adoption. Its clearest finding is a split it can put numbers on: practitioners praise the efficiency of the tools 54 times against 10 complaints, and the reliability 3 times against 55. Adoption kept rising through the study window, from about 27% of relevant posts describing active operational use to 58%, while the share refusing the tools did not fall at all. Figure 2 counts the split by factor.

Two stacked rounded panels of diverging bars for coded forum opinions by factor, from a SOUPS 2026 analysis of 892 posts about LLM tools. Negative counts run left from the center axis and positive counts right: capabilities 97 and 174, efficiency 10 and 54, cost 30 and 5, autonomy 41 and 6, security and privacy 46 and 6, reliability 55 and 3. The top panel holds the two net-positive factors; the bottom holds the four net-negative ones. Footnotes note counts rather than percentages, that a post can raise more than one factor, and that the forum sample skews toward practitioners who engage with new tooling.
// Figure 2. Forum opinions on LLM tools counted by factor, negative views to the left and positive to the right.

Asked Directly, Practitioners Worry About Over-Reliance and Lost Skills

The second study, the one quoted in the section on why autonomy stops short, asked; it did not observe. Thurner and colleagues interviewed 25 practitioners across 17 organizations, including seven SOC analysts, four CISOs and three red teamers, and had each of them draw the SOC they wanted before asking what stood in the way. Two limits apply: (1) the participants were screened for prior LLM use, so this is a sample of adopters, and (2) most work in Western Europe.

The interviews match the forum analysis above. Models were valued for report automation, threat-intelligence summarization and context enrichment, and rated not yet feasible for incident analysis. Of the five risks the researchers asked them to rate, over-reliance ranked highest, ahead of hallucination.

Participants placed the cause less in the models than in their own environments. The paper reframes the question from whether an LLM is capable to whether the SOC is mature enough to deploy it. E7 cites "the inability of the SOC to feed the LLM with the correct data," and R3 says "you can train a perfectly good model, but if you're giving it bad data, then that won't go well." Scaling the model does not touch that.

Figure 3 collects their words under the study's own participant labels, beside the forum study's excerpts, which its authors paraphrase to protect the posters, so those rows are their rendering, not the analysts' exact words.

25-interview study · verbatim
A4, SOC analyst"If your ticket system doesn't work and you build an LLM on top of it, you're just creating chaos."
E7, security engineer"The pressure to adopt is immense, and we are trying to keep up with the pace of change… There's no turning back now."
M6, SOC manager"They start trusting the LLM… they skip the review, because it's easier, right? And then they skip it again and again. Habits shift, and in the end, the LLM makes more and more decisions, and we are not even noticing it."
M2, CISO"If an LLM makes a mistake, who is responsible? The analyst? The SOC manager? The CISO? The vendor?"
M3, CISO"In a SOC, there's no safe environment to see what happens. If the model gets something wrong and an analyst blindly trusts the response, that could mean a breach slips through… You don't get a second chance to patch the mistake."
A7, SOC analyst"LLMs can't detect alerts they weren't trained on, so it's really just handling the Tier 1, repetitive stuff."
R2, red teamer"If something goes wrong, it can go seriously wrong. Therefore, LLMs should probably lean toward false positives, leaving the final verification to humans."
Forum analysis · paraphrased by its authors
Poster, tier-3 analystCut mean time to triage from around 45 minutes to under two, and attached the rule that came with it: trust, but verify.
PosterReported spending nearly four times as much effort correcting the model's behavior as addressing incidents.
PosterGave up on the tools for security work after watching a model invent details about a piece of malware the analyst had named himself.
PosterArgued the fit fails at the level of the task: security is mostly outliers and anomalies, which is what these models handle worst.
Poster, SOC managerStill waiting for the relief he expected; his team spends a large share of its time on tier-1 and tier-2 work that should have been automated by now.
PosterThe best one-line policy in the 892 posts: use it like an intern, let it gather information and raise its hand when it notices something, and do not let it touch anything critical.
// Figure 3. Voices from the two studies, grouped by provenance: the interview transcripts verbatim, the forum excerpts as their authors paraphrased them.

The interview study also names the human factors behind over-reliance, which the oversight section takes up by another route. A4 put the long-run version as a question about maps: "How many people can still read a conventional map nowadays?"

Analysts Ask the Model to Explain, Rarely to Decide

A third kind of evidence is not testimony at all. In an observational study, Singh and colleagues logged 3,090 queries from 45 SOC analysts over ten months and found that only about 4% asked for a binary judgment such as "is this malicious?" Most of the rest were about understanding: explaining an obfuscated command line, decoding an unfamiliar script, summarizing telemetry, drafting text. Analysts took the speed and kept the decision.

Practitioners Talking Shop Expect the Analyst Role to Widen, Not Disappear

The last source is the one with the least apparatus: practitioners talking, on podcasts and panels, to other practitioners. I read the transcripts of a dozen videos from 2025 and 2026 and kept eight. Figure 4 collects most of the quotes I kept.

Talks and podcasts · seven of the eight videos, 2025 to 2026 · auto-captions, not checked against audio
Hubbard, SANS instructor"…I still have to know how to do it, right?"
Saban, agentic SOC founderAlert fatigue becomes "agent results fatigue."
Riot Games, SOC manager"I feel like it is removing people's ability to critically think…"
Markel, SOC directorWants remediation "a human can account for, be able to speak to and defend."
ZoomInfo, senior director"…we're just going to trust a computer that's known to hallucinate all the time over a human"
Covington, BHIS SOC"AI makes good analysts great, but it will not make a bad analyst good."
Orcsik, SOC team leader"…you don't have time to train, you don't have time to drill."
Maxwell, security operations leader"Where do those tier two and three analysts come from if they weren't tier one analysts first…"
Strouse, cybersecurity recruiter"You can't get to mid-level or high level if you don't have an entry level."
Al Taie, cybersecurity manager"The tier one analyst as a conclusion can produce tier three analyst outcome with the AI."
// Figure 4. Ten lines from seven of the eight talks and podcasts, the three panelists of one session counted separately.

Start with someone who trains analysts rather than selling them tools. John Hubbard, a SANS instructor who has spent his career in SOCs, wrote in May 2026 that the question he is asked most, what happens to the analyst role, "has been haunting me for a while," and that he now puts it to nearly everyone he meets.

His answer is neither the empty desk nor the status quo: the roles widen; they do not disappear. Analysts become generalists approving the output of several specialized agents, and the skill that keeps them there is the one this post keeps returning to. "You still have to be able to verify whether what the AI did was correct," he writes, which is why he tells analysts to keep their deep technical skills sharp. Hubbard arrives at the same division of labor the closing sections argue for, from a classroom.

// how the transcripts were collected

Every quote comes from YouTube's auto-generated captions, checked against the transcript text and lightly cleaned — the captions render "SOC" as "sock" — but not against the audio, so read them as close paraphrase.

Three of the eight are hosted by vendors or a managed-security provider, and one guest is a vendor founder. I say which below.

Hubbard also hosted Oren Saban on the SANS Blueprint podcast in July; Saban formerly worked on Microsoft's Security Copilot and now builds an agentic SOC platform, so his answers are a vendor's. Saban names the failure mode: alert fatigue becomes "agent results fatigue", and Hubbard's worry is that "people get so used to it being right that they just completely pass over it."

A panel hosted by Expel's CEO in September, again a vendor's microphone, put three working SOC leaders on the stage, from Riot Games, Markel and ZoomInfo. All three would automate triage, and none would hand over containment or detection engineering. Riot's manager wants "a human to be able to own that mistake," and Markel's director wants remediation "a human can account for, be able to speak to and defend."

// beyond consent

Two of the three went further. ZoomInfo's director asks why SOCs train juniors into "a strict playbook that gives them one option and no thought" and then trust "a computer that's known to hallucinate all the time over a human." Riot's manager put it as the sentence that stays with me: "I feel like it is removing people's ability to critically think, and I think that people are becoming too dependent on it as a whole."

One more voice comes from the floor. Hayden Covington, who works in Black Hills Information Security's SOC, says AI added alert volume at first, so that "instead of triaging alerts, you are now triaging and debugging this tool," and that it makes good analysts better without rescuing bad ones; his prescription is to "train your humans first and then give them access to AI tools," which also happens to be what his employer sells.

Two voices go straight at the pipeline, the progression from Tier 1 to Tier 2 and Tier 3. Rob Maxwell, a security operations leader formerly at Chime, speaking on a podcast hosted by Dropzone AI, asks the question this post has been asking: "Where do those tier two and three analysts come from if they weren't tier one analysts first?"

Pete Strouse, a cybersecurity recruiter, sees the hiring side of it, with companies asking "what do I need a junior person for" once a senior has automation, and warns that "you can't get to mid-level or high level if you don't have an entry level." That is a market observation, not SOC experience, and it is the one piece of hiring-side evidence here.

By their own account, then, the tools have arrived and the work has not got easier; in one survey a quarter of the respondents already report the skills cost. That sets up the next question: what the industry has built to sell this automation, and what it promises.

07 What the Vendors Are Selling

An industry has formed to sell that automation, and the offers are concrete enough to examine. Three things matter: how many vendors there are, where their products stop, and what evidence stands behind the claims.

123 Vendors Sell Seven Different Bets Under One AI SOC Label

KuppingerCole counted 123 AI SOC vendors in August 2026 — thirty-nine in its April leadership assessment and eighty-four more under screening — and separates seven different bets hiding under one label: (1) a copilot bolted onto an existing SIEM or SOAR, (2) case management run by agents inside SOAR, (3) autonomous triage-and-close, (4) investigation support without automation, (5) knowledge-graph-first architectures, (6) continuous hunting, and (7) detection engineering upstream against MITRE ATT&CK. Its expectation is that about twenty of the 123 remain independent vendors by 2030.

The funding followed the same pattern: (1) 7AI closed a $130M Series A at a $700M valuation, (2) Exaforce $125M at $725M a year after its Series A, (3) Torq $140M at $1.2B, and (4) Zscaler bought Red Canary for $675M. One market-research house sizes the category at $18.1B this year, growing to $47.1B by 2031, which is a number to read as ambition, not revenue.

Pricing splits by vendor type. The platform vendors publish theirs: Microsoft by compute unit, Wazuh by agent count, Elastic by ingested gigabyte plus a charge per workflow and per agent run, IBM by token. Every pure-play startup asks you to request a price. The counterargument, that dozens of small local models can do the triage work for less, is in Cheap, Fast, and Local.

Every Vendor on the Map Claims Alert Triage, and Every Mark Is Self-Reported

Sixteen of the 123 appear in Figure 5, chosen to span the different bets rather than the largest budgets. All sixteen claim alert triage, and every mark is the vendor's own.

Read the table as a map of the argument, not a buyer's guide. Alert triage is the disposition of a single alert; investigation is the work of establishing what happened; containment is the action taken against a host, an account or a connection; and detection engineering is the upstream task of writing and tuning the rules and analytics that decide what fires at all.

VendorAlert triageInvestigationContainmentDetection engineering
AI-native triage
Dropzone AI●●●●
Prophet Security●●●●
7AI●●●●
Exaforce●●●–
Autonomous SOC and AI SIEM
Intezer●●●●
Hunters●●●●
Automation and SOAR
Torq●●●–
Tines●●●●
Platform vendors
IBM QRadar with watsonx●●––
Microsoft Security Copilot and Sentinel●●●●
Google SecOps with Gemini●●●●
CrowdStrike Charlotte AI●●●●
Palo Alto Cortex XSIAM and AgentiX●●●●
SentinelOne Purple AI●●●●
MDR and open source
Expel●●●●
Wazuh●–●●
// Figure 5. Sixteen vendors' claimed coverage of the analyst's loop, grouped by what they are and colored to match: ● means claimed, – not claimed.

Vendors Let the Agent Close Routine Alerts and Leave the Consequential Call to People

What they sell converges more than the vendor count suggests. The marketing settles on one division of authority: the agent disposes of the ordinary alert, and the person owns the consequential call. Where they differ is how much of the loop the agent covers before that call, and how far it is trusted to act on its own. IBM's QRadar assistant is advisory, summarizing an offense and recommending a response a human executes. Cisco and Splunk, Elastic, and Zscaler's MDR sit in the human-approves-each-action tier.

The agentic products claim autonomy within guardrails, and the guardrails are visible in the documentation: Microsoft's triage agent closes false positives and deliberately leaves true positives open for an analyst, Palo Alto requires approval for high-impact actions, and Prophet Security widens an agent's authority only as its record justifies it, describing the result as "role elevation, not role elimination". The managed detection providers are selling the outcome, not the tool, which is the same product with the triage labor priced in.

The Boldest Claims Promise to Remove the Triage Tiers Entirely

Read past the feature grid and the boldest claims make the triage analyst unnecessary. Torq's site carries a Carvana customer line: the agentic AI "now handles 100% of Carvana's Tier-1 security alerts and has automated 41 different runbooks within just one month of deployment". ReliaQuest's headline is "100% of Alerts Investigated by AI / So That You Don't Have To", with the elimination of Tier 1 and Tier 2 work stated as the product, against 202,000 alerts a day and 99.4% investigation accuracy. Intezer's product page claims every alert investigated, fewer than 2% escalated and 98% verdict accuracy.

Dropzone prices the equivalence rather than implying it: up to 4,000 investigations a year per AI analyst, "the equivalent of the average output of a human tier 1 analyst". Swimlane's Victrix case study has that MSSP reporting "100% AI accuracy compared to an analyst's judgment".

A second tier of claims is about speed instead of staffing: up to 98% less mean time to repair on Palo Alto's XSIAM page, five-minute detection and fourteen-minute response on Expel's MDR page, four-minute mean time to respond on Prophet's site, three-minute agent investigations on Zscaler's MDR page.

The Claims Rest Almost Entirely on the Vendors' Own Pages

What backs any of it? Vendor pages, almost entirely. Not one of the percentages quoted in this section arrives with a confusion matrix behind it, and the customer stories behind them, Carvana, Panther's Loglass and HealthEquity, AiStrike's Sunrun, the Microsoft customer saving 200 hours a month by auto-closing false positives, are anecdotes published by the seller. None reports how many real incidents the system waved through. The Zero Day Clock makes the same complaint about the field as a whole: the market is fluent about capability and silent about how its error rates were measured.

The strongest third-party material I found is analyst-firm content the vendors distribute themselves — an IDC snapshot of SentinelOne, an IDC MarketScape excerpt ReliaQuest hosts — and it assesses a vendor's own deployment, not the field. The one study with an outside party's name on it, the Cloud Security Alliance benchmark from October 2025, measured analysts assisted by one vendor's tool on two investigations.

// no independent evaluation

I could not find an independent, multi-vendor, end-to-end evaluation of an autonomous SOC anywhere: no adjudicated comparison, no error rate on live traffic, nothing that tests what happens when the agent is wrong.

So the boldest claims do not hold up, and the failure is specific rather than total. The coverage claims are measurable without measuring correctness, which is why every vendor makes them.

An agent that investigates every alert and escalates nothing has achieved 100% coverage by construction.

The time savings are better supported, by the Microsoft trials and by the survey material in Part 1, and they fit the narrower thing the products do. What is missing is the number a handover depends on: how often the automated verdict is wrong in the direction that hurts, a real incident closed as benign. Until someone publishes that with an outsider checking the numbers, the human layer is not being replaced so much as assumed. No vendor here promises that the analyst goes outright; the boldest promise only to remove the triage tiers.

08 The Oversight Paradox

The first job in a SOC is the queue: triaging alerts, closing the obvious false positives, escalating what looks real. The vendors sell its removal, and what remains is machines doing the scale work and humans doing the judgment. That assumes the humans stay sharp without the work that trained them, and human-factors research gives the reason they will not.

The more reliable and pervasive the automation, the less attention anyone pays to it, and the less able they are to catch it when it fails.

Organizations write the analyst into the workflow as the backstop for exactly that failure, then automate away the daily practice that made the backstop worth having. Figure 6 traces the cycle that follows.

A five-station feedback loop drawn clockwise: deploying high-throughput AI automation leads to passive triage, then less deliberate mental effort, then skill atrophy, then automation surprise with lost oversight and edge-case failure. A red dashed arrow closes the loop back to the first station. Labels for complacency, automation bias and automation surprise sit beside the stations they apply to, and a dial at the center shows oversight capacity near its low end.
// Figure 6. The oversight feedback loop, from high-throughput automation to catastrophic failure on an edge case.

Fluent Models Make Complacency, Automation Bias and Surprise Worse

Parasuraman and Manzey's review of decades of automation research names mechanisms that apply directly to a SOC, and finds that complacency and bias show up in expert users too and are not removed by simple practice or instruction. Three matter:

Generative models make all three worse. A rule engine emits a flag. A language model emits a fluent, confident paragraph arguing that this command line is a harmless admin script, and fluent explanations lower the reader's own critical effort. Hallucinations are hard to catch because they read well.

The interview study adds what changes in practice. As practitioners delegate more, their worry moves from the false positive that has defined SOC work to the false negative, the attack an over-trusted model waves through. One CISO, M1, gives the study its title: "We went from chasing ghosts to missed attacks." The asymmetry is the reason. A false positive can be reviewed and discarded; a missed attack costs far more.

Accountability is one reason the human layer cannot be thinned past a point. CISO M2's question in Figure 3, who is responsible when a model errs, has a finding behind it: the interview study reports that when a decision is shared with an AI system, analysts feel less personally accountable for it.

Automating the First Job Removes the Training Ground for Future Supervisors

Diagnostic intuition, the ability to look at proxy logs and a process tree and feel that something is wrong, comes from years of hands-on triage. That first job was the grind that built it. Junior analysts triaged thousands of alerts, learned what normal looks like, and moved up to incident response, threat hunting and architecture. Figure 7 shows the progression, and the rung that goes missing.

Two panels of ascending steps. Top, in green: Tier 1 high-volume alert triage builds intuition, Tier 2 deep forensics refines root-cause analysis, and Tier 3 threat hunting and architecture executes strategic defense. Bottom, in red: an AI agent pipeline replaces Tier 1 and eliminates junior triage, a dashed empty step asks 'Where do future experts come from?', and a crossed-out arrow shows the connection to a still-required Tier 3 analyst broken.
// Figure 7. The traditional tier progression, and the break where entry-level triage is automated.

If that first job is automated entirely, where do the people who supervise the AI come from? Rinta-Kahila and colleagues documented the pattern in a case study of cognitive automation, where skills eroded in a self-reinforcing cycle as workers relied on the system. When the experienced analysts retire, the organization is left with operators who cannot spot errors or run an incident by hand if the tooling is compromised. Oversight becomes symbolic: present on the org chart, hollow in practice.

The SOC literature already has a name for the mechanism. In the Vicious Cycle that Sundaramurthy and colleagues described in 2015, underskilled analysts are not trusted, so they are not empowered to grow, so they underperform and leave, and the job-design paper cited in Part 1 starts from it. Automating that job removes the practice that builds both the skill and the trust. Interviewees in the 2026 study saw the contradiction themselves.

Only experts can reliably spot LLM errors, yet the tools are aimed at juniors doing the Tier 1 work A7 describes in Figure 3.

The job-design paper also codes job postings, with its limits stated: it is a preliminary workshop paper, one researcher coded the postings, and postings show what organizations claim to require, not what they do. A posting is not demand and not a hire: it can be a pipeline-building exercise, a formality, or a role that is never filled. Across 106 public SOC job postings in 11 countries, communication skills appear in half of them while SIEM tools appear in fewer than a fifth, and only a third name any of 43 certifications. The postings advertise tools and credentials, not learning: learning agility appears in just 6.6% of them.

They also disagree with practitioners, since SANS 2025 has 43% of respondents naming SIEM as the top skill they seek in new hires. The postings were collected in late 2024 and say nothing about AI, which the authors list as future work. Organizations advertise those requirements at the door.

Vendors Say the Entry Rung Is Being Redesigned, Not Removed

The vendors selling the Tier 1 replacement describe a different outcome, and their version deserves a hearing because it is the one being bought. At Infosecurity Europe in June, three of them argued that the entry rung is being redesigned rather than removed.

Dropzone AI's Brett Candon calls the result a "tier-1.5 analyst," a first job spent supervising and auditing AI investigation work instead of gathering data by hand, with job satisfaction up and promotions arriving faster. Abnormal AI's field CISO, Patricia Titus, kept the rung deliberately: after her team stopped hiring what she called "five permanent tier-1 ticket takers," she rebuilt the remaining work as a university intern program, for the reason this post has been giving. "I think we would be foolish to eliminate tier-1 SOC analysts, largely because what happens if something happens and AI stops working…"

Vega Security's Yonni Shelmerdine supplies the other half: "AI isn't going to replace the SOC, it's a cyber defense engineer who will." A vendor cannot hire your juniors for you, and the panel had an interest in optimism, so the test is whether the internships outlive the hiring freeze. But they do disagree with the pipeline argument: the empty desk is not what the people selling automation say they are selling.

Critics Call Tier 1 a Burnout Mill, but Something Must Replace What It Taught

The sharpest objections to the pipeline argument come from people who do not sell SOC tools, and they pull in opposite directions. Forrester's Allie Mellen, an industry analyst and not a practitioner, says on a podcast that being an L1 "has always been a rite of passage, but only because it burns you out and you're miserable," and that the mistake is marketing the tools to new analysts, because "they're wrong sometimes."

Sinan Al Taie, a cybersecurity manager at Master Electronics, speaking on a podcast run by a managed-security provider, bets the other way: that a Tier 1 analyst with AI can produce Tier 3 outcomes, and that people who use AI "will replace people who don't." He is betting on uplift, not erosion, and nothing in this post tests it, since no study cited here measures juniors working with AI against seniors without it.

Mellen's point is the strongest objection to the pipeline argument, and Al Taie's is the most optimistic. The grind should not be defended for its own sake. The open question is what replaces the learning it provided.

Skill Retention Should Be Managed Like Any Other Operational Risk

Saka, writing from the Max Planck Institute for Security and Privacy, argues in a position paper for the human-AI teaming workshop at SOUPS 2026 that skill retention should be handled like any other operational risk. The core idea is competency risk: the chance that automated assistance weakens skills operators need for meaningful oversight. The paper proposes sorting tasks by how critical the skill is and how heavily the task is delegated to AI. Its schema is preliminary and untested, so treat what follows as a design proposal. Figure 8 draws that sort.

A two-by-two matrix with competency criticality on the horizontal axis and AI delegation on the vertical axis. High delegation with low criticality is 'Low atrophy concern' (automated IP lookups, initial log clustering); high delegation with high criticality is the red 'Critical oversight zone' (exploit hypothesis, incident-triage decisions) where delegation threatens crisis skills. Low delegation with low criticality is 'Low-priority routine' (internal shift notes); low delegation with high criticality is the green 'Core human practice zone' (deep binary forensics, proactive threat hunting) kept as deliberate manual workflows.
// Figure 8. A two-by-two of competency criticality against AI delegation, with example tasks in each cell.

The top-right cell, high criticality with high delegation, is where the danger sits, and practices follow from that. Three come from the paper, and the fourth is my own addition:

The job-design paper adds a flow-theory rule of thumb: persistent boredom atrophies skill and persistent anxiety blocks upskilling, so automate routine work but keep what people do calibrated to their skill. Passive approval of model output is the boredom zone; a diet of only the hardest residual cases is the anxiety zone. The paper tests none of it.

I would add one practical point. Keeping that judgment alive is an engineering requirement and has to be designed in: drills, honest metrics, and a deliberate path for juniors to learn what normal looks like.

A SOC that automates its apprenticeship out of existence gets its oversight for one budget cycle and loses it afterward.

09 Conclusions

The debate usually arrives as a binary, full automation or human-led defense, and I think the binary misreads both sides. The Bitter Lesson holds for the knowledge it was derived from, the kind people wrote down as rules. Defenders will not secure modern networks by hand-writing signatures. But the claim that AI therefore does everything simply ignores an opponent who attacks the statistical assumptions of the models themselves.

What remains is a layered system in which the machine and the human each do what they are structurally suited to. Figure 9 shows the machine and human layers.

Layered diagram. A telemetry stream from endpoints, cloud, networks and identities feeds a blue machine layer of four tiles: ingestion and correlation, dedup and summaries, pre-authorized containment, and formal proofs with memory safety. A blue down arrow carries 'synthesized context and hypothesis scaffolding' to an amber human layer of ambiguous anomalies, business logic and novel attacks, attribution and threat hunting, and ethics, law and risk governance, and an amber up arrow carries 'feedback, recalibration, domain context' back. A side panel shows a vertical bar of analyst queries to an LLM from an observational study of 3,090 queries by 45 analysts over ten months: about 4% asked for a verdict, and the rest were mostly about understanding commands and telemetry.
// Figure 9. The co-teaming layers: a machine layer for scale and speed, a human layer for judgment, with context flowing down and feedback flowing up.

The argument in this post predicts that split. The machine layer takes scale-bound work: ingesting billions of logs, deduplicating and summarizing alerts, enriching indicators, modeling attack paths across identity graphs, and executing pre-authorized containment inside a limited blast radius. The human layer takes director-level work: threat-hunting hypotheses, business context, logic-abuse analysis, supervision of model reliability, and the consequential calls that carry legal, financial and ethical accountability.

I am not alone in landing there. Position papers on SOC co-teaming, Albanese and colleagues' and Sarker and colleagues', and Sudhakaran and Kshetri's comparison of an AI agent with human forensic investigators, reach hybrid designs for many of the same reasons: the machine is better at what is computable, and the human owns what is not.

That division holds only if the human layer keeps being staffed. The judgment it relies on is learned in the work the machine layer is taking over, so the apprenticeship has to be rebuilt on purpose, as a tier-1.5 rung, an internship or a schedule of AI-off drills. Otherwise the human layer thins to a name on the org chart within a hiring cycle.

Cybersecurity cannot be solved once with code or compute, because it is a contest between human intentions mediated by infrastructure. Defense will move toward general, scalable architectures, and I expect the last handwritten rule to be retired sooner than most security teams plan for.

As long as the opposing side is attackers, the work that remains is judgment, context and accountability.

References

The bitter lesson and misuse detection
Analysts, SOCs and human-AI teaming
The analyst voice, outside the vendor surveys
SOC practitioner surveys
Practitioner and vendor commentary
Vendor claims and benchmarks cited in What the Vendors Are Selling
Talks and podcasts (YouTube)
Automation, attention and skill erosion
Threat landscape, workforce and competitions