The race nobody should be running
Picture an analyst on a team that just turned on an AI triage agent. The agent closes a phishing alert in 40 seconds. She used to take 12 minutes. Over the next month she has two ways to read that. She can treat the agent as a faster version of herself and try to keep pace, or she can treat it as a junior investigator who works fast, never gets tired, and still needs someone to tell it what matters in this environment and when it has missed something.
The first reading ends badly for her. The second describes most of the job that is coming.
The big picture: AI agents now do the first pass on a growing share of SOC work, and analyst skill requirements are moving with them. Hiring data from the AI Workforce Consortium shows cybersecurity postings that ask for AI skills doubled in a year, with requirements shifting away from SIEM proficiency and query writing toward agent supervision and output validation. Most teams have not trained for that shift. They bought the agent, and the analysts were left to work out the rest.
Why it matters: Supervising an agent well takes the same judgment the agent was supposed to save you from needing. Research on AI-assisted work shows that judgment weakens when people hand off the thinking along with the typing. A SOC that adopts agents without a skills plan can end up with analysts who approve work they can't evaluate. In our last post we looked at where new analysts start when Tier 1 is automated. This post covers what every analyst, junior or senior, now has to get good at.
By the numbers:
- 73% of practitioners say AI changed their team's training requirements in 2026, up from 51% in 2025, in the 2026 SANS AI Survey of 536 practitioners. Active AI use in security rose from 50% to 78% over the same year, and only 27% describe their deployment as mature production.
- 63% of those practitioners report significant AI shortcomings in threat detection and response, up from 45%.
- 38% of organizations provide comprehensive AI security training to staff, while 54% have AI governance policies on paper, per the SANS | GIAC 2026 workforce research.
- 28.5% of G7 cybersecurity job postings required AI skills from October 2025 to March 2026, double the 14.2% a year earlier, according to the AI Workforce Consortium.
- 35% of cybersecurity teams are leaning more on AI or automation to cover technical skills gaps, up 12 points in a year, and only 8% of organizations regularly run AI-specific response exercises, per ISACA's 2026 State of Cybersecurity.
- In a randomized trial of 52 developers learning a new library, the AI-assisted group scored 50% on a follow-up comprehension quiz against 67% for those working by hand, with the largest gap on debugging (Shen and Tamkin, 2026).
What "directing" an agent means on a real case
"Upskill for AI" usually turns into a prompt engineering course. Prompting matters, but it's the smallest part of the job. When I watch strong analysts work alongside investigation agents, I see four distinct things they do, and only one of them involves typing a question.
They scope the case. Before the agent runs anything, a good analyst decides what question the investigation has to answer. "Is this sign-in malicious?" is a weaker question than "Did anyone other than the account owner use this session, and did they touch anything after the MFA prompt?" We talk about this internally as questions before tokens. A well-scoped question gives the agent a target and gives the analyst a way to tell later whether the answer is complete.
They supply context the agent can't see. The agent knows what the telemetry says. It doesn't know that finance runs a payroll batch on the last business day, that one executive's assistant logs in from the executive's laptop, or that a service account was rebuilt last Tuesday and its new behavior is expected. We covered why that knowledge is so hard to hire for in Talent Shortage vs. Skills Gap. Directing an agent means getting that knowledge into the investigation, either case by case or, better, as context the agent can use every time.
They challenge the reasoning. This is the skill most teams mean by "validation," and it's harder than reading a summary and nodding. It means checking that the evidence supports each step, asking what the agent didn't look at, and knowing which data sources were in scope. An agent reporting "no anomalous activity" is only useful if you know whether it had the proxy logs. Dean De Beer, our CTO, puts it plainly in The Recomposition of Security Work: "Agent output critical evaluation is the foundational universal skill in an agentic future."
They own the decision. Someone still decides to disable the account, isolate the host, or close the case as benign. That analyst has to understand what the action costs if the call is wrong, and has to be able to explain the decision afterward to an incident review, an auditor, or a customer. The record of why a conclusion was accepted, changed, or rejected is part of the work product.
None of these four is new. Senior analysts have always scoped, added context, challenged, and decided. What has changed is that these skills are now the main job for everyone on the floor, including people who haven't had years of manual cases to build them.
The skills that fade when AI does the work
The uncomfortable part is that heavy AI use can erode the exact skills directing an agent requires.
The clearest evidence comes from outside security. In a randomized controlled trial published in January 2026, Judy Hanwen Shen and Alex Tamkin had 52 mostly junior software engineers learn an unfamiliar asynchronous programming library. Half used an AI assistant. Both groups finished in about the same time. The AI group scored noticeably lower on a follow-up quiz, and the biggest gap was on debugging, the skill you'd need to catch an AI's mistake. The authors also found that how people used the assistant mattered more than whether they used it. Participants who delegated the work wholesale scored lowest, averaging 39%. Participants who generated code and then asked follow-up questions to understand it scored 65% or higher.
A CHI 2025 study from Carnegie Mellon and Microsoft researchers surveyed 319 knowledge workers about 936 real tasks. People with more confidence in the AI applied less critical thinking to its output. People with more confidence in their own ability applied more. The researchers also found that workers often skipped critical thinking when they lacked the skills to inspect, improve, and guide the AI's response.
Put those findings in a SOC and the risk is easy to see. A junior analyst who has never worked an identity compromise by hand gets a tidy AI timeline that says the activity is benign. She has low confidence in her own judgment and growing confidence in the tool, because it has been right the last 40 times. That's the combination both studies flag.
Security practitioners see this already. ISC2 found that 89% of AI-using practitioners had seen AI recommendations lead to incorrect outcomes, and 63% were spending more time validating output, figures we covered in the career pathing post. More validation time is only useful if the people doing it can find what's wrong.
John Hubbard, who teaches SOC analyst training at SANS, makes the same point in his look at AI in 2026 security operations: working with AI doesn't mean you need less knowledge, because you still have to verify whether what it did was correct.
The research also points to the fix. The developers who kept learning used the AI to understand the work instead of only to produce it. A SOC can design for that.
A skills map for analysts who direct agents
This table is our working framework. It pairs each skill with what it looks like on a case, a way to practice it, and something you can measure. Treat it as a starting point to adapt, since we haven't tested it across many SOCs.
Skill | What it looks like on a case | How to practice it | What to measure |
|---|---|---|---|
Case scoping | Writes the investigative question before the agent runs, names what "done" means | Have analysts write the question for closed cases and compare it with what was actually answered | Share of cases where the conclusion answers the stated question |
Context supply | Adds environment facts the agent lacks, and flags when an agent's "normal" is wrong for this org | Turn recurring analyst corrections into written environment notes the agent uses next time | Repeat corrections on the same fact (should fall over time) |
Evidence evaluation | Checks each reasoning step against raw telemetry instead of another summary | Explain-back: restate the agent's conclusion and its strongest evidence in your own words before accepting | Accuracy of overrides on later review |
Coverage awareness | Knows which sources the agent queried and which it couldn't reach | Require a "what wasn't checked" line on every AI-assisted closure | Closures later reopened because of a missed source |
Error detection | Catches a wrong conclusion that reads as fluent and confident | Seeded-error drills: replay real cases with a planted mistake in the agent's output | Catch rate on seeded errors, by analyst |
Consequence judgment | Matches the level of scrutiny to what the action would break | Walk through containment decisions by consequence tier in tabletop exercises | Approval decisions overturned in post-incident review |
Decision record | Writes why the conclusion was accepted, changed, or rejected | Grade a sample of decision notes in weekly case review | Share of decisions a second analyst can follow without asking |
Two of those rows need a closer look.
Seeded-error drills are the most direct way to train and test error detection, and few teams run them. Take a closed case, change one thing in the agent's output (a wrong timestamp, a missed host, a conclusion the evidence doesn't support), and give it to an analyst to review. You learn whether your reviewers find the mistake, and they learn what agent errors look like in your environment. A reviewer who never catches a planted error is telling you something an override rate can't.
Coverage awareness is the one people skip. AI output tends to be confident about what it found and quiet about what it didn't look at. De Beer's paper calls out data source and tool-chain awareness as the thing that lets an analyst judge whether a finding is complete as well as accurate. Teaching analysts to ask "what couldn't the agent see?" as a habit pays off on every case.
Building the program
The career pathing post laid out four practices for keeping junior analysts learning: scheduled manual reps, graded review, budgeted mentoring, and rotation. Those still apply. The practices below are about building the director skills across the whole team, seniors included.
Teach how agents fail before you teach how to use them. Chris Cochran, SANS field CISO and VP of AI security, told CSO Online that training has to cover how AI can fail and how to catch it when it does. In practice that means a library of your own agent's mistakes, pulled from real overrides and incident reviews, used as training material. Generic AI literacy courses can't give you that.
Make context-writing a deliverable. Every time an analyst corrects an agent because of something specific to your environment, that correction should become a written note the agent and the next analyst can use. This is how institutional knowledge stops living in one person's head. It's also a skill you can grade, since a good environment note is specific, sourced, and dated. The next post in this series is about retaining that knowledge as senior analysts leave, and this habit is where it starts.
Tie sign-off authority to demonstrated skill. An analyst who can approve an agent's conclusion to close a benign alert doesn't automatically qualify to approve host isolation. Define consequence tiers for agent actions (read-only lookups, case closure, containment, changes to production) and qualify analysts for each tier through drills and reviewed cases. Write down who holds which authority and revisit it as the agents change.
Protect the training time. SANS found that lack of time is the biggest barrier to training, and our burnout post argued for treating training hours the way you treat on-call coverage. If the agent frees up analyst time, put part of it on the calendar for drills and case review before the queue absorbs it.
Move your metrics off throughput. Gartner analyst Pete Shoard recommends measuring case accuracy, cycle time, and decision confidence in AI-enabled SOCs instead of alert volume, as SC Media summarized. The same report predicts that by 2028 engineering staff will outnumber traditional analysts. If you still rank analysts by cases closed per shift, you're rewarding the behavior the studies above warn about.
Pick tools that show their work. You can only train people to challenge reasoning they can see. De Beer's paper argues that platforms exposing their reasoning chains are formation infrastructure, and that tools returning only a verdict end up reducing headcount without building anyone's skill. That's a buying criterion as much as a training one, and we went through how to test for it in RFP Fatigue. We built Command Zero around glass-box investigations for this reason: every question the platform asks and every piece of evidence it uses is there for the analyst to inspect.
Where AI helps, and where it adds pressure
AI can make analysts better at this work, and it can make them worse, depending on how the work is set up.
On the helpful side, an analyst who reviews AI-completed investigations sees far more reasoning in a month than she would have built by hand. When an agent follows an expert-encoded method, a junior gets to watch a senior's investigative approach applied to case after case, which used to require sitting next to that senior. Explain-back and follow-up questions turn each reviewed case into a lesson. The developers in the Shen and Tamkin trial who used AI to understand their code show that this works.
The pressure is just as real. Review is tiring in a way investigation isn't. Approving the 41st correct verdict of the day is exactly when attention drops, and reliable tools make the drop worse. Our CTO's warning applies here: an override rate of zero should worry you, because it usually means reviewers have stopped looking. The skills list also keeps growing. The AI Workforce Consortium lists prompt injection recognition and investigating with AI-layer telemetry among new SOC analyst skills, on top of everything analysts already needed. And training budgets don't grow as fast as the list.
Eric Hulse, our VP of Engineering, describes in The Middle of the Shift Map how work leaving the middle of the SOC moves left into detection engineering, right into response, or into supervising AI triage. Each of those paths needs different training. A single "AI skills" course won't cover all three.
What industry experts are saying
Chris Cochran of SANS summarized the governance principle in the same CSO piece: "Give AI room to scale, but never fully remove the human." The article's broader point, drawn from interviews with several CISOs, is that reskilling works when practitioners get protected time and sandbox environments, backed by a formal training plan.
The AI Workforce Consortium's report frames the shift in hiring terms. Agentic AI is taking over high-volume work like alert triage and threat intelligence correlation, and the analyst's role moves toward orchestration and oversight. The consortium also reports fast growth in demand for durable skills like systems thinking and ethical reasoning, which suggests employers see judgment as the scarce part.
Gartner's Pete Shoard expects detection engineers to take on prompt design, workflow testing, and AI output validation, and analysts to focus on validating AI conclusions and communicating response strategy. The report's phrase for the target state is "end-to-end case ownership."
Dean De Beer's paper is the most specific about what each role has to learn. Tier 1 analysts move toward validating agent output and challenging reasoning steps. Tier 2 analysts orchestrate teams of agents and test hypotheses across evidence chains. For them, he writes, "Knowing when to override an agent becomes as important as the underlying security knowledge." Senior analysts take on the hardest change: turning investigative intuition into explicit system design. The paper also names the worst case, a generation of Tier 2 analysts who "will have supervised agent outputs but never conducted a manual investigation."
The limits of the argument
Between the lines: "Direct the agent" can turn into an excuse to stop teaching the fundamentals. It shouldn't. Every skill in the table above depends on knowing what a real compromise looks like in raw telemetry. An analyst who has never built a timeline by hand will struggle to spot a wrong one. ISC2's finding that most practitioners say AI hasn't reduced the need for foundational skills points the same way.
The evidence has gaps worth stating. The strongest data on skill erosion comes from software development and general knowledge work. Nobody has run the same trials in a SOC. The Shen and Tamkin trial was small and measured learning of a new tool in a single session. It says nothing direct about skill decay over months. The Carnegie Mellon and Microsoft study relies on self-reported behavior. The survey figures here come from organizations with something to sell: SANS and ISACA sell training and certifications, and the AI Workforce Consortium's members include large AI and technology vendors. We read the Gartner figures in a press summary and haven't seen the full report. We haven't found an independent study that measures how well SOC analysts catch AI errors after training, so treat our skills map and drills as practical recommendations that still need testing.
Command Zero has a stake here. We sell an AI SOC platform built on encoded expert methodology and transparent investigations, and this post argues for exactly those things. We've tried to show where that argument stops. No platform trains an analyst on its own, and a transparent tool used by a team that never reads the reasoning is no better than a black box. Run a seeded-error drill on your own team and judge the results yourself.
The bottom line: Racing the agent is a losing game for analysts. The job is scoping the case, supplying the context, challenging the reasoning, and owning the call, and those skills can fade quickly if nobody trains them on purpose. Start this week with one thing: take five closed AI-assisted cases, plant one error in each, and see how many your team catches. That number will tell you more about your upskilling needs than any survey.
Terms in this post
Terms defined in Command Zero's glossary are linked below. Terms without a glossary entry yet are defined here.
- SOC (Security Operations Center)
- SOC analyst
- SOC skills gap
- AI SOC
- SOC Tier 1
- SOC Tier 2
- Alert triage
- Detection engineering
- Encoded expertise
- Question-based method
- Human-in-control
- Prompt injection
What does it mean for a SOC analyst to direct an AI agent?
Directing an AI agent means scoping the investigative question, supplying environment context the agent lacks, checking the agent's reasoning and evidence, and making and documenting the final decision. The agent does much of the evidence gathering. The analyst stays accountable for whether the conclusion is right.
What skills do SOC analysts need to work with AI agents?
The core skills are case scoping, context supply, evidence evaluation, awareness of which data sources the agent covered, error detection, judgment about the consequences of an action, and clear decision records. All of them rest on foundational investigation skills, like reading raw telemetry and building a timeline.
Will AI agents replace SOC analysts?
AI agents are taking over much of the high-volume first-pass work, such as alert triage and enrichment. Analyst work is shifting toward oversight, investigation ownership, detection engineering, and response decisions. Hiring data shows rising demand for analysts with AI supervision and validation skills.
How do you train SOC analysts to validate AI output?
Effective methods include seeded-error drills, where analysts review real cases with a planted mistake, explain-back reviews before accepting an AI conclusion, required notes on what the agent didn't check, and regular review of decision records. Catch rates on seeded errors give a measurable view of progress.
Can relying on AI weaken analyst skills?
Research outside security suggests it can. A 2026 randomized trial found developers who used AI while learning a new library scored lower on comprehension and debugging, especially when they delegated the work wholesale. Using AI to understand the work, through follow-up questions and explanation, reduced the effect.
What is AI agent supervision in a SOC?
AI agent supervision is the human oversight of AI systems that investigate or act on security events. It includes reviewing agent conclusions, approving or rejecting actions according to their consequences, correcting the agent's context, and recording why each decision was made.
What is automation bias in security operations?
Automation bias is the tendency to trust automated output more than the evidence supports. In a SOC it shows up as analysts accepting an AI verdict without checking the underlying telemetry, and it tends to get worse as a tool becomes more reliable.
What is a seeded-error drill?
A seeded-error drill is a training exercise where a reviewer is given a real or replayed AI-assisted investigation containing a deliberately planted mistake. The drill measures whether the reviewer finds the error and teaches what agent mistakes look like in that environment.



