The questionnaire every vendor passes
Ask ten AI SOC vendors whether their agents investigate alerts autonomously, explain their reasoning, integrate with your SIEM, and keep a human in the loop. You will get ten yeses. The RFP was built to separate vendors, and in this category it has quietly stopped doing that job.
The big picture: RFP fatigue runs in both directions. Buyers are tired of scoring 200-line questionnaires where every answer reads the same. Vendors are tired of writing them, and increasingly hand the first draft to an AI model. The result is a document that measures how well a proposal team writes, which is a different thing from how well a platform investigates. In a category where Gartner's own evaluation guidance is framed around telling viable products apart from AI washing, the gap between the written answer and the running system is the whole evaluation.
Why it matters: This series started with the real cost of an internal SOC, MDR, and an AI SOC platform, then argued for renting the engineering and keeping the judgment, then showed how outsourcing turns into a liability when the wrong things live with the provider. Every one of those decisions eventually lands in a procurement process. If that process can't tell a working investigation engine from a well-written answer, the operating model you chose gets decided by whoever writes the best proposal.
By the numbers:
- Gartner analysts Craig Lawson and Andrew Davies predict that by 2028, 70% of large SOCs will pilot AI agents to augment Tier 1 and Tier 2 work, but only 15% will achieve measurable improvements without structured evaluation, per the research note Validate the Promises of AI SOC Agents With These Key Questions, as reported in a vendor-sponsored Help Net Security article.
- The same article notes that Gartner's Hype Cycle for Security Operations, 2026 moved AI SOC agents to the Peak of Inflated Expectations, a year after placing them at the Innovation Trigger with single-digit adoption.
- In Darktrace's State of AI Cybersecurity 2026 survey of more than 1,800 security professionals, 56% of CISOs strongly agreed that AI improves their defensive capabilities, compared with 25% of hands-on security operations practitioners, according to Kiteworks' summary of the report. The same research found 74% of respondents are limiting AI autonomy in the SOC until explainability improves.
- Prophet Security's 2026 State of AI in Security Operations research found that 72% of teams using AI had tried building their own tooling, and 46% of those builds were abandoned, never reached production, or were replaced by a commercial product. Data privacy and model training (44%) and explainability (41%) topped the list of adoption barriers, per the same Help Net Security piece.
- On the vendor side of the table, the average RFP response takes 33 hours of work, and teams submit an average of 166 a year, according to Loopio's 2026 RFP Response Trends and Benchmarks report produced with APMP, as summarized by Stealth Agents. Loopio also reports that 79% of response teams now use AI in their RFP workflows, per CB Insights' coverage of the release.
- 79% of technology buyers experience regret tied to problems that range from bad requirements to insufficient contracts and poor implementations, per the public abstract of Gartner's June 2026 research for IT sourcing and procurement leaders.
- Overstated AI claims now carry legal risk for the seller. The FTC's May 2026 action against three marketing firms was the agency's 13th AI-washing case since 2024, and seven of the last eight involved claims made to other businesses rather than consumers, by DLA Piper's count.
Read together, the numbers describe a market where adoption is outrunning verification. Buyers are piloting fast, the executives signing off are more optimistic than the analysts using the tools, and the documents used to choose a vendor are increasingly machine-written.
Why the RFP stopped working for AI SOC
Questionnaires worked reasonably well for categories where a feature either existed or didn't. A SIEM either parsed a log source or it didn't. AI SOC platforms break that model in three ways.
The vocabulary collapsed. "Autonomous," "agentic," "explainable," and "human in the loop" now appear in nearly every vendor's answers, and they mean different things in each. Constellation Research's Chirag Mehta observed at RSAC 2026 that vendors are using the same terms to describe different architectures and operating models, and that buyers need to look closely at what is actually autonomous, what still requires approval, and what data the system reasons across. A checkbox labeled "autonomous investigation" can't capture any of that.
The answer describes the system on its best day. A written response, a scripted demo, and a curated sandbox all show the platform working on data it has seen before. The cases that decide whether an AI SOC is worth buying are the ones nobody scripted: an identity alert that only makes sense once you see the endpoint telemetry, a phishing case that touches a SaaS app the vendor has a shallow connector for, a true positive that looks routine on its face.
The system you buy keeps changing. A traditional tool behaves the same way on renewal day as it did on signing day, give or take a release. An AI system's behavior can shift every time the underlying model changes. Command Zero cofounder and CPO Alfred Huger put it bluntly in Yes, Build Your Own AI SOC: "LLMs are extraordinary at many things; determinism is not one of them." A questionnaire answered once in the spring says little about how the platform will reason after its next model upgrade.
None of this means buyers should skip structured questions. Gartner's framework, as described in the Help Net Security piece, is built around them: workload fit, meaningful outcomes, vendor durability, analyst development, autonomy boundaries, integration, and transparency. The point is to attach each question to evidence the buyer can observe.
Turn each RFP question into a test
The most useful change a buyer can make is mechanical. Keep the question, but rewrite the requirement so it can only be satisfied by something the platform does in front of you.
RFP question | The answer you will get | The evidence to require instead |
|---|---|---|
Do your agents investigate alerts autonomously? | Yes | Run the platform against two to four weeks of your own closed alerts, including alert types you currently suppress for capacity reasons, and compare its verdicts to your team's dispositions |
Is the AI explainable? | Yes, full transparency | For ten investigations you pick, show every question asked, every query run against every source, and what came back. Your analyst then reruns one of those queries in the source tool and confirms the result |
Do you integrate with our stack? | Yes, 100+ integrations | Show which sources the agent actually queried during the trial, whether it requires copying data into its own storage first, and what the investigation does when one source is slow or unavailable |
What is your accuracy? | 95%+ | Report false negatives first, on replayed cases your team already confirmed as true positives. State whether the vendor's headline figures came from a proof of concept or from sustained production |
Is the investigation fully software-driven? | Yes | State in the contract what share of investigations are reviewed or edited by vendor staff before they reach you. Human review is fine. Undisclosed human review changes what you are buying |
How do you handle model changes? | We use the latest models | Share regression results from the last model change: which known cases were rerun, what changed, and how customers were notified |
How is it priced? | Per alert, per seat, or per investigation | Price the platform at 2x and 5x your current alert volume, and with the suppressed alert types turned back on |
How is our data handled? | We are SOC 2 compliant | Name every subprocessor that touches your data, including model providers, say whether your data or your analysts' feedback trains anything, and describe the export you would receive at exit |
Two rows in that table do most of the work. The explainability row tests whether the reasoning exists outside the vendor's own interface. The accuracy row forces the conversation onto the failure that matters most. A false positive costs an analyst some minutes. A false negative, an investigation that confidently closes a real intrusion as benign, costs whatever the attacker does next.
Run a proof of value on your own alerts
A demo shows what a platform can do. A proof of value (POV) shows what it does in your environment, with your data, on your problems. The difference is mostly in how the trial is scoped.
Baseline before the trial starts. You can't measure improvement without a starting point. Record current investigation times, escalation rates, and how many alerts go uninvestigated. The Prophet research found respondents reported 28% of alerts are never investigated, and up to 40% of organizations have turned off specific detection rules because of capacity. Those suppressed alerts are some of the best test material you have.
Pick the hard cases on purpose. Include multi-source investigations that cross identity, endpoint, email, and SaaS. Include cases your senior analysts remember because they were hard. A trial that only covers cases where basic telemetry is enough tells you how the platform handles the easy scenario and nothing about the one you are buying it for.
Keep the answers blind. Replay cases where your team already knows the outcome, and don't tell the vendor which ones are true positives. Score the verdicts afterward.
Score the reasoning, then the verdict. A correct verdict reached for the wrong reasons will fail on the next case. Have analysts grade whether each investigation asked the questions they would have asked, queried the sources they would have checked, and reached its conclusion from evidence they can see.
Put practitioners on the scoring panel. The 56% versus 25% gap between CISOs and practitioners in the Darktrace data is a warning about who evaluates. The analysts who will live with the platform are the ones best placed to notice when an investigation looks thorough and isn't.
Plan for what a short trial can't show. Durability problems such as model drift, connector breakage, and degraded accuracy after an upgrade don't appear in two weeks. Write a 90-day checkpoint into the contract with the same scoring method, and tie renewal terms to it.
What to measure in the POV | Why it matters | What a passing result looks like |
|---|---|---|
False negative rate on replayed true positives | The most expensive failure mode | Zero confident benign verdicts on confirmed incidents; any misses come with a visible reason |
Reproducibility of evidence | Separates auditable reasoning from narrative | An analyst can rerun any query from the investigation in the source tool and get the same result |
Coverage across sources | Single-source AI closes tickets on what it can see | Investigations pull from every relevant source without manual steps |
Analyst agreement and override rate | Shows whether reviewers are thinking or stamping | Analysts agree with most verdicts and can explain every override |
Cost at volume | Pricing that works at pilot scale can break in production | A written price at 2x and 5x volume that you can budget against |
Exit readiness | Ties back to the exit test | A full export of cases, evidence, and custom logic delivered in a format you can read |
Where AI SOC evaluations go wrong
Structured evaluation is the fix, and it has its own failure modes.
- The bake-off becomes its own RFP fatigue. A POV costs analyst hours. Running five of them in parallel produces shallow scoring on all five. Narrow to two or three finalists using the evidence requirements above, then go deep.
- Speed gets overweighted. Investigation time is the easiest number to collect and the most flattering. As we argued in Why MTTD and MTTR Are No Longer the Whole Story, once a machine does first-pass work, speed mostly measures the machine. Accuracy and reasoning quality decide whether the speed is worth anything.
- Visible reasoning gets mistaken for correct reasoning. A long, confident investigation trail can still be wrong. Command Zero CTO Dean De Beer's research, The Recomposition of Security Work, warns that agent outputs increasingly read like expert analysis, and that reviewers need the skill to spot failure inside them. The question he poses for executives applies to every evaluator: "what is this agent capable of missing?"
- Autonomy gets evaluated as a yes or no. Most teams are granting autonomy one action category at a time. Evaluate how the platform scopes autonomy by action type and reversibility, the line we drew in Autonomous Investigation vs. Autonomous Response.
- The incumbent's bundled AI gets a pass. Gartner's Hype Cycle guidance advises evaluating AI features already embedded in incumbent SIEM or XDR platforms before adding standalone systems, as summarized by Prophet Security. Hold both to the same evidence standard. An AI feature that comes with a renewal still needs to prove it can reason across data its parent platform doesn't own.
What industry experts are saying
Gartner's Craig Lawson and Andrew Davies frame structured evaluation as the deciding variable. Their prediction that most large SOCs will pilot AI agents but only 15% will see measurable gains "without structured evaluation" puts the burden on the buying process itself, according to Help Net Security's coverage of their note.
Constellation Research's Chirag Mehta came out of RSAC 2026 focused on the gap between the booth and the SOC. Broad platform positioning is easy to present at an event, he wrote, while "proving it in production is a different standard," in his floor observations. He also noted little appetite among buyers for blind trust in autonomous systems.
Former Gartner analyst Gorka Sadowski put the same idea in plainer terms ahead of the conference, telling buyers to arrive with questions and adding, "No more 'trust us, bro,'" on LinkedIn.
Alfred Huger has argued that the hardest part of an AI SOC is the part buyers rarely ask to see. In Yes, Build Your Own AI SOC, he describes evaluation infrastructure, regression-testing investigations across model changes and measuring false negatives, as the least glamorous and most important piece of the system. "Most teams that build their own have no eval harness at all and don't budget for one," he wrote. The same test applies to vendors. Ask to see the harness.
Dean De Beer ties evaluation capacity to long-term risk. In The Recomposition of Security Work, he lists keeping internal capability to evaluate vendor agent systems among the signs an organization is on the productive path. Among the warning signs he lists: "Agent outputs accepted without structured analyst challenge." An RFP process that scores written answers is a version of that failure, moved upstream to the day of purchase.
[EDITOR'S NOTE, remove before publishing] No original quote from Eric Hulse or Dov Yoran is included because none could be sourced to a public record on this topic. Suggested ask for Eric: the one test he would run in the first week of any AI SOC proof of value. Suggested ask for Dov: what CISOs tell him about the RFPs they have stopped trusting. The Sadowski quote comes from a public LinkedIn post; confirm the link resolves before publishing.
Closing the series: the RFP is where the operating model gets decided
The four posts in this series land in the same place. Each one produced a test, and each test belongs in the buying process.
Post | The question | The test to put in your RFP |
|---|---|---|
Where does each operating model break at scale? | Price the platform at 2x and 5x your current alert volume | |
Which half are you renting? | For every function, name who does the work, who sees the reasoning, who decides, and who could take it over | |
What do you keep if the contract ends? | Request a full export of cases, evidence, detections, and custom logic during the evaluation | |
This post | Is the capability real? | Replay your own confirmed cases blind and score the reasoning before the verdict |
A buyer who runs all four tests is no longer scoring proposals. They are scoring evidence, and vendors can't write their way past that.
The limits of the argument
Between the lines: Not every organization can run a rigorous proof of value. A small team with no spare analyst hours may reasonably lean on peer references, analyst research, and a shorter trial. The evidence requirements above still apply. They just get checked with less depth.
The data behind this post has limits too. The Gartner figures arrive secondhand through a vendor-sponsored article and public abstracts. The Darktrace findings come through a third party's summary of the report. The Prophet Security survey and the Loopio benchmarks are vendor research, and Loopio's data describes the people writing RFP responses rather than the people reading them. The FTC case count is a law firm's tally. No independent study has compared outcomes between AI SOC purchases made through traditional RFPs and those made through structured proofs of value, so this argument rests on how the evaluation is designed more than on measured results.
Command Zero has a stake here. We sell an AI SOC platform, and every test in this post is one buyers can and should run on us. We built the platform around glass-box investigations that query data where it already lives and show every question and result behind a verdict, so we have an obvious interest in buyers asking for exactly that. Treat this post with the same skepticism you would bring to any vendor's evaluation guide, including the parts that sound most reasonable.
The bottom line: RFP fatigue is a signal that the questionnaire has outlived its usefulness for this category. Every vendor can say yes, and some of the yeses are now written by the same kind of AI the buyer is trying to evaluate. Keep the questions, but attach each one to something you can watch the platform do on your own alerts: replay confirmed cases blind, rerun the queries yourself, count the false negatives, price the volume you'll actually have, and pull the export before you sign. The vendors with real capability will welcome that process. The ones who don't have shown you the answer.
Terms in this post
Terms defined in Command Zero's glossary are linked below. Terms without a glossary entry yet are defined here.
What is RFP fatigue?
RFP fatigue is the exhaustion buyers and vendors experience when long, repetitive requests for proposal consume significant time on both sides while producing answers that look so similar the document no longer helps distinguish one vendor from another.
What is AI washing?
AI washing is marketing that overstates or misrepresents how much a product relies on artificial intelligence or what that AI can do. Agent washing is the same practice applied to claims about autonomous AI agents.
What is a proof of value in security operations?
A proof of value (POV) is a time-boxed trial in which a security platform runs against the buyer's own data and use cases, with success criteria defined in advance, to show whether it delivers measurable outcomes in that specific environment.
What is the difference between a proof of concept and a proof of value?
A proof of concept tests whether a product can technically perform a function. A proof of value tests whether that function produces a meaningful, measurable improvement against the buyer's own baseline.
What is replay testing for an AI SOC?
Replay testing is an evaluation method in which an AI SOC platform investigates historical alerts whose outcomes the buyer's team has already confirmed, so its verdicts can be scored against known answers.
What is a false negative in an AI SOC investigation?
A false negative is an investigation that concludes a real threat is benign. In AI SOC evaluation it is usually the most expensive error, because the case gets closed and the attacker keeps going.
What is an evaluation harness?
An evaluation harness is the testing system that checks whether an AI system still produces correct results after a change, such as a model upgrade, by rerunning known cases and comparing the outcomes.
