Skip to main content
Mallory
← Back to News
ai-platform-securitydetection-content-updateadversary-emulation-tradecraft

Elastic Introduces Evidence-Based Benchmarking for Agentic SOC LLMs

Emerging Technology Risk · 2 sources · Updated 2mo ago · First seen Aug 4, 2026

Elastic Introduces Evidence-Based Benchmarking for Agentic SOC LLMs

Elastic has outlined an evidence-based framework to benchmark large language models used in agentic security operations centers. The methodology keeps the Elastic AI Agent, prompts, tools, and datasets fixed while routing runs to different models and anonymizing model identities for blind scoring. Evaluations measure whether models select the correct skills, make appropriate tool calls with valid parameters, execute tasks in the right order, and produce findings grounded in collected evidence—not merely whether their final prose appears convincing.

The Agent Builder suite uses a synthetic Windows intrusion involving Chrysalis DLL side-loading, with the EICAR test-file hash serving as a deterministic malicious VirusTotal result and live case-management and Slack-response tools. Additional suites test Attack Discovery's ability to correlate roughly 95 alerts into eight intrusions and Automatic Migration's conversion of 52 Splunk detection rules, including macro-dependent rules, into Elastic rules. The scoring rubric evaluates both answer quality and correct tool or skill use, caps unsupported answers when required evidence-gathering calls are absent, and reports reliability independently from quality.

New surface, new adversariesTrack how attackers are adapting to this technology.
Track this risk
Event timeline

How this story unfolded

Events from the most recent confirmed update back to the earliest known activity.

1 event
  1. Reported Aug 4, 20263mo ago

    Elastic develops agentic SOC LLM evaluation framework

    Elastic built an evidence-based framework to evaluate LLMs in agentic security operations workflows by scoring execution traces, including skill routing, tool calls, parameters, order, and grounded findings, alongside final responses. The framework evaluates Agent Builder, Attack Discovery, and Automatic Migration tasks while keeping the agent, prompts, tools, and datasets constant across models.

    Benchmarking the Agentic SOC: How we evaluate LLMs for security workflows
Linked entities

Related entities

Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.

Open in app →

Affected products

1 linked
Elastic Security

Organizations

1 linked
Elastic
Do you run this?Follow how adversaries are adapting to this technology, and where it touches your stack today.
Track This Risk
Mallory platform

See the full picture, correlated to your attack surface.

This page covers what’s public. Mallory adds the parts that aren’t: which of your assets are affected, which threat actors are using it right now, and what to deploy next.

Exposure mappingMap indicators from this story to your assets and identify affected systems in minutes.
Threat actor evidenceEvery observed campaign, victim, and pivot linked to actors named in this story.
Associated malwareMalware, exploits, and IOCs connected to the activity described here.
Detection signaturesYARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Scheduled alertsGet matching new stories delivered to your team as they break, not the next morning.
AI threadsAsk questions about this story and take action on the answers.