Loading story…
Elastic has outlined an evidence-based framework to benchmark large language models used in agentic security operations centers. The methodology keeps the Elastic AI Agent, prompts, tools, and datasets fixed while routing runs to different models and anonymizing model identities for blind scoring. Evaluations measure whether models select the correct skills, make appropriate tool calls with valid parameters, execute tasks in the right order, and produce findings grounded in collected evidence—not merely whether their final prose appears convincing.
The Agent Builder suite uses a synthetic Windows intrusion involving Chrysalis DLL side-loading, with the EICAR test-file hash serving as a deterministic malicious VirusTotal result and live case-management and Slack-response tools. Additional suites test Attack Discovery's ability to correlate roughly 95 alerts into eight intrusions and Automatic Migration's conversion of 52 Splunk detection rules, including macro-dependent rules, into Elastic rules. The scoring rubric evaluates both answer quality and correct tool or skill use, caps unsupported answers when required evidence-gathering calls are absent, and reports reliability independently from quality.
Track how attackers are adapting to this technology.
1 event from the most recent confirmed update back to the earliest known activity.
Elastic built an evidence-based framework to evaluate LLMs in agentic security operations workflows by scoring execution traces, including skill routing, tool calls, parameters, order, and grounded findings, alongside final responses. The framework evaluates Agent Builder, Attack Discovery, and Automatic Migration tasks while keeping the agent, prompts, tools, and datasets constant across models.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Elastic has released the **Detection Engineering Behavior Maturity Model (DEBMM)**, a five-tier framework for improving the processes used to build, maintain, test, and refine SIEM search-based threat-detection rules. The model progresses from **Foundation** through Basic, Intermediate, Advanced, and Expert levels, pairing qualitative practices with quantitative measures for evaluating detection-engineering capabilities. DEBMM prioritizes core controls—including rule governance, documentation, version control, telemetry quality, threat modeling, testing, and reviews of the threat landscape—before organizations adopt practices such as false-negative triage, external validation, threat hunting, and advanced automation. Elastic positions the framework as complementary to the NIST Cybersecurity Framework, MITRE ATT&CK, ISO/IEC 27001, hunting maturity models, and existing detection-engineering maturity matrices, and advises organizations to pursue continuous, risk-based improvements aligned with available skills, staffing, and budget.
Sep 9, 2026
Security commentary and research are converging on a concern that today’s **LLM evaluation methods** do not reflect real-world defensive needs, particularly in **security operations**. SentinelLABS reports that widely cited benchmarks (including those associated with major vendors) tend to measure narrow, clean, and reproducible tasks that map poorly to SOC workflows, which are continuous, collaborative, and frequently disrupted by changing conditions; it also notes a trust issue where many benchmarks rely on **LLMs grading other LLMs**, sometimes using the same vendor’s models for both, creating a closed loop that can be gamed. In parallel, industry discussion is shifting from “general-purpose chatbot” hype toward a future of **specialized models** (e.g., *small language models* and task-specific “digital workers”), arguing that AI’s durable value will come from embedding into decision-making and operational processes rather than novelty use cases. Taken together, the sources suggest CISOs should treat vendor benchmark claims cautiously and prioritize evaluations tied to actual SOC outcomes (reliability under change, workflow fit, and measurable defensive impact) rather than generic model scores.
Jun 29, 2026
Elastic Security Labs detailed a detection pipeline that uses **ES|QL COMPLETION** to reduce noise from `curl` and `wget` execution alerts tied to **MITRE ATT&CK `T1105` Ingress Tool Transfer**, a technique widely used by threat actors to download second-stage payloads, post-exploitation tools, and ransomware from command-and-control infrastructure. The approach preserves deterministic detections, then extracts destination hosts, applies allow-lists, redacts secrets from command lines, aggregates remaining events, and submits only a small set of high-value records to an LLM for a structured verdict of **true positive**, **false positive**, or **suspicious**.
Aug 26, 2026