Research Highlights
Paper: Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks
First published 27 January 2025, arXiv 2501.16466; IEEE Symposium on Security and Privacy, May 2026
Authors: Brian Singer, Carnegie Mellon University; Keane Lucas, Anthropic; Lakshmi Adiga, Meghna Jain, Lujo Bauer, and Vyas Sekar, Carnegie Mellon University
Summary: We found that leading LLM-assisted cybersecurity tools struggle to autonomously carry out complex, multi-host red team attacks. Our new system, Incalmo, uses LLMs for high-level planning while specialized agents execute individual tasks, successfully compromising critical assets in 37 of 40 test environments compared with just 3 of 40 for existing systems. Successful attacks took 12–54 minutes and cost less than $15 in LLM credits.
Paper: Establishing Systems-Level Competitive Leaderboards for Autonomous Network Security Operations
Work in progress
Authors: Brian Singer, Lakshmi Adiga, Marko Morrison, Lujo Bauer, and Vyas Sekar, Carnegie Mellon University
Summary: We propose a new framework for evaluating autonomous AI attackers and defenders in realistic network environments, moving beyond simplified, one-off cybersecurity challenges. Our system enables competitive, system-level testing and shows that LLM-based autonomous defenders can slow down or sometimes stop autonomous LLM-based attackers.
Policy Memorandum: The Case for a Federated National Agentic Cyber Range
First published 14 September 2026
Authors: Eric Heim, Ramayya Krishnan, and Vyas Sekar, Carnegie Mellon University
Summary: This memo argues that AI has dramatically lowered the cost of finding software vulnerabilities, while validation and remediation remain expensive and slow. It proposes a national federated agentic cyber range: a network of accredited testing sites operating under shared rules to safely evaluate software, support organizations that cannot afford testing, and feed verified findings into GOLD EAGLE and CERT/CC’s VINCE platform