CyLab white paper explores the growing gap between AI-assisted performance and expertise

A new white paper from Carnegie Mellon University's CyLab Security and Privacy Institute examines how artificial intelligence may be changing what cybersecurity competitions measure, and the extent to which performance reflects underlying expertise.

The paper, "Skill or Shortcut? AI, Competitive Cybersecurity Learning, and the Growing Gap Between Performance and Expertise," draws on six years of competition data from CyLab’s annual global Capture-the-Flag (CTF) competition. The analysis identifies significant shifts in participant performance, solve timing, and competitive outcomes that accelerated alongside the widespread availability of advanced AI systems.

Rather than attempting to measure learning outcomes directly, the white paper focuses on observable changes in competition behavior. The findings raise broader questions about how educators, employers, and learners should interpret competitive performance in an environment where increasingly capable AI systems can assist with problem solving.

CyLab Security Academy serves more than one million registered users worldwide and hosts one of the largest cybersecurity competitions in the world. The annual competition attracts participants ranging from middle and high school students to university students and working professionals, providing a unique longitudinal view into how competitive cybersecurity behavior evolves over time.

Among the most notable observations documented in the paper are dramatic reductions in solve times for difficult challenges, increased score compression among top-performing teams, and significant changes in how competitors progress through challenges during the competition window. In the 2026 competition, the top 50 teams all achieved identical scores, leaving completion time rather than score as the primary differentiator among top-ranked competitors.

"What we observed wasn't simply that participants were solving challenges faster," said Megan Kearns, program director of CyLab Security Academy and co-author of the paper. "The more important question is whether competition performance continues to reflect the understanding and judgment cybersecurity education is intended to develop."

The paper also documents examples of participants experimenting with increasingly autonomous AI-assisted workflows during competition. While these observations represent only a subset of participants, they highlight emerging questions about supervision, verification, and the role of human judgment when AI systems are incorporated into technical problem-solving processes.

The authors emphasize that the paper is not an argument against AI. Instead, it examines how the growing capabilities of AI systems may be changing the meaning of competitive outcomes in cybersecurity environments.

"For decades, cybersecurity competitions have served both as learning experiences and as indicators of technical capability," said Kearns. "As AI systems become more capable, educators need to think carefully about what competition results tell us, what they don't tell us, and how we continue developing the skills that matter most."

The paper argues that cybersecurity education is entering a period of transition. Historically, competition performance has been viewed as a proxy for a combination of knowledge, persistence, and technical problem-solving ability. As AI-assisted workflows become more common, the relationship between performance and expertise may become increasingly difficult to interpret through competition results alone.

The findings also reinforce a broader evolution already underway within CyLab Security Academy. Originally launched as an annual competition, the platform transitioned to a year-round educational environment in 2020, expanding beyond competitive events to support ongoing cybersecurity learning and educator engagement.

Today, the Academy continues to expand its educational offerings across cybersecurity, AI security, and blockchain security, areas that reflect the evolving skills and knowledge required of future security practitioners. The white paper's findings will help inform future educational initiatives focused on technical skills, critical thinking, and the responsible use of increasingly capable AI systems.

While the paper focuses on competition data, its implications extend beyond cybersecurity competitions. AI systems continue to grow increasingly capable across technical domains, and educators, employers, and learners face a resulting challenge: understanding what performance signifies and how expertise should be evaluated in an environment where intelligent assistance is increasingly available. The authors argue that answering those questions will be essential to the future of cybersecurity education.

The full white paper, "Skill or Shortcut? AI, Competitive Cybersecurity Learning, and the Growing Gap Between Performance and Expertise," is available through CyLab Security Academy.

CyLab Research Team

Megan Kearns, Luke T. Jones, Ivan Liang, and Max Yin