Malicious actors are now exploiting generative AI to carry out cyberattacks: scamming victims using AI-generated deepfakes, deploying malware developed with the help of AI coding tools, using chatbots to pull off phishing campaigns, and hacking widely used open-source code repositories with AI agents. And these AI-driven threats are rising. In early April, Anthropic’s Frontier Red Team, which evaluates the potential safety and security risks posed by the company’s AI models, announced that the company’s Claude Mythos Preview model has identified thousands of high- and critical-severity vulnerabilities. The list includes some in “every major operating system and every major web browser,” despite the model not being explicitly trained for this. Those findings prompted Anthropic to also establish Project Glasswing to help thwart AI-assisted cyberattacks. Its launch partners—which include tech giants such as Amazon Web Services (AWS), Apple, Google, Microsoft, and Nvidia—will use Mythos Preview to scan and secure software. While generative AI’s coding, reasoning, and autonomous capabilities have become powerful enough to spot potential code security weaknesses, these same skills also enable it to exploit those flaws. Cybersecurity experts believe that finding the right and safe balance for using AI in detecting code vulnerabilities seems feasible—as long as layers of verification are built into the process, and human judgement and expertise remain an essential part of the process. AI Discovers Critical Code Vulnerabilities Among the vulnerabilities discovered by Mythos Preview are a 27-year-old bug in OpenBSD, an open-source Unix-like operating system that enables a remote attacker to crash any machine running the OS; a web browser exploit that could allow a cybercriminal with their own website domain to read data from another domain, such as a user’s bank; and a number of weaknesses in cryptography libraries that could, for instance, let hackers decrypt encrypted communications or forge certificates. Finding bugs is nothing new, especially for cybersecurity researchers, but AI serves as yet another tool in the toolbox, says Jeremy Katz, vice president of code security at Sonar, a company that offers code verification solutions. Large language models (LLMs) are adept at fulfilling directed queries to search for specific security vulnerabilities. “You can point an AI agent at a large codebase, and they’re very good at finding the needle in a haystack,” he adds. For Nayan Goel, a principal application-security engineer at the financial services company Upgrade, speed and semantics set AI models apart. They can pinpoint vulnerabilities faster than humans, and their ability to reason about the semantics of code, following data flows across different abstraction layers, is a cut above the pattern-matching functionalities of traditional static analysis tools. “That’s the kind of cross-component reasoning that is structurally beyond what rule-based tools can do,” Goel says. “And what this new generation of tools is doing is closer to how a security researcher would actually think.” Andrew Wesie, cofounder and chief technology officer at cybersecurity company Theori, takes a similar optimistic view. “We have an approach that may actually help us find all the bugs—that was always considered to just be a pie-in-the-sky dream. And we’re at the point where that does work.” Despite their promising potential, LLMs are still prone to generating false positives. That could mean incorrectly flagging a bug as a security vulnerability, for example, or overstating a bug’s severity. This makes it challenging to find the signal among the noise, especially for the volunteers maintaining important open-source resources, who face pressure to provide prompt fixes. Katz has witnessed this as someone who works closely with open-source maintainers on coordinated vulnerability disclosure. “I’m seeing a drastic uptick in the number of things being reported. In many cases, they’re real bugs that would be good to fix but not actually a security vulnerability—that fine line is getting lost. And just the amount of time to triage is becoming pretty large.” Another drawback involves AI tools that can be attacked (such as through prompt injections) but can also do the attacking themselves. Mythos Preview, for example, can chain together separate but related vulnerabilities to form a step-by-step exploit that grants root access to the Linux kernel, the core or “seed” of the OS. Balancing AI Security Tools With Human Review Harnessing AI’s benefits while avoiding its shortcomings is possible, according to cybersecurity experts. Tools such as Claude Code Security and Google’s CodeMender conduct what’s called an adversarial self-review pass, which means they can challenge and critique their own results before presenting them. This additional layer of scrutiny, which can also include an LLM or AI agent sending its findings to another model or agent for validation, could lessen false positives and build checks and balances into the process. But Goel emphasizes that the issues AI models flag must still be checked and confirmed by humans. “These tools produce probabilistic outputs. They’re not the final verdict,” he says. “They cannot act as a substitute for your secure design reviews or penetration testing reviews. You still need somebody who understands the business logic behind your code and reviews that. And anytime AI gives us a finding, it goes through a verification process. There’s always a human in the loop so we create these trust boundaries.” Goel also cites dynamic threat modeling and red teaming as other ways to achieve a safe balance for using AI in hunting code vulnerabilities. Dynamic threat modeling evaluates likely threats to AI systems and how to mitigate them as systems evolve, while red teaming assesses the safety and security of AI systems and the possible risks they might introduce. Uncovering the middle ground for code vulnerability detection also requires some process changes. Shifting security earlier in the software development process, when programmers are crafting code, can make a huge difference. “Organizations need to implement ongoing education and upskilling programs that give developers the skills they need to mitigate flaws in software before they can be released,” says Matias Madou, cofounder and chief technology officer at the software security firm Secure Code Warrior. “By ensuring that we have developers who can effectively create and review secure code from the start, we’re taking the necessary steps to protect against potential disaster.” As AI gets better at identifying the right code-security weaknesses and accurately classifying their severity, the next challenge becomes closing the gap between detecting and fixing vulnerabilities at scale. “The last bit of that workflow is remediation,” says Jeffrey Martin, vice president of product at Theori. “We as security professionals understand that a vulnerability needs to be remediated, and that remediation follows certain patterns, and we should be able to scale out and solve that problem as well. We feel that’s the next area that AI can really help with.”