I’ve been gently ducking AI questions on here, but since Beaumont directly asked the question on Fedi, I will ask here:
Have you seen an AI-enabled attack in your environment?
I have not seen a model/agent execute or orchestrate an attack, as far as I am aware.
That being said, the average complexity of any given malware sample I receive has notably ticked up. Unfortunately, fighting fire with fire is an (immediate) reasonable response.
I have written a tool using AI that looks at Splunk and uses a rubric to determine the likelihood that an attack is AI generated. Of course, there is no way to be certain, but when I review the findings I see where it is coming from.
So far, around 3.7% of the total attacks are AI generated. Casual perusal of the blocked requests makes me think it is right more than it is wrong.
This is not a surprise. With Burp AT, it’s cheap and easy to generate attacks to a targeted URL that was found through recon (ask me how I know).
It is a great question. I guess it really depends on what we consider an attack and what we consider LLM-enabled. For the purposes of this post I will define attacks as incidents and LLM-enabled as using LLMs to speed up development rather than actually instigating the attacks on “their own”. So far, none that we have observed. I avoid the subject because the hype that gets attached to the term, but I have no illusions that threat actors will be using LLMs to speed up the development of exploits. To quote a point @cR0w made awhile ago, the underlying techniques are still the same so I think the solution is often “boring” things like a good internal security program, solid purple team engagements and compensating controls rather than automagic AI defense or buying into the hype by resigning decisions to a model.
While I generally agree that the issue is one of scale, not kind, I do see some areas that require specific attention regarding LLMs.
First, as we now know, there’s the possibility that models attempting to fulfill goals will compromise your network without attacker intent. That would be as true from trusted partners or internal sources as external attackers. It’s still true that strong security architecture protects against much of this, but perhaps this unguided attack vector increases the urgency for those best practices.
Second, as noted, the use of these tools to increase the complexity/obfuscation of even quotidian malware means higher labor for responders. It’s utterly stupid that we’re just burning tokens against each other to arrive at essentially the same result, but that’s basically the situation. Or, I guess, we just accept that static indicators in malware samples are now a lot harder to come by, and perhaps of even lower value than before. That’s possible, but it also means that dynamic analysis becomes even more important.
Yet, one can’t exist without the other. To build a proper dynamic analysis environment, you should know going into it what the sample will look for to detect your sandbox. Aaaand we’re back to laborious static analysis.
Edited to add: I don’t want to come off as dismissive of any position here. I think I’m just generally skeptical all the way around that we really understand the implications or the ground truth. Anyone who claims to have it all figured out is either selling you something or missing a trick.
Yes, they of course require special attention and I do think that the urgency for best practices has increased. Techniques that were more difficult to implement for less experienced attackers are of course becoming more commonplace as well a decrease in the speed of exploitation which does indeed change the threat model. LLMs can help in some of the analysis, speeding up defenders but in general I caution against viewing LLMs as the holy grail “solution” rather than just a tool in the toolbox. While the underlying techniques for compromise remain that same, the distribution of techniques and the time period has changed.
I’d like to quote a post from Bluesky:
Cyber insurer (Chatham house forbids me from saying who) at BH CISO summit: “so far this year 0 payouts for claims even remotely connected to a AI-driven breach, 85% of payouts related to social engineering”. Sobering statistic.
While it’s only anecdotal it fits my personal (and thus also anecdotal) evidence - I have yet to see an actual “AI-driven” attack. I’ve seen incidents with indications that the threat actor/s used an LLM for parts of their work (such as getting an LLM to write a script to automate exfiltration or parsing through data for them), but even those are rather rare in comparison to attacks without (seemingly) any involvement of AI.
(None of this is to say that they don’t exist. But I’d be willing to wager that these types of attacks make up a minuscule amount of intrusions.)