AI slop and faux experiences are coming in your bug bounty applications

So-called AI slop, that means LLM-generated low-quality photos, movies, and textual content, has taken over the web within the final couple of years, polluting web sites, social media platforms, at the very least one newspaper, and even real-world occasions. 

The world of cybersecurity shouldn’t be resistant to this drawback, both. Within the final yr, individuals throughout the cybersecurity business have raised considerations about AI slop bug bounty experiences, that means experiences that declare to have discovered vulnerabilities that don’t truly exist, as a result of they had been created with a big language mannequin that merely made up the vulnerability, after which packaged it right into a professional-looking writeup. 

“Persons are receiving experiences that sound cheap, they give the impression of being technically right. After which you find yourself digging into them, making an attempt to determine, ‘oh no, the place is that this vulnerability?’,” Vlad Ionescu, the co-founder and CTO of RunSybil, a startup that develops AI-powered bug hunters, instructed TechCrunch. 

“It seems it was only a hallucination all alongside. The technical particulars had been simply made up by the LLM,” mentioned Ionescu. 

Ionescu, who used to work at Meta’s purple staff tasked with hacking the corporate from the within, defined that one of many points is that LLMs are designed to be useful and provides constructive responses. “If you happen to ask it for a report, it’s going to provide you a report. After which individuals will copy and paste these into the bug bounty platforms and overwhelm the platforms themselves, overwhelm the shoppers, and also you get into this irritating scenario,” mentioned Ionescu. 

“That’s the issue individuals are working into, is we’re getting a variety of stuff that appears like gold, but it surely’s truly simply crap,” mentioned Ionescu. 

Simply within the final yr, there have been real-world examples of this. Harry Sintonen, a safety researcher, revealed that the open supply safety challenge Curl obtained a faux report. “The attacker miscalculated badly,” Sintonen wrote in a publish on Mastodon. “Curl can odor AI slop from miles away.”

In response to Sintonen’s publish, Benjamin Piouffle of Open Collective, a tech platform for nonprofits, mentioned that they’ve the identical drawback: that their inbox is “flooded with AI rubbish.” 

One open supply developer, who maintains the CycloneDX challenge on GitHub, pulled their bug bounty down fully earlier this yr after receiving “virtually fully AI slop experiences.”

The main bug bounty platforms, which primarily work as intermediaries between bug bounty hackers and corporations who’re keen to pay and reward them for locating flaws of their merchandise and software program, are additionally seeing a spike in AI-generated experiences, TechCrunch has realized. 

Michiel Prins, the co-founder and senior director of product administration at HackerOne, instructed TechCrunch that the corporate has encountered some AI slop. 

“We’ve additionally seen an increase in false positives — vulnerabilities that seem actual however are generated by LLMs and lack real-world affect,” mentioned Prins. “These low-signal submissions can create noise that undermines the effectivity of safety applications.”

Prins added that experiences that comprise “hallucinated vulnerabilities, obscure technical content material, or different types of low-effort noise are handled as spam.”

Casey Ellis, the founding father of Bugcrowd, mentioned that there are undoubtedly researchers who use AI to seek out bugs and write the experiences that they then undergo the corporate. Ellis mentioned they’re seeing an total enhance of 500 submissions per week. 

“AI is extensively utilized in most submissions, but it surely hasn’t but brought on a big spike in low-quality ‘slop’ experiences,” Ellis instructed TechCrunch. “This’ll most likely escalate sooner or later, but it surely’s not right here but.”

Ellis mentioned that the Bugcrowd staff that analyzes submissions opinions the experiences manually utilizing established playbooks and workflows, in addition to with machine studying and AI “help.”

To see if different corporations, together with people who run their very own bug bounty applications, are additionally receiving a rise in invalid experiences or experiences containing non-existent vulnerabilities hallucinated by LLMs, TechCrunch contacted Google, Meta, Microsoft, and Mozilla. 

Damiano DeMonte, a spokesperson for Mozilla, which develops the Firefox browser, mentioned that the corporate has “not seen a considerable enhance in invalid or low-quality bug experiences that may look like AI-generated,” and the rejection price of experiences — that means what number of experiences get flagged as invalid — has remained regular at 5 – 6 experiences per 30 days, or lower than 10% of all month-to-month experiences.

Mozilla’s workers who evaluation bug experiences for Firefox don’t use AI to filter experiences, as it will possible be troublesome to take action with out the chance of rejecting a reputable bug report,” DeMonte mentioned in an e-mail.

Microsoft and Meta, corporations which have each guess closely on AI, declined to remark. Google didn’t reply to a request for remark. 

Ionescu predicts that one of many options to the issue of rising AI slop will probably be to maintain investing in AI-powered techniques that may at the very least carry out a preliminary evaluation and filter submissions for accuracy. 

In actual fact, on Tuesday, HackerOne launched Hai Triage, a brand new triaging system that mixes people and AI. In response to HackerOne, this new system is leveraging “AI safety brokers to chop by means of noise, flag duplicates, and prioritize actual threats.” Human analysts then step in to validate the bug experiences and escalate as wanted.

As hackers more and more use LLMs and corporations depend on AI to triage these experiences, it stays to be seen which of the 2 AIs will prevail.