Israeli security researchers unveiled how hackers could exploit an AI’s “hallucinations” to jeopardize an organization’s software supply chain. In a Vulcan Cyber blog post, Bar Lanyado, Ortel Keizman, and Yair Divinsky demonstrated how false information produced by ChatGPT about open-source software could be used to inject malicious code during development. The researchers observed ChatGPT generating non-existent URLs, references, code libraries, and functions. By leveraging these fabricated elements, attackers could disseminate malicious packages without resorting to easily detectable methods like typosquatting or masquerading, according to the researchers.
Continuing with the researchers’ findings, they highlighted that if an attacker manages to craft a package to substitute the “fake” packages suggested by ChatGPT, there is a possibility of enticing a victim to download and utilize it. They argued that this scenario’s likelihood is rising due to an increasing number of developers shifting from conventional code solution platforms like Stack Overflow to AI-based alternatives such as ChatGPT.
According to Daniel Kennedy, the research director for information security and networking at 451 Research, the authors predict that as generative AI gains popularity, it will attract developer queries that were previously directed to platforms like Stack Overflow. However, the generated responses from AI may be inaccurate or refer to non-existent packages. This opens up an opportunity for malicious actors to exploit the situation by creating a code package with that name, including malicious code, and having it consistently recommended to developers by the generative AI tool.
Kennedy further explained that the researchers at Vulcan Cyber took this concept a step further by prioritizing the most commonly asked questions on Stack Overflow and feeding them to the AI. They then examined instances where recommendations were made for non-existent packages.
The researchers followed a specific methodology in their study. They began by querying Stack Overflow to identify the most frequently asked questions across over 40 different topics. From each subject, they selected the initial set of 100 questions.
Using ChatGPT’s API, they fed all the collected questions to the AI model. This process aimed to simulate an attacker’s approach to obtaining a high number of recommendations for non-existent packages within the shortest possible timeframe.
For each response received, the researchers examined the package installation command for a discernible pattern and extracted the recommended package. They then proceeded to verify if the recommended package actually existed. In cases where the package was non-existent, the researchers attempted to publish it themselves.
Henrik Plate, a security researcher at Endor Labs, a dependency management company in Palo Alto, California, highlighted that malicious packages created using ChatGPT’s code have already been detected on package installers PyPI and npm.
According to Plate, large language models like ChatGPT can also assist attackers in developing malware variants that incorporate similar logic but exhibit different forms and structures. This includes distributing malicious code across multiple functions, altering identifiers, generating fictitious comments and dead code, or utilizing comparable techniques.
Ira Winkler, the chief information security officer at CYE, a global provider of automated software security technologies, observed that the current issue with software lies in its heavy reliance on existing code rather than independent writing.
He explained, “Software today is essentially a combination of various existing software pieces, which is a very efficient approach as developers don’t have to write common functions from scratch.
Bud Broomhead, the CEO of Viakoo, a developer of cyber and physical security software solutions based in Mountain View, California, emphasized that authenticating code is not always a straightforward process. He pointed out that many digital assets, particularly IoT/OT devices, often lack digital signing or other methods of establishing trust, leaving room for potential exploits.
Regarding the use of generative AI in both cyber offense and defense, Broomhead mentioned that we are still in the early stages. He commended Vulcan and other organizations for their efforts in detecting and alerting about emerging threats, which allows language learning models to be fine-tuned to prevent such exploits.
He added that it’s important to remember that only a few months ago, he could request ChatGPT to create new malware, and it would comply. However, now it requires specific and focused guidance to unintentionally generate such content. Broomhead expressed hope that even these specific approaches will soon be prevented by AI engines, highlighting the ongoing progress in enhancing AI systems to mitigate potential risks.
