LLMs used to code can introduce serious errors
Cette publication existe aussi en Français
In one of the most comprehensive studies, UTSA researchers have demonstrated how a particular error could cause serious problems for programmers using AI, specifically LLMs, to help write code.
Hallucinations in LLMs occur when the model produces content that is factually incorrect, nonsensical, or completely unrelated to the input task. Most current research has focused mainly on hallucinations in classical natural language generation and prediction tasks such as machine translation, summarisation, and conversational AI. However, the research team focused on package hallucination, which occurs when an LLM generates or recommends the use of a third-party software library that does not exist. What makes package hallucinations a fascinating area of research is how something so simple — a single, everyday command — can lead to serious security risks.
Joe Spracklen, a UTSA doctoral student in computer science, led the study on how large language models (LLMs) frequently generate insecure code. “It doesn’t take a convoluted set of circumstances or some obscure thing to happen,” Spracklen said. “It’s just typing in one command that most people working in those programming languages type daily. That’s all it takes. It’s very direct and very simple.”
“It’s also ubiquitous,” he added. “You can do very little with your basic Python coding language. It would take you a long time to write the code yourself, so it is universal to rely on open-source software to extend the capabilities of your programming language to accomplish specific tasks.”
LLMs are becoming increasingly popular among developers for assisting in assembling programs. According to the study, up to 97% of software developers incorporate generative AI into their workflow, and 30% of code written today is AI-generated. Further, many popular programming languages, like PyPI for Python and npm for JavaScript, rely on a centralised package repository. Because the repositories are often open source, bad actors can upload malicious code disguised as legitimate packages.
Attackers have employed various tricks to get users to install their malware for years. Package hallucinations are the latest tactic.
According to one of the researchers, Associate Professor Dr. Murtuza Jadliwala, director of the SPriTELab (Security, Privacy, Trust, and Ethics in Computing Research Lab): “So, let’s say I ask ChatGPT to help write some code for me and it writes it. Now, let’s say the generated code includes a link to some package, and I trust it and run the code, but the package does not exist, it’s some hallucinated package. An astute adversary/hacker could see this behaviour (of the LLM) and realise that the LLM is telling people to use this non-existent package, this hallucinated package. The adversary can then just trivially create a new package with the same name as the hallucinated package (being recommended by the LLM) and inject some bad code into it. Next time the LLM recommends the same package in the generated code and an unsuspecting user executes the code, this malicious package is downloaded and executed on the user’s machine,” Jadliwala explained.
The UTSA researchers evaluated the occurrence of package hallucinations across different programming languages, settings and parameters, exploring the likelihood of erroneous package recommendations and identifying root causes.
UTSA researchers conducted 30 different tests and discovered that 440,445 out of 2.23 million code samples generated in Python and JavaScript using LLM models referenced hallucinated packages. Among the LLMs tested, the study noted that “GPT-series models were found to be four times less likely to generate hallucinated packages compared to open-source models, with a hallucination rate of 5.2% versus 21.7%.” Researchers also found that Python code was less prone to hallucinations than JavaScript.
While cross-referencing generated packages with a master list may help mitigate hallucinations, the UTSA researchers contend that the best solution is to address the foundation of LLMs during their own development. The team has shared its findings with model providers, including OpenAI, Meta, DeepSeek, and Mistral AI.
The multi-institutional collaboration featured three additional researchers from UTSA: doctoral student A.H.M. Nazmus Sakib, postdoctoral researcher Raveen Wijewickrama, and Associate Professor Dr. Murtuza Jadliwala, director of the SPriTELab (Security, Privacy, Trust, and Ethics in Computing Research Lab). Additional collaborators were Anindya Maita from the University of Oklahoma (a former UTSA postdoctoral researcher) and Bimal Viswanath from Virginia Tech.
DOI: https://doi.org/10.48550/arXiv.2406.10279
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
