The frequency and sophistication of malicious projects being uploaded to Hugging Face and other open-source artificial intelligence (AI) model repositories are increasing, often evading the sites’ security checks. This escalating issue highlights the need for companies engaging in internal AI projects to implement robust mechanisms for detecting security flaws and malicious code within their supply chains.
A recent analysis by ReversingLabs, published on February 3, revealed that Hugging Face’s automated checks failed to detect malicious code in two AI models hosted on the repository. The threat actor utilized a common vector, data files in the Pickle format, with a novel technique dubbed “NullifAI,” to evade detection.
Although the attacks appeared to be proofs-of-concept, their success in being hosted with a “No issue” tag indicates that companies should not solely rely on Hugging Face’s and other repositories’ safety checks for their own security, according to Tomislav Pericin, chief software architect at ReversingLabs.
“You have this public repository where any developer or machine learning expert can host their own content, and malicious actors are exploiting that,” he says. “Depending on the ecosystem, the vector will vary, but the idea remains the same: someone will host a malicious version of something, hoping you’ll inadvertently install it.”
The rapid adoption of AI by companies, with the majority establishing internal projects using open-source AI models from repositories like Hugging Face, TensorFlow Hub, and PyTorch Hub, underscores the importance of security. According to a Morning Consult survey of 2,400 IT decision-makers sponsored by IBM, 61% of companies are utilizing models from the open-source ecosystem to create their own AI tools.
However, many components can contain executable code, leading to various security risks, including code execution, backdoors, prompt injections, and alignment issues. The latter refers to how well an AI model aligns with the intent of developers and users.
In an Insecure Pickle
A significant issue is the widespread use of the Pickle file format, which is not secure and can be used to execute arbitrary code. Despite warnings from security researchers, the Pickle format continues to be used by many data scientists, according to Tom Bonner, vice president of research at HiddenLayer, an AI-focused detection and response firm.
“I had hoped that we’d made enough noise about it that Pickle would’ve been phased out by now, but it’s not,” he says. “I’ve seen organizations compromised through machine learning models — multiple organizations, in fact. While it’s not an everyday occurrence like ransomware or phishing campaigns, it does happen.”
Hugging Face has explicit checks for Pickle files, but the malicious code discovered by ReversingLabs sidestepped these checks by using a different file compression for the data. Other research by application security firm Checkmarx found multiple ways to bypass the scanners, such as PickleScan used by Hugging Face, to detect dangerous Pickle files.
Despite having malicious features, this model passes security checks on Hugging Face. Source: ReversingLabs
“PickleScan uses a blocklist which was successfully bypassed using both built-in Python dependencies,” Dor Tumarkin, director of application security research at Checkmarx, stated in the analysis. “It is plainly vulnerable, but by using third-party dependencies such as Pandas to bypass it, even if it were to consider all cases baked into Python, it would still be vulnerable with very popular imports in its scope.”
Instead of Pickle files, data science and AI teams should adopt Safetensors — a library for a new data format managed by Hugging Face, EleutherAI, and Stability AI — which has been audited for security. The Safetensors format is considered much safer than the Pickle format.
Deep-Seated AI Vulnerabilities
Executable data files are not the only threat; licensing is another issue. While pretrained AI models are often referred to as “open source AI,” they usually do not provide all the necessary information to reproduce the AI model, such as code and training data. Instead, they provide the weights generated by the training and are covered by licenses that may not be open source compatible.
Creating commercial products or services from such models can potentially result in violating the licenses, says Andrew Stiefel, a senior product manager at Endor Labs. “There’s a lot of complexity in the licenses for models,” he says. “You have the actual model binary itself, the weights, the training data, all of which could have different licenses, and you need to understand what that means for your business.”
Model alignment — how well its output aligns with the developers’ and users’ values — is another wildcard. For instance, DeepSeek allows users to create malware and viruses, researchers found. Other models, such as OpenAI’s o3-mini model, which boasts more stringent alignment, have already been jailbroken by researchers.
These problems are unique to AI systems, and the boundaries of how to test for such weaknesses remain a fertile field for researchers, says ReversingLabs’ Pericin. “There is already research about what kind of prompts would trigger the model to behave in an unpredictable way, divulge confidential information, or teach things that could be harmful,” he says.
Companies should ensure they understand any licenses covering the AI models they are using. Additionally, they should pay attention to common signals of software safety, including the source of the model, development activity around the model, its popularity, and the operational and security risks, Endor’s Stiefel says.
“You kind of need to manage AI models like you would any other open source dependencies,” Stiefel says. “They’re built by people outside of your organization, and you’re bringing them in, so that means you need to take that same holistic approach to looking at risks.”
Source Link
