Cybersecurity researchers have uncovered two malicious machine learning (ML) models on Hugging Face that utilized an unusual technique of “broken” pickle files to evade detection.
“The pickle files extracted from the mentioned PyTorch archives revealed the malicious Python content at the beginning of the file,” ReversingLabs researcher Karlo Zanki said in a report shared with The Hacker News. “In both cases, the malicious payload is a typical platform-aware reverse shell that connects to a hard-coded IP address.”
The approach has been dubbed nullifAI, as it involves clear-cut attempts to sidestep existing safeguards put in place to identify malicious models. The Hugging Face repositories have been listed below:
- glockr1/ballr7
- who-r-u0000/0000000000000000000000000000000000000
The pickle serialization format, used commonly for distributing ML models, has been repeatedly found to be a security risk, as it offers ways to execute arbitrary code as soon as they are loaded and deserialized.
Further analysis has revealed that such broken pickle files can still be partially deserialized owing to the discrepancy between Picklescan and how deserialization works, causing the malicious code to be executed despite the tool throwing an error message. The open-source utility has since been updated to rectify this bug.
“The explanation for this behavior is that the object deserialization is performed on Pickle files sequentially,” Zanki noted.
“Pickle opcodes are executed as they are encountered, and until all opcodes are executed or a broken instruction is encountered. In the case of the discovered model, as the malicious payload is inserted at the beginning of the Pickle stream, execution of the model wouldn’t be detected as unsafe by Hugging Face’s existing security scanning tools.”
