Introduction to the Issue of Scams Using AI Deepfakes

Scams utilizing AI deepfakes of celebrities have increasingly become a prominent concern for Meta over the past couple of years. The Oversight Board has recently weighed in on this matter, seeming to confirm criticisms that Meta is not doing enough to enforce its own rules. This lack of enforcement makes it easier for scammers to carry out these schemes without significant repercussions.

The Oversight Board’s Conclusion

The Oversight Board concluded that Meta is likely allowing a significant amount of scam content to remain on its platforms to avoid potentially overenforcing a small subset of genuine celebrity endorsements. This decision was made following a case involving an ad for an online casino-style game called Plinko, which used an AI-manipulated video of Ronaldo Nazário, a retired Brazilian soccer player. Despite the ad showing obvious signs of being fake and being reported as a scam more than 50 times, Meta did not remove it until the Oversight Board agreed to review the case. The ad was viewed over 600,000 times.

Fundamental Flaws in Meta’s Approach to Content Moderation

The Oversight Board highlighted fundamental flaws in how Meta approaches content moderation for reported scams involving celebrities and public figures. Meta enforces its policy only when there’s an escalation to ensure the person depicted did not actually endorse the product. Additionally, individual reviewers’ interpretation of what constitutes a ‘fake persona’ can vary, introducing inconsistencies in enforcement. This results in a significant amount of scam content likely slipping through the cracks.

Recommendation to Meta

The Oversight Board recommended that Meta update its internal guidelines, empower content reviewers to identify such scams, and train them on indicators of AI-manipulated content. However, a spokesperson for Meta disputed the claims, stating that many of the Board’s assertions are inaccurate. Meta pointed to a test it began last year using facial recognition technology to fight "celeb-bait" scams as part of its efforts to combat scams.

Meta’s Response and Ongoing Efforts

In response to the Oversight Board’s recommendation, Meta stated it is testing the use of facial recognition technology, enforcing aggressively against scams, and empowering users to protect themselves through various on-platform safety tools and warnings. While appreciating the Oversight Board’s views, Meta considers many of the claims to be inaccurate and plans to respond to the full recommendation within 60 days as per the bylaws.

The Scale of the Problem

Scams using AI deepfakes of celebrities have become a major issue for Meta, exacerbated by the increasing accessibility and affordability of AI technology. Earlier this year, it was reported that dozens of pages were running ads featuring deepfakes of Elon Musk and Fox News personalities promoting supplements that claimed to cure diabetes. Despite some pages being disabled after reporting, similar scam ads persist on Facebook.

Public Figures Taking a Stand

Actress Jamie Lee Curtis publicly criticized Mark Zuckerberg for not removing a deepfaked Facebook ad that featured her, prompting Meta to remove the ad. The Oversight Board also noted the scale of the problem, finding thousands of video ads for the Plinko app, including ones featuring AI deepfakes of Cristiano Ronaldo and Mark Zuckerberg.

External Concerns and Reports

The Oversight Board is not alone in raising the alarm about scams on Meta’s platforms. The Wall Street Journal recently reported that Meta accounted for nearly half of all reported scams on Zelle for JPMorgan Chase between the summers of 2023 and 2024. British and Australian regulators have also found similar levels of fraud originating from Meta’s platforms, with the company being reluctant to add friction to its ad-buying process or ban advertisers with a history of scams.

Conclusion and Disclosure

In conclusion, the issue of scams using AI deepfakes on Meta’s platforms remains a significant concern, with both the Oversight Board and external entities like The Wall Street Journal highlighting the need for more effective measures to combat these scams. If you purchase something through a link in this article, we may earn a commission.