The Children’s Commissioner for the UK is advocating for a prohibition on AI-powered deepfake applications that generate nude or sexual images of minors, as stated in a recent report. The report highlights that these “nudification” apps have become so widespread that many young girls have stopped sharing photos on social media platforms. Although creating or uploading CSAM images is against the law, apps used to create deepfake nude images remain legal.

Children’s Commissioner Dame Rachel de Souza expressed that “children have informed me that they are terrified by the mere existence of this technology, let alone its potential use. They fear that anyone — a stranger, a classmate, or even a friend — could utilize a smartphone to manipulate them by creating a naked image using these customized apps.” De Souza emphasized, “There is no positive justification for the existence of these apps.”

De Souza pointed out that nudification AI apps are readily available on mainstream platforms, including the largest search engines and app stores. At the same time, they “disproportionately target girls and young women, and many tools appear to only work on female bodies.” She added that young people are demanding action to be taken against the misuse of such tools.

To address this issue, de Souza is urging the government to introduce a total ban on apps that utilize artificial intelligence to generate sexually explicit deepfakes. She also wants the government to establish legal responsibilities for GenAI app developers to identify the risks their products pose to children, create effective systems to remove CSAM from the internet, and recognize deepfake sexual abuse as a form of violence against women and girls.

The UK has already taken steps to ban such technology by introducing new criminal offenses for producing or sharing sexually explicit deepfakes. It also announced its intention to make it a criminal offense if a person takes intimate photos or videos without consent. However, the Children’s Commissioner is focused more specifically on the harm such technology can cause to young people, noting that there is a link between deepfake abuse and suicidal ideation and PTSD, as The Guardian pointed out.

“Even before any controversy arose, I could already tell what it was going to be used for, and it was not going to be good things. I could already tell it was going to be a technological wonder that’s going to be abused,” said one 16-year-old girl surveyed by the Commissioner.

In the US, the National Suicide Prevention Lifeline is 1-800-273-8255 or you can dial 988. The Crisis Text Line can be reached by texting HOME to 741741 (US), 686868 (Canada), or 85258 (UK). Wikipedia maintains a list of crisis lines for individuals outside of those countries.