Generative AI drives surge in deepfake porn

Social Affairs

Generative AI drives surge in deepfake porn

Published: 30 Aug. 2024, 16:44 Updated: 30 Aug. 2024, 16:47

CHO JUNG-WOO
cho.jungwoo1@joongang.co.kr

In an image generated by ChatGPT, anonymous users are depicted using Telegram. [CHATGPT]

Generative AI, which automatically creates synthetic photos and videos, is fueling the spread of deepfake pornography amid intensified efforts in Korea to eradicate such digital sex crimes.

So-called deepfake bots, which can create illegal synthetic photos and videos in just 30 seconds, were easily found on Telegram.

On the platform, which offers end-to-end encryption, a search for keywords like "deepfake" on Thursday revealed hundreds of channels and chats promoting generative AI features that create such illegal content.

When pressing the start button, a prompt quickly followed, instructing the user to send a photo of a woman, saying that "magic will happen" in just 30 seconds. After uploading an image generated by ChatGPT, a face synthesized onto a nude body appeared in just 30 seconds. The service also offered options to adjust the appearance of specific body parts and change outfits.

After the bot offered complimentary synthesized photos, it prompted users to make a payment. Prices started at $4.9, charging 49 cents per photo. Higher payments unlocked additional features, such as watermark removal and enhanced photo resolution. Payments were made via cryptocurrency.

Some deepfake bots required users to subscribe to other similar channels, while others offered "cash" rewards for inviting new users.

Users on Telegram share links to deep fake bots. [SCREEN CAPTURE]

Each of these channels and chatrooms had at least hundreds to tens of thousands of participants. One channel, open to the public, had gathered around 20,000 members.

Only channel creators and administrators could send messages on these channels, which were often linked to other chatrooms sharing illegal videos.

In response to the surge in deepfake bots, police have begun investigating eight channels on the platform.

"We are using various methods to track down these services," a police official said, adding that the number of deepfake bots under investigation could be more.

Seo Hye-jin, the human rights director of the Korean Women Lawyers Association, noted that it is impossible to count the number of such channels as they "constantly form and disappear."

"The issue lies in how these illegally synthesized photos and videos are used, given that the intent behind producing this content varies," Seo said.

Some experts argue that Telegram's introduction of a reward system, which provides ad revenue to channel owners based on the number of subscribers, has contributed to the misuse of these channels for profit.

"Telegram shares ad revenue with channel owners who have more than 1,000 subscribers," said Kim Myung-joo, an information security professor at Seoul Women's University. "Owners receive 50 percent of the ad revenue."

Seo also pointed out that the profit-sharing system has created an incentive structure akin to a Ponzi scheme, encouraging users to bring in more members to maximize profits.

BY KIM SEO-WON, CHO JUNG-WOO [cho.jungwoo1@joongang.co.kr]