Late stars rest in virtual reality

Television

Published: 16 Dec. 2020, 16:07 Updated: 16 Dec. 2020, 16:54

Late stars rest in virtual reality

An AI-generated image of late singer Turtle Man of trio Turtles appears on stage. [MNET]

Technology may not be able to bring the dead back to life, but it can offer the living a chance to reminisce on the past.

Television music programs are making the most of the finest technology to recreate performances by late legendary singers such as the late Kim Hyun-sik, Kim Kwang-seok and Shin Hae-chul.

On Dec. 9, cable channel Mnet aired the first episode of a two-part program “One More Time” featuring images generated by artificial intelligence (AI) of the late Turtle Man, a singer from the trio Turtles, who died of a heart attack in 2008. For the show, producers gathered data of the singer's voice and gestures and then used face editing technology to create a performance with the two other members of the group.

An AI-generated image of late singer Turtle Man of trio Turtles performs on stage. [MNET]

Standing in the center of the stage, Turtle Man sang the words “Start again […] You’re the main character of our life,” performing choreography along with his former bandmates Geum Bi and Z-E beside him. Together, the three sang a rearranged version of singer Gaho’s song “Start Over.”

Hologram images of stars had already been used, such as when rapper 2Pac appeared at the Coachella Festival in 2012 and Michael Jackson appeared at the 2014 Billboard Music Awards. But the latest AI technology goes beyond a simple reenactment and utilizes it to create new — such as Turtle Man singing a song he would have never even heard.

“We gathered photographs and videos of Turtle Man in his daily life as well as on stage, then used the data to recreate a natural and lively expression using Generative Adversarial Network [GAN] technology,” said CJ ENM, the operator of Mnet. “We wanted to deliver a sense of consolation through the marriage of technology and story.”

The second episode of the series aired on Wednesday, featuring singer Kim Hyun-sik, to commemorate the 30th anniversary of his death.

SBS will reveal its four-part program ″Match of the Century! AI vs Human″ next month, starting with an AI-impersonation of late singer Kim Kwang-seok. [SBS]

Above is a picture of late singer Kim Kwang-seok. SBS will reveal its four-part program ″Match of the Century! AI vs Human″ next month, starting with an AI-impersonation of Kim. [SBS]

Channel SBS’s upcoming four-part special series “Match of the Century! AI vs Human” takes on a similar context. For the teaser of the first episode, which will air in January, SBS revealed an AI program using the voice of Kim Kwang-seok to sing Kim Bum-soo’s “I Miss You” (2002) on Dec. 1. The following episodes of the program will feature other advancements of AI that have been made over the past years.

“I was so shocked to see how much AI has developed since we aired [the match between] AlphaGo and Lee Se-dol in 2016 and Handol in 2019,” said SBS. “We have prepared [AI] in six fields, including fields where it can generate precise results such as a match between [golfer] Park Se-ri and AI golfer Ldric [based on Tiger Woods] who come face-to-face for long drives and putts, to other fields that are hard to put a score on, such as songwriting, psychology and audio montage.”

AI company Supertone, which took part in both the Mnet and SBS show, said that voice restoration is like making a musical instrument. Along with text-to-speech (TTS) technology and the information of the musical notes as well as a singer’s voice, the AI can be taught to sing any song the way that the original singer would have sung it.

“It’s possible to make AI speakers read written texts such as the news, but there has been difficulty bringing out emotional expressions,” said Supertone. “But after three years of research, we were awarded the best article award at the IEEE International Conference on Acoustics, Speech and Signal Processing (Icassp) Interspeech last year and commercialized a singing voice synthesis system for the first time in the world. It can not only sing, but act as well, so we are working to extend it to virtual influencers.”

Technological advancements have also made it possible for celebrities to be replaced when they cannot be present on stage.

BTS performs its song ″Life Goes On″ during the 2020 MAMAs, where an AI-generated image of member Suga performed in his place. [MNET]

On Dec. 6 at the Mnet Asian Music Awards (MAMAs), Suga of BTS could not attend as he was recovering from shoulder surgery. In his stead, a “virtual” Suga was generated using volumetric technology to fill his place during the band’s performance of “Life Goes On.” His “presence” was based on video data filmed by multiple cameras from different angles.

“We shot Suga ourselves, extracted the 3-D data, erased noises and then cleaned it again to make his image look the same as the other members,” said Vive Studios, which took part in the making of Suga’s image for the MAMAs.

Vive Studios began as a computer graphics and visual effects company. It first garnered major attention after unveiling a digital persona of Kang Na-yeon who passed away in 2016 at the age of just 7. During a documentary titled "Meeting You," which aired on MBC in February this year, Kang's mother was reunited with her child through virtual reality.

Online concerts have become essential this year due to Covid-19 and often showcase the latest technological advancements. In October, BTS held its online concert using the best of AR (augmented reality) and XR (extended reality) and is expected to excel once again during the “New Year’s Eve Live” concert on Dec. 31, where it will stand with other bands from the Big Hit family. The concert will also include a tribute stage to the late Shin.

SM Entertainment, another leader in K-pop, has also been actively pioneering the combining of culture and technology. In 2013, it held a virtual concert for Girls’ Generation, a hologram musical of “School Oz” in 2015 and launched its first online concert series “Beyond LIVE” in April. Last month, it debuted its girl group aespa, which is made up of human members and their digital avatars.

But using the latest technology also comes with the latest dangers of misuse or overuse.

“Because we are using the voices of real people, we can only do this with the consent of the person or the family of the deceased,” said Supertone. “We are only trading with companies and not individuals to prevent any leak or misuse.”

BY MIN KYUNG-WON [yoon.soyeon@joongnang.co.kr]