Kakao admits mistakes in server outage and shifts blame

Tech

Published: 24 Oct. 2022, 19:35 Updated: 25 Oct. 2022, 12:13

Kakao admits mistakes in server outage and shifts blame

Heads of Kakao, Naver and SK C&C swear to tell the truth before answering questions from lawmakers over a recent fire that broke out at an SK C&C data center and knocked out Kakao services for over 13 hours, during an audit held at the National Assembly on Monday. From left are: Kakao founder Kim Beom-su; Naver CEO Choi Soo-yeon; Naver founder Lee Hae-jin; and SK C&C CEO Park Sung-ha. [JOINT PRESS CORPS]

Kakao's CEO described to the National Assembly on Monday the rookie mistake that took down 32,000 of the company's servers and with them what has become critical national infrastructure. He also shifted blame for the outage to SK C&C, operator of the building in which the servers are housed.

"Although we had established backup servers, the tools that the developers needed to start the backups were not backed up and backup did not automatically kick in," Hong Eun-taek told the lawmakers.

The CEO did not elaborate further on the company's disaster recovery infrastructure, only saying that it has been investing as much as possible in backup since becoming profitable a few years ago. He also said he understood that to be internationally competitive, backup has to be greatly improved.

"We know that the data center is important enough to decide on whether we have a future or not,” Kakao founder Kim Beom-su said, in reference to a Kakao-owned site now under development.

On Oct. 15, Kakao servers were knocked out as a result of a battery fire at an SK C&C data center in Pangyo, Gyeonggi. Messenger, banking, transportation, payment and e-commerce services were unavailable, degraded or intermittent for more than 10 hours as backup systems failed to kick in fast enough. Naver's services hosted on servers in the same building recovered within three to four hours.

In comments made Monday to the lawmakers, Kim said that the owner of the facility, SK C&C, was late in informing Kakao of the fire.

The company has said that it was made aware of a fire at 4:03 p.m. on the afternoon of Oct. 15 when one of its employees called SK C&C, 44 minutes after batteries caught fire at 3:19 p.m.

“Around 16,000 servers had already been shut down before we were informed by SK C&C," Kim said.

Firefighters enter the SK C&C data center building located in Pangyo, Gyeonggi, on Oct. 15 to extinguish a fire that broke out on the afternoon. [NEWS1]

SK C&C claims it called a Kakao staff at 3:37 p.m., 18 minutes after the fire first started, and notified them of the situation.

Park Sung-ha, SK C&C CEO, told the National Assembly that company staff had notified all of its clients that the power would be shut off to control the fire.

Naver CEO Choi Soo-yeon said no such notification had been received.

Lawmakers asked the companies to submit data to the National Assembly to prove when notifications were made and received.

"We have always been aware of the importance of a data center, which is why we began preparing for our own data center since 2018," said Kim, founder of Kakao and 23.8 percent shareholder of the company. "However, it takes four to five years, and so we apologize for having failed to prepare it in advance, regardless of all other circumstances."

"The accident has clearly exposed our weaknesses, especially on having failed to secure our own data center and running a separate backup server," he said. "We will take into account everything that has been pointed out today and take measures so that the same situation does not reoccur in the future."

Kakao is working on the details of compensation and will cover users of all services, both free and paid, according to the Kakao founder.

“There is no precedent on compensation for free services anywhere in the world, so we will strive to come up with new standards after taking in specific cases from users and forming a committee,” Kim said.

BY YOON SO-YEON [yoon.soyeon@joongang.co.kr]