Some Kakao servers still offline four days after Pangyo fire

Tech

Published: 18 Oct. 2022, 17:00 Updated: 18 Oct. 2022, 18:55

Some Kakao servers still offline four days after Pangyo fire

Lee Jong-ho, Science and ICT minister, bows while apologizing to the public for the inconvenience caused by Kakao’s service crash while appearing at the National Assembly on Tuesday in Daejeon. [YONHAP]

Most Kakao services have been restored and 70 percent of the company's servers at the site where a fire broke out on Saturday are now working.

The company remains mum on its backup infrastructure and whether it had a "hot site" in another location, the global standard in a disaster recovery.

Kakao servers were knocked out as a result of a fire at an SK C&C data center in Pangyo, Gyeonggi, that started Saturday afternoon. The company's messenger services and services offered by related companies were unavailable, degraded or intermittent as backup systems proved slow and ineffective.

The company's messenger — the ubiquitous KakaoTalk — mobility, navigation, music and TV streaming services have been restored, according to the Ministry of Science and ICT and Kakao on Tuesday.

Email and the KakaoTalk Channel, an e-commerce service, have yet to be fully restored. The company said it is aiming to have the email back online by the end of the day Tuesday.

It warned that even when all services are back, delays could be experienced due to heavy traffic.

Naver also had servers in the building. By Tuesday, all its services were running normally, though images were failing to load on some blogs.

The fire in the building was limited to a basement-level room with a rack of batteries for backup power, so the rest of the building is free of damage and debris. The main issue is the power.

According to SK C&C, power is back on in 95 percent of the building. It adds that Kakao servers are concentrated in the part of the building without power.

According to Kakao, 30 percent of its 32,000 servers have yet to be powered up, causing the delay in the restoring of e-mail and commerce-related services.

“There are two particular areas where the power isn't supplied,” said an SK C&C official. “And that’s where the Kakao servers are concentrated.”

Koreans are claiming to have suffered damages as a result of the outage, with some people paying a dear price for their reliance on one service. One person claimed the loss of a job opportunity due to a missed coding test scheduled through KakaoTalk.

“I just called them and received a reply that there is no retest,” the person posted.

Bakers, hairdressers and other operators of small business say they are seeking to take legal action against the company.

The Korea Federation of Micro Enterprise is collecting complaints and claims of damages. As of Tuesday afternoon, over 400 businesses have filed their cases.

Seoul city government is offering legal consulting for businesses affected.

“We are currently surveying the damage in working with various associations including those for restaurants and taxis,” Mayor Oh Se-hoon said via Facebook on Tuesday. “The Seoul government will provide legal consulting in cooperation with associations of businesses affected.”

Kakao’s headquarters in Pangyo, Gyeonggi. Kakao services crashed on Saturday after a fire broke out at SK C&C where the company’s servers are housed. Despite continuous efforts, some of Kakao services struggled to recover on the fourth day of the incident. [YONHAP]

SK C&C insurance has 7 billion won ($4.9 million) in errors and omissions liability insurance for technology-related damage, but this might not help customers.

It "only covers Kakao, the tenant of the data center, and it is unlikely to compensate for losses caused on Kakao users that have been indirectly affected,” an insurance company executive said.

In a statement released Monday, Kakao said it will discuss compensating Kakao users and clients once its services are normalized.

Kakao suggested that SK C&C, which operates the center, will be paying some of the damages and said that the event will have little effect on its performance.

Analysts estimate the loss caused by the Kakao crisis between 15 and 22 billion won.

The company has yet to detail its disaster recovery plan and to say whether it had mirror servers operating in other locations to act as fallbacks in the event of a major failure.

Kakao rose 2.17 percent Tuesday following a sharp decline on Monday. Kakao Pay rose 6.21 percent, Kakao Bank 3.61 percent and Kakao Games 4.81 percent.

BY LEE HO-JEONG, KWEN YU-JIN HAN JEE-HYE [lee.hojeong@joongang.co.kr]