Machines no match for humans in translations
Human translators proved their superiority in the first translation competition with artificial intelligence programs here on Tuesday. Their translations were significantly better in terms of quality and accuracy.
“We have proven that - for a while at least - artificial intelligence cannot surpass the ability of human translators in translating from Korean to English and from English to Korean,” said Kwak Jung-chul, president of the Korean Association of Translators and Interpreters, who headed the evaluation committee for the competition.
“Things may change when these programs develop further with more data, but it seems that human translators passionate about their careers will persevere and remain in this field.”
In the competition hosted by the International Interpretation and Translation Association (IITA), Sejong University and Sejong Cyber University at Sejong University’s campus in eastern Seoul, three artificial intelligence translation programs - Naver’s Papago, Google Translate and Systran - competed in translating against four human translators.
The four humans, professional translators with at least five years of experience, were given 50 minutes to translate one work each. They translated one literary work and one non-literary work from Korean to English and the same from English to Korean. Each was under 250 words.
Each of the three programs translated all four of the works in 10 minutes. The artificial intelligence programs relied on its data in their systems, which are connected to the internet, and the humans were allowed to access the internet also while translating. None of the works have ever been translated before and therefore their translations were not available online, according to the organizers.
Each translation was scored by Kwak and two professional translators from his association.
The scoring rubric included three categories: accuracy, language expression, and logic and organization. Each category was awarded up to five points, the perfect score being 15 per work.
“In putting together the total score for all four works,” Kwak said, “the artificial intelligence program that scored highest among the three scored 28 out of 60, and the human translators scored 49 out of 60.”
Kwak did not disclose the specific scores per artificial intelligence translation program over concerns that this might “damage the program’s business.”
The works translated were a Fox Business News article, “How a Movie Propelled Lego Back to the World’s Most Powerful Brand”; a section of Thomas L. Friedman’s “Thank You for Being Late”; a section of an opinion piece written by writer Kim Seo-ryung; and a section of “Mothers and Daughters,” a novel by Kang Kyeong-ae.
“Some 90 percent of the artificial intelligence programs’ translations had grammatical errors,” Kwak said. “The programs failed to understand the message that writers were trying to convey.”
For example, one artificial intelligence program translated a sentence from the Friedman book - “Steve reached into the top pocket of his jeans and pulled out the first iPhone” - into Korean that read, “Steve went into his jean’s pocket and pulled out the first iPhone.”
Developers of the programs were positive despite the results.
“We believe that the capacity of the translation program at Systran is at some 85 to 90 percent of human translators’ capacity. But when it comes to translating certain nuances and emotions conveyed by the sentences, the programs could fall short,” said Kim Yoo-suk, chief strategy officer at Systran, a U.S.-founded international translation company now headquartered in Paris.
“But ultimately, artificial intelligence programs for translation can help professional interpreters and translators with their work.”
BY ESTHER CHUNG, KIM JEE-HEE [firstname.lastname@example.org]