Exams aren’t just for humans VCG via Getty Images
The results are in. An artificial intelligence has gone to the top of its class after passing an English exam. Though it can鈥檛 beat more able human students, it achieved the best mark yet for a machine.
Hai Zhao at Shanghai Jiao Tong University in China and his colleagues trained their AI on more than 25,000 English reading comprehension tests.
Each contained a 200 to 300-word story followed by a series of related multiple-choice questions. The tests were sourced from English proficiency exams aimed at Chinese students aged from 12 to 18 years.
Advertisement
While some answers could be directly found in the text, over half of them required a degree of reasoning. For example, one of the questions asked you to choose the best headline for a story from four options.
After the training, the AI sat a final听exam consisting of 1400听tests it听hadn鈥檛 seen before. It achieved an overall score of 74听per cent, better than .
Zhao鈥檚 AI uses a system that can identify parts of the story that are relevant to the question, then selects the answer that is most similar in meaning and logic.
The next听best was a system made by听Tencent, a听leading Chinese technology firm, which scored 72听per听cent on the same exam. Tencent鈥檚 AI learned to compare the information carried by each option and use their differences as cues to look for evidence in the text.
Despite topping the leader board, Zhao is determined to improve his system鈥檚 abilities. 鈥淲hat our AI got is very average, a C+ at most,鈥 he says. 鈥淔or students who want to get into good universities in China, they will aim for 90听per cent.鈥
To increase its score, the team will try to modify the AI so that it can understand information embedded in sentence structure and feed it with more data to expand its vocabulary.
Understanding human language is a major headache for AI, as it is often imprecise and involves hidden contextual and societal clues that machines struggle to pick up on.
It is unclear what rules AIs follow when they learn our languages, says听Guokun Lai at Carnegie Mellon University in Pennsylvania, who originally collated the tests in 2017 for听AI research. 鈥淭hey seem to be able听to [understand our logic] after听reading tonnes of sentences and听stories.鈥
Reference: arXiv,听arxiv.org/abs/1901.09381
Topics:
