Selecting data for multilingual multi-domain neural machine translation on low resource languages

Abstract

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT REQUEST OF AUTHOR.] While machine translation has achieved impressive results on the world's most widely spoken languages, thousands of languages do not have the quantity of data necessary to train a state-of-the-art system. We propose here a technique to identify the best available datasets for augmentation in many-to-one multilingual neural machine translation systems by quantifying the factors that most affect translation performance - data set domain, relation between source side languages, translation quality, and data set size. Previous research has considered these factors qualitatively and in isolation of each other, but selecting an augmenting data set from various possibilities requires a quantitative synthesis of all these factors. We evaluate a number of techniques to measure each of these factors and learn a system combining them. The focus is on the Luyia languages of western Kenya as a case study for an extreme low resource scenario, but the application of these techniques to similar languages is also explored.