[please excuse cross-posting]
Hi Everyone, We have exciting news to share with you as part of our effort to support research on low resource machine translation: 1) we are releasing Flores 101, a large evaluation benchmark for multilingual machine translation in over 100 languages, 2) we are organizing a multilingual machine translation task at WMT and 3) we are supporting WMT participants to the large-scale multilingual task with compute credits that we encourage people to apply for. If you want to learn more, please read below for more details and feel free to respond to this email if you have any questions. We hope you’ll participate at this year’s WMT MMT task and leverage the upcoming Flores 101 in your future research, and that together as a community we can make collective progress on improving translation quality on low resource languages. Best regards, Paco Guzmán on behalf of the FLORES 101 Team flo...@fb.com Context Translation is a key technology to connect people and ideas together across language barriers. However, current translation technology works very well mostly in a few languages and it covers only a few domains. Many people around the world still lack access, partly, due to the lack of compute and data resources to create translation models. A prerequisite for developing new modeling techniques is having reliable evaluation. As a baby step in this direction back in 2019, we started FLORES<https://arxiv.org/abs/1902.01382>, which came with two evaluation datasets for Nepali-English and Sinhala-English, that we later expanded<https://github.com/facebookresearch/flores> to include Pashto and Khmer. Today, we announce the FLORES101 evaluation benchmark: a full Many-to-Many evaluation dataset across over 100 languages, most of which are low-resource. True to the original multi-domain spirit of FLORES, this dataset consists of 3000 English sentences across several domains (news, books, and travel) all taken from Wikipedia, maintaining document-level context as well as document metadata, such as topics, hyperlinks, etc. These sentences are then professionally translated though several rounds of thorough evaluation. You can see the full list of languages at the bottom of the page [here<http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html>]. We are making the entire dev and devtest splits of FLORES101 available to the research community (2000 sentences total, aligned Many to Many), in June 2021, along with a tech report describing the dataset in detail. To ensure robust and fair evaluation, we’ll keep the test split blind and not publicly accessible. Instead, we’ll host an evaluation server based on open-source code, which will enable us to track the progress of the community in low-resource languages. Importantly, such a setup will enable comparison of models on several axes besides translation quality, such as compute and memory efficiency. The evaluation server will also be available starting June 2021. Finally, we want to continue encouraging the research community to work on low-resource translation. As part of this, we are launching a WMT multilingual machine translation track and encourage people to apply for compute grants so that GPU compute is less of a barrier for translation research. You can see more detailed information and apply for the compute grant [here<http://www.statmt.org/wmt21/flores-compute-grants.html>]. We propose two small tracks --- one for low-resource European languages and another one for low-resource Southeast Asian languages --- along with the full track of 100+ languages.
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
