Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings

ArXi:2110.02887v1 Announce Type: cross Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer.