AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

ArXi:2603.09982v1 Announce Type: cross Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens.