AI RESEARCH
Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
arXiv CS.CL
•
ArXi:2508.20474v2 Announce Type: replace-cross This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint.