Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

ArXi:2604.27398v1 Announce Type: new For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real models. First, we note that mean pooling can collapse information beyond the first-order statistics of the token embeddings, such as second-order statistics that capture their spatial structure, potentially mapping distinct token embedding distributions to similar text embeddings.