Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

ArXi:2604.09709v1 Announce Type: cross Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation.