How Does Fourier Analysis Network Work? A Mechanism Analysis and a New Dual-Activation Layer Proposal

ArXi:2512.14873v3 Announce Type: replace Fourier Analysis Network (FAN) was recently proposed as a simple way to improve neural network performance by replacing part of Rectified Linear Unit (ReLU) activations with sine and cosine functions. Although several studies have reported small but consistent gains across tasks, the underlying mechanism behind these improvements has remained unclear. In this work, we show that only the sine activation contributes positively to performance, whereas the cosine activation tends to be detrimental.