Residual connections haven't changed for 10 years and Kimi just replaced them with attention

In standard residual connections, each layer simply adds its output to the sum of all previous layers with equal weight, no selectivity at all. Attention Residuals replaces this with a softmax attention mechanism: each layer gets a single learned query vector that attends over all previous layer outputs, producing input-dependent weights that let the layer selectively retrieve what it actually needs. On scaling law experiments, Block AttnRes achieves the same loss as a baseline trained with 1.25x compute.