AI RESEARCH
TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
arXiv CS.CV
•
ArXi:2605.07256v1 Announce Type: new Transformer architecture search (TAS) discovers optimal vision transformer (ViT) architectures automatically, reducing human effort to manually design ViTs. However, existing TAS methods suffer from the feature collapse problem, where subnets within a supernet fail to learn subnet-specific features, mainly due to the shared weights in a supernet, limiting the performance of individual subnets. To address this, we propose TAS-LoRA, a novel method that