AI RESEARCH
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
arXiv CS.CL
•
ArXi:2604.26294v1 Announce Type: new We present tensor and sequence parallelism (TSP), a parallel execution strategy that folds tensor parallelism and sequence parallelism onto a single device axis. In conventional multi-dimensional parallelism layouts, tensor parallelism (TP) shards model weights while sequence parallelism (SP) shards tokens, reducing per-device parameter or activation memory, respectively. Traditionally, each scheme is assigned its own mesh dimension.