AI RESEARCH
On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding
arXiv CS.LG
•
ArXi:2510.15620v2 Announce Type: replace Semantic top-K selection with cross-encoder rerankers underpins on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation. However, its latency and memory demands dominate end-to-end budgets on edge hardware. Revisiting the objective of top-K selection, we reveal that only relative rankings matter, not exact per-candidate scores. We further observe sequence-level sparsity: relative rankings progressively stabilize in intermediate layers, enabling early pruning prior to completing full inference.