AI RESEARCH
SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training
arXiv CS.LG
•
ArXi:2601.23155v2 Announce Type: replace Information-based data selection for instruction tuning is compelling: maximizing the log-determinant of the Fisher information yields a monotone submodular objective, enabling greedy algorithms to achieve a $(1-1/e)$ approximation under a cardinality budget. In practice, however, we identify alleviating gradient conflicts, misalignment between per-sample gradients, is a key factor that slows down the decay of marginal log-determinant information gains, thereby preventing significant loss of information.