AI RESEARCH
MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design
arXiv CS.LG
•
ArXi:2412.14590v2 Announce Type: replace Quantization has become one of the most effective methodologies to compress LLMs into smaller size. However, the existing quantization solutions still show limitations of either non-negligible accuracy drop or low system efficiency. In this paper, we propose MixLLM that explores the optimization space of mixed-precision quantization between output features, based on the insight that different features matter differently in the model.