AI RESEARCH
Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
arXiv CS.CV
•
ArXi:2510.07143v3 Announce Type: replace Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectiveness of these methods is commonly evaluated by measuring the accuracy drop on existing MLLM benchmarks before and after compression. However, these benchmarks are originally designed to assess general perception and reasoning abilities, rather than the specific challenges posed by visual token compression, leading to a fundamental task mismatch.