AI RESEARCH
Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
arXiv CS.CL
•
ArXi:2406.10985v2 Announce Type: replace Large language models (LLMs) have shown promising efficacy across various tasks, becoming powerful tools in numerous aspects of human life. However, Transformer-based LLMs suffer a performance degradation when modeling long-term contexts due to they discard some information to reduce computational overhead. In this work, we propose a simple yet effective method to enable LLMs to take a deep breath, encouraging them to summarize information contained within discrete text chunks.