RAG - Sliding Window, Token Based Chunking and PDF Chunking Packages

Sliding Window Chunking Sliding Window Chunking is a intensive chunking mechanism. In this method, a window size is defined based on a character or token limit. Instead of creating completely separate chunks, the window moves forward gradually while keeping part of the previous content. The character or token limit is called the window size The amount the window moves forward each time is called the step size This is a stricter form of overlapping chunking. How it Works Suppose: Window size = 500 characters Step size = 100 characters The first chunk may contain characters 1-500.