best option for chunking data

Large body of text, multiple files, inconsistent format. llms seem to be hit or miss when it comes to chunking. is there a application that I don't know about that can make it happen? the text is academic medical articles with tons of content. I want to chunk it for embedding purposes submitted by /u/Immediate_Occasion69 [link] [comments]