llama.cpp Gemma 4 using up all system RAM on larger prompts

Something I'm noticing that I don't think I've noticed before. I've been testing out Gemma 4 31B with 32GB of VRAM and 64GB of DDR5. I can load up the UD_Q5_K_XL Unsloth quant with about 100k context with plenty of VRAM headroom, but what ends up killing me is sending a few prompts and the actual system RAM fills up and the process gets terminated for OOM, not a GPU or CUDA OOM, like Linux killing it because llama.cpp was using 63GB of system