Qwen3.5-122B-A10B GPTQ Int4 on 4× Radeon AI PRO R9700 with vLLM ROCm: working config + real-world numbers

r/LocalLLaMA
Generative AI Open Source AI AI Tools

First, this not possible without u/djdeniro; u/sloptimizer and u/Ok-Ad-8976, where i learnt the recipes to start this. Hardware: 4× AMD Radeon AI PRO R9700 (32 GB each) with vLLM on a Gigabyte MC62-G40 + Threadripper Pro 5955WX, 6/8 dimm slots filled with 16gb ddr4 2133 rdimms - yes i bought off ebay and 2 were throwing ECs during burn-in. Big surprise: for my real 41k-context workflow, prefill was dramatically faster than llama.cpp. Measured result on one real task: - TTFT / prefill: 34.9 s - Total time: 101.7 s - vLLM reported about 4150 tok/s prompt throughput - basically blazing fast.