How do I get character consistency without a LoRA?

Hey, I’m pretty new to local AI image generation and I’m trying to figure something out. I want to use SDXL/NoobAI/Flux to generate images of a historical figure, and combine that with a LoRA style from Civitai. The problem is I can’t keep the face consistent. Every time I generate an image, the face looks completely different, and I can’t get it to match the original person or even stay similar between generations. I have tried IP-Adapter Face but it did not work and I don't know why. Not sure what I’m doing wrong or how people manage to keep characters consistent.