Using image embeddings as input for new image generation, basically “embedding2image” / IP-Adapter?

Hi everyone, I have a question before I start digging too deeply into this. I have some images that I really like, but images that come out of the Stable Diffusion universe (photo, etc.). What I would like to do is use those images as the starting point for generating new ones, not in an img2img pixel-to-pixel way, but as a semantic / stylistic input. My rough idea was something like: take an image I like encode it into an embedding use that embedding as input conditioning for a new generation So in my mind it is a bit like “embedding2image.