Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion

ArXi:2605.04291v1 Announce Type: new We present a discrete diffusion-based language model using Glauber dynamics from statistical physics. Our main insight is that instead of trying to train a discrete state space diffusion model using Glauber dynamics with a uniform transition kernel as the forward process, one can set up an ``energy function'' based on pretrained causal/masked language models. When viewed as the stationary distribution, this energy function allows us to significantly improve the quality of the generated text.