Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

ArXi:2406.03736v4 Announce Type: replace Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form.