A Survey on Continuous Diffusion and Flow Language Models
Continuous diffusion and flow language models generate text by transforming noise into continuous representations of tokens. They can refine many positions in parallel while preserving dependencies between them, and their continuous states allow generation to be steered using gradients of a differentiable objective. Joint generation is especially useful in the few-step regime, where standard parallel discrete-diffusion samplers can incur a factorization error by sampling multiple tokens conditionally independently. Recent advances have made continuous models competitive with discrete diffusion on language benchmarks and have enabled scaling to billions of parameters. This survey explains the principles behind these advances and organizes the field by how models represent text, learn generative dynamics, and generate samples. We examine the methods that enable few-step generation, including flow maps and distillation, and connect them to developments in image and video generation. We then review scaling, applications, and evaluation, and identify open questions in representation learning, scaling, few-step distillation, post-training, multimodality, and safety.
Continuous diffusion and flow language models generate text by transforming noise into continuous representations of tokens. They can refine many positions in parallel while preserving dependencies between them, and their continuous states allow generation to be steered using gradients of a differentiable objective. Joint generation is especially useful in the few-step regime, where standard parallel discrete-diffusion samplers can incur a factorization error by sampling multiple tokens conditionally independently. Recent advances have made continuous models competitive with discrete diffusion on language benchmarks and have enabled scaling to billions of parameters. This survey explains the principles behind these advances and organizes the field by how models represent text, learn generative dynamics, and generate samples. We examine the methods that enable few-step generation, including flow maps and distillation, and connect them to developments in image and video generation. We then review scaling, applications, and evaluation, and identify open questions in representation learning, scaling, few-step distillation, post-training, multimodality, and safety.