Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers
Hugging Face table of contents What is NeMo Automodel? Supported diffusion models What this collaboration unlocks A look at the fine-tuning workflow 1.
Pre-encode the dataset 2. Launch training with the existing FLUX YAML 3. Generate from the fine-tuned checkpoint 4. Performance Other Finetuned/LoRA examples Try it today Coming next: Pythonic recipe APIs Resources A joint post from NVIDIA and Hugging Face. Special thanks to Sayak Paul from Hugging Face for their contributions to the integration work and for co-authoring this blog. Diffusion models power some of the most exciting open-source releases of the last two years — such as FLUX.1-dev for text-to-image and Wan 2.1 and HunyuanVideo for text-to-video. The 🤗 Diffusers library has become the de facto home for these models, giving researchers and builders a single, consistent interface for inference, adaptation, and pipeline composition. In addition, training and fine-tuning diffusion models are also on the rise, requiring utilities that offer memory-efficient sharding, latent caching, multiresolution bucketing, and configurations that scale gracefully from one GPU to hundreds. To cater to these technical demands, we offer the NVIDIA NeMo Automodel open-source library. Today, we’re highlighting the collaboration between NVIDIA and Hugging Face that brings production-grade, distributed diffusion training to any Diffusers-format model on the Hugging Face Hub — with no checkpoint conversion and no model rewrites for any new model. The integration is documented in the Diffusers training guide and is fully open source under Apache 2.0. NeMo Automodel is an open-source PyTorch DTensor-native training library, part of the NVIDIA NeMo framework, built around two design principles that matter for the Diffusers ecosystem: AutoModel currently supports flow-matching models only. Under the hood, it uses flow matching as the training objective, with latent-space training (via pre-encoded VAE outputs) and multiresolution bucketed dataloading to accelerate throughput. NeMo Automodel integration ships with ready-to-use fine-tuning recipes for the open diffusion models below. The list reflects the recipes currently in examples/diffusion/finetune.
