NVIDIA NeMo Automodel integrates with Hugging Face Diffusers for distributed video and image model fine-tuning

NVIDIA and Hugging Face have integrated NeMo Automodel with the Diffusers library, enabling production-grade distributed training for diffusion models without checkpoint conversion. The integration, released under Apache 2.0, allows users to fine-tune any Diffusers-format model directly from the Hugging Face Hub.

Technical capabilities

NeMo Automodel is a PyTorch DTensor-native training library that uses Diffusers model classes for loading and pipelines for generation. The integration eliminates the need to convert checkpoints between training and inference formats—fine-tuned weights load directly into DiffusionPipeline.

The library implements multiple parallelism strategies as configuration options rather than code rewrites: FSDP2, tensor parallel, expert parallel, context parallel, and pipeline parallel. It includes memory-efficient sharding, latent caching, and multiresolution bucketing for training at scale.

Supported models

The integration ships with fine-tuning recipes for six model families:

• Wan 2.1 T2V: 1.3B (fits single 40GB A100) and 14B parameter versions for text-to-video
• Wan 2.2 T2V A14B: 27B total parameters (MoE), 14B active per step
• FLUX.1-dev: 12B parameters for text-to-image
• FLUX.2-dev: 32B parameters for text-to-image
• HunyuanVideo 1.5: 13B parameters for text-to-video
• Qwen-Image: 20B parameters (MMDiT) for text-to-image

All models except Wan 2.2 include LoRA recipe support for parameter-efficient fine-tuning.

Training workflow

The workflow consists of three steps:

1. Pre-encode dataset: Cache VAE latents and text embeddings using distributed preprocessing across GPUs. For the 78-image Rider-Waite tarot dataset example, preprocessing outputs .pt cache files and sharded metadata.

2. Launch training: Use existing YAML configs with command-line overrides for dataset-specific settings. The example fine-tunes FLUX.1-dev with eight-way FSDP2 and an effective batch size of 32.

3. Generate from checkpoint: Fine-tuned models load directly into Diffusers pipelines without conversion.

The library currently supports flow-matching models only, using flow matching as the training objective with latent-space training and multiresolution bucketed dataloading.

Installation and availability

NeMo Automodel is available via Docker container (nvcr.io/nvidia/nemo-automodel:26.06), pip install, or from source. The recommended installation includes PyTorch, TransformerEngine, and CUDA-compiled dependencies pre-built.

Multi-node orchestration currently works with SLURM, with Kubernetes support coming next. The team is also developing Pythonic recipe APIs as an alternative to YAML configuration.

What this means

This integration removes a significant friction point in diffusion model development. Previously, scaling training beyond single-node setups required custom code and checkpoint format conversions. By making distributed training a configuration choice rather than a code rewrite, NVIDIA and Hugging Face have made it practical for researchers to fine-tune models like HunyuanVideo (13B) and FLUX.2-dev (32B) that were previously out of reach for many teams. The direct Hub integration means fine-tuned models remain compatible with the existing Diffusers ecosystem, including quantization tools, LoRA adapters, and custom samplers.