NVIDIA NeMo Automodel integrates with Hugging Face Diffusers for distributed video and image model fine-tuning
NVIDIA and Hugging Face have integrated NeMo Automodel with the Diffusers library, enabling distributed fine-tuning of video and image diffusion models without checkpoint conversion. The integration supports models including FLUX.1-dev (12B), Wan 2.1 (1.3B/14B), and HunyuanVideo (13B) with full fine-tuning and LoRA options.
NVIDIA NeMo Automodel integrates with Hugging Face Diffusers for distributed video and image model fine-tuning
NVIDIA and Hugging Face have integrated NeMo Automodel with the Diffusers library, enabling production-grade distributed training for diffusion models without checkpoint conversion. The integration, released under Apache 2.0, allows users to fine-tune any Diffusers-format model directly from the Hugging Face Hub.
Technical capabilities
NeMo Automodel is a PyTorch DTensor-native training library that uses Diffusers model classes for loading and pipelines for generation. The integration eliminates the need to convert checkpoints between training and inference formats—fine-tuned weights load directly into DiffusionPipeline.
The library implements multiple parallelism strategies as configuration options rather than code rewrites: FSDP2, tensor parallel, expert parallel, context parallel, and pipeline parallel. It includes memory-efficient sharding, latent caching, and multiresolution bucketing for training at scale.
Supported models
The integration ships with fine-tuning recipes for six model families:
- Wan 2.1 T2V: 1.3B (fits single 40GB A100) and 14B parameter versions for text-to-video
- Wan 2.2 T2V A14B: 27B total parameters (MoE), 14B active per step
- FLUX.1-dev: 12B parameters for text-to-image
- FLUX.2-dev: 32B parameters for text-to-image
- HunyuanVideo 1.5: 13B parameters for text-to-video
- Qwen-Image: 20B parameters (MMDiT) for text-to-image
All models except Wan 2.2 include LoRA recipe support for parameter-efficient fine-tuning.
Training workflow
The workflow consists of three steps:
-
Pre-encode dataset: Cache VAE latents and text embeddings using distributed preprocessing across GPUs. For the 78-image Rider-Waite tarot dataset example, preprocessing outputs .pt cache files and sharded metadata.
-
Launch training: Use existing YAML configs with command-line overrides for dataset-specific settings. The example fine-tunes FLUX.1-dev with eight-way FSDP2 and an effective batch size of 32.
-
Generate from checkpoint: Fine-tuned models load directly into Diffusers pipelines without conversion.
The library currently supports flow-matching models only, using flow matching as the training objective with latent-space training and multiresolution bucketed dataloading.
Installation and availability
NeMo Automodel is available via Docker container (nvcr.io/nvidia/nemo-automodel:26.06), pip install, or from source. The recommended installation includes PyTorch, TransformerEngine, and CUDA-compiled dependencies pre-built.
Multi-node orchestration currently works with SLURM, with Kubernetes support coming next. The team is also developing Pythonic recipe APIs as an alternative to YAML configuration.
What this means
This integration removes a significant friction point in diffusion model development. Previously, scaling training beyond single-node setups required custom code and checkpoint format conversions. By making distributed training a configuration choice rather than a code rewrite, NVIDIA and Hugging Face have made it practical for researchers to fine-tune models like HunyuanVideo (13B) and FLUX.2-dev (32B) that were previously out of reach for many teams. The direct Hub integration means fine-tuned models remain compatible with the existing Diffusers ecosystem, including quantization tools, LoRA adapters, and custom samplers.
Related Articles
NVIDIA Nemotron 3 Embed 8B Tops RTEB Leaderboard with 78.5% Score, 1B Variant Cuts Error Rate 27%
NVIDIA's Nemotron-3-Embed-8B-BF16 ranks #1 on the RTEB leaderboard with a 78.5% score, while the 1B variant reduces error rate by 27% over its predecessor. The open-weight models feature 32k context windows and production-ready deployment options including a Blackwell-optimized NVFP4 variant.
Nvidia Launches Cosmos 3 Edge World Model for Physical AI, Forms Japan Industrial Coalition
Nvidia released Cosmos 3 Edge, a world model designed for robots and vision AI agents to perceive and navigate physical environments in real time. The company announced partnerships with Japanese industrial giants including Fujitsu, Hitachi, and Kawasaki Heavy Industries as part of its physical AI expansion.
AWS launches Managed Knowledge Base for Bedrock with 6 enterprise connectors and automatic ACL enforcement
Amazon Web Services launched Managed Knowledge Base for Bedrock in general availability, offering a fully managed retrieval solution with six native enterprise connectors including SharePoint, Confluence, and Google Drive. The service handles document parsing up to 500 MB for PDFs, 2 GB for audio, and 10 GB for video, with real-time access control list verification at query time.
LM Studio launches Bionic, agentic app for local and cloud open-source models
LM Studio released Bionic, a Mac app that runs open-source AI models locally or via cloud for coding, document processing, and research tasks. The app includes offline voice transcription using Mistral's Voxtral model and supports models like GLM 5.2 and Kimi K2.7 Code for codebase editing.
Comments
Loading...