ByteDance Open-Sources Bernini-R Video Diffusion Model With Semantic Planning Architecture
ByteDance released Bernini-R, an open-source video generation and editing model that combines an MLLM-based semantic planner with a DiT-based renderer. The model requires Hopper-class GPUs (H100/H800/H200) for optimal performance and supports multiple tasks including text-to-video, video editing, and reference-guided generation.
ByteDance Open-Sources Bernini-R Video Diffusion Model With Semantic Planning Architecture
ByteDance released Bernini-R, an open-source video generation and editing model that combines an MLLM-based semantic planner with a DiT-based renderer. The company made the inference code and model weights available on June 1, 2025, following a research paper published May 22.
Technical Architecture
Bernini-R is built on the Wan2.2 base model (Wan-AI/Wan2.2-T2V-A14B) and uses a dual-checkpoint architecture with separate high-noise and low-noise transformer weights. The model requires:
- Python 3.11.2
- CUDA 12.4 (minimum 12.3)
- PyTorch 2.5.1+cu124
- Hopper GPU (H100/H800/H200) recommended for FlashAttention-3 support
- Other CUDA GPUs fall back to FlashAttention-2 or PyTorch SDPA
The system uses pinned dependencies: diffusers 0.35.2, accelerate 0.34.2, and transformers 4.57.3.
Capabilities and Performance
The model supports seven task types:
- Text-to-image (t2i)
- Image editing (i2i)
- Text-to-video (t2v)
- Video editing (v2v)
- Motion-aware video editing (mv2v)
- Reference + video editing (rv2v)
- Reference-to-video (r2v)
According to ByteDance, Bernini reaches "the first tier among leading closed-source commercial models" on video editing tasks. This claim is based on a self-built arena platform where human annotators blindly vote on paired edits, aggregated into Bradley-Terry scores and win-rate matrices. No specific numerical benchmarks were disclosed.
Default inference outputs 480p video at 16fps (848px max image size). The system supports higher resolutions, with examples showing 720p/24fps output at 1280px max size.
Deployment Requirements
Single-GPU inference works for image tasks (t2i, i2i with --num_frames 1). Video tasks use 8-GPU configurations via torchrun with Ulysses sequence parallelism. The --ulysses flag controls N-way sequence parallel per sample, with remaining GPUs running data parallel over task lists.
Multi-GPU setups require Open-VeOmni (Apache-2.0, Python 3.11) for sequence parallelism, though single-GPU inference does not need this dependency.
Model Access
Two distribution methods are available:
- Diffusers format (recommended): ByteDance/Bernini-R-Diffusers on Hugging Face — self-contained directory bundling VAE, UMT5 text encoder, tokenizer, and Bernini-R weights
- Separate checkpoints: Base Wan2.2 model plus Bernini-R high-noise/low-noise weights from ByteDance/Bernini-R
ByteDance recommends using a prompt enhancer (--use_pe flag) through an OpenAI-compatible endpoint for best generation quality. The enhancer requires configuring BERNINI_PE_API_KEY, BERNINI_PE_BASE_URL, and BERNINI_PE_MODEL environment variables.
What This Means
Bernini-R represents ByteDance's entry into open-source video generation, competing with models from Stability AI and others in the video diffusion space. The Hopper GPU requirement (H100/H800/H200) creates a high barrier to entry — these GPUs cost $25,000-40,000 each and are primarily available through cloud providers. The dual-checkpoint architecture and multi-GPU requirements suggest this is designed for research labs and companies with substantial compute budgets rather than individual developers. ByteDance's self-reported performance claims need independent verification, as the company provided no standardized benchmark scores against public datasets.
Related Articles
Elorian launches Glimpse, a visual reasoning model priced at $0.60/$3 per 1M tokens
Elorian, the Palo Alto startup that raised a $55M seed at a $300M valuation, has released Glimpse as a research preview with API access at $0.60 per 1M input tokens and $3 per 1M output tokens. The model can crop, pull video frames, and run code mid-reasoning. Benchmark claims come from Elorian's own evaluations.
Moondream 3 Preview lists on OpenRouter: 9B MoE vision model, 2B active, 32K context, $0.30/$2.50
Moondream 3 Preview is a vision-language model with 9B total and 2B active parameters (mixture-of-experts), a 32K-token context window and an optional reasoning mode. OpenRouter lists it at $0.30 per 1M input tokens and $2.50 per 1M output tokens, served by Prime Intellect.
Moondream 3.1: 9B MoE vision model with 2B active parameters, 32K context, $0.30/$2.50 per 1M tokens
Moondream has released Moondream 3.1, a mixture-of-experts vision-language model with 9B total and 2B active parameters and a 32K-token context window. It is listed on OpenRouter at $0.30 per 1M input tokens and $2.50 per 1M output tokens, with an optional reasoning mode.
Nvidia releases Nemotron 3.5 Super VL: open 120B multimodal MoE with 12B active parameters
Nvidia has released Nemotron 3.5 Super VL, an open multimodal Mixture-of-Experts model with 120 billion total and 12 billion active parameters, built for continuously running enterprise agents. Two compact Bolt models for local PCs and edge devices launch alongside it, though the 4 GB variant ships later this month.
Comments
Loading...