Video Prediction Policy
- inverse dynamics
- 2024
- Stable Video Diffusion
- World action model
Video Prediction Policy reads actions from the internal future representations of a video diffusion model, without waiting to render pixels at control time.
Key facts
Rows below are the register record in lib/models.ts. A dash means the value is not confirmed from a primary source. Classification follows Reuss, NVIDIA Technical Blog (15 June 2026) and the model's own paper.
| Name | Video Prediction Policy |
|---|---|
| Lab | — |
| Year | 2024 |
| Paradigm | inverse dynamics |
| Backbone | Stable Video Diffusion |
| Source | https://arxiv.org/abs/2412.14803 |
What the paper claims
- The authors hypothesize that video diffusion models already contain current appearance and predicted dynamics in their latent tensors, and that a policy can learn implicit inverse dynamics from those representations. Hu et al., 2024
- They fine-tune open Stable Video Diffusion (1.5 billion parameters) into a text-guided video prediction model on internet human and robot manipulation data, then train a diffusion-policy head on the video model's internal features from the first forward pass. Hu et al., 2024
- Because they use internal representations rather than fully denoised RGB, they argue the policy can run closed-loop at high frequency compared with methods that generate finished future images first. Hu et al., 2024
What it evaluates
- The abstract reports an 18.6% relative improvement on the CALVIN ABC-D generalization benchmark against the previous state of the art in their comparison, and a 31.6% increase in success rates on complex real-world dexterous manipulation tasks. Hu et al., 2024
- They evaluate in two simulated settings (including CALVIN and Meta-World) and two real-world settings. Hu et al., 2024
On this register
Inverse dynamics. Generates the future video, then infers the actions that would produce it. That axis is defined in what is a world action model? and in Reuss, NVIDIA Technical Blog (15 June 2026).
Other inverse dynamics models on the register: UniPi, mimic-video, LingBot-VA, DVA.
No other register row currently lists the backbone Stable Video Diffusion. Matching is exact; nearby names such as Wan 2.1 versus Wan 2.2 are kept separate.
Coverage
No stories on this model yet.