GENIMA
- joint prediction
- 2024
- Stable Diffusion
- World action model
GENIMA fine-tunes Stable Diffusion to draw the robot's future joint poses onto RGB images, then a controller turns those drawings into joint commands.
Key facts
Rows below are the register record in lib/models.ts. A dash means the value is not confirmed from a primary source. Classification follows Reuss, NVIDIA Technical Blog (15 June 2026) and the model's own paper.
| Name | GENIMA |
|---|---|
| Lab | — |
| Year | 2024 |
| Paradigm | joint prediction |
| Backbone | Stable Diffusion |
| Source | https://arxiv.org/abs/2407.07875 |
What the paper claims
- The method is a behavior-cloning agent that fine-tunes Stable Diffusion to draw joint-action targets as colored spheres on RGB images. A downstream controller maps those visual targets into a sequence of joint positions. Shridhar, Lo & James, 2024
- The diffusion stage uses SD-Turbo with ControlNet; the controller is an ACT policy trained on ground-truth targets overlaid on random backgrounds so that it follows the drawn joints rather than the scene. Shridhar, Lo & James, 2024
- Reuss groups this as the early action-as-image recipe, later scaled by Cosmos Policy, which encodes actions as synthetic latent video frames. His comparison table files GENIMA as inverse dynamics; this register files it as joint prediction because the action is generated as an image together with the observation. The register field is left as stored. Reuss, NVIDIA Technical Blog (15 June 2026)
What it evaluates
- On 25 RLBench tasks the authors report beating ACT on 16 of 25 and Diffusion Policy on 25 of 25. They also report 9 real-world manipulation tasks. Shridhar, Lo & James, 2024
- They report robustness tests on randomized object colors, distractors and lighting, and generalization to novel objects, and compare RGB-to-joint control with 3D next-best-pose agents that use depth, keypoints or motion planners. Shridhar, Lo & James, 2024
On this register
Joint prediction. Emits video and actions together. That axis is defined in what is a world action model? and in Reuss, NVIDIA Technical Blog (15 June 2026).
Other joint prediction models on the register: GR-1, Unified Video Action, Cosmos Policy, DreamZero.
No other register row currently lists the backbone Stable Diffusion. Matching is exact; nearby names such as Wan 2.1 versus Wan 2.2 are kept separate.
Limits the authors state
- The authors list these limits: the agent only distills expert demonstrations and does not discover new behaviors; training needs calibrated cameras to render targets; the robot must be visible from some viewpoint; the diffusion agent runs at 5 Hz against a 50 Hz controller; motions can be jerky after a new target image; the controller sometimes misses the drawn target. Shridhar, Lo & James, 2024, §5 and Appendix J