What does MolmoMotion predict?

MolmoMotion predicts where marked 3D points on an object will move in the next few seconds, given a video frame, the point locations, and a text description of the intended action (e.g., 'Move and rotate the wooden bowl').

Why is motion forecasting useful for robotics?

A robot manipulating an object must anticipate how that object will move before making contact. MolmoMotion enables this by predicting future 3D trajectories, which can then drive robotic control policies.

Is the model open-source?

Yes. Allen AI is releasing the model weights, the MolmoMotion-1M dataset, and the PointMotionBench benchmark openly on Hugging Face and GitHub.

Allen AI releases MolmoMotion, a language-guided 3D motion forecasting model

Q: How large is the training dataset?

MolmoMotion-1M contains 1.16 million videos paired with 3D point trajectories and action descriptions, making it the largest collection of its kind.

Motion forecasting moves from observation to prediction

Allen AI introduced MolmoMotion, a vision-language model that forecasts how objects will move in 3D space over the next few seconds. According to the Hugging Face Blog, the model takes three inputs—a video frame showing an object, specific 3D points marked on that object, and a text instruction describing an action (such as “Rotate the wooden bowl”)—and outputs predicted 3D trajectories for those marked points. This shift from observing motion that has already occurred to predicting motion before it happens unlocks downstream applications in robotic manipulation and video generation.

A class-agnostic representation scaled across 1.16 million videos

Allen AI designed MolmoMotion to represent motion as object-attached 3D points in world coordinates rather than rendering full video frames. According to the Hugging Face Blog, this representation is class-agnostic—it generalizes across human bodies, rigid objects, hands, and other categories without requiring category-specific templates. The model was trained on MolmoMotion-1M, a dataset spanning 1.16 million videos, each paired with 3D point trajectories and action descriptions. The team also released PointMotionBench, a human-validated benchmark containing 2.7K video clips designed to measure object-centric 3D motion forecasting accuracy.

Downstream applications in robotics and video synthesis

Allen AI validated MolmoMotion on tasks requiring future-state reasoning. According to the Hugging Face Blog, the model achieved substantially stronger performance than existing forecasting methods on both robotics planning—where predicted trajectories drive robot control—and trajectory-conditioned video generation, where predicted motion guides the synthesis of physically plausible next frames. The open release of model weights, datasets, and benchmark code on Hugging Face and GitHub enables further research and customization by the community.

Why This Matters

Motion forecasting bridges perception and action in embodied AI systems. Robots that can predict object trajectories before contact improve grasp success rates and reduce failures from unexpected dynamics. For video generation, accurate trajectory prediction ensures that synthesized frames maintain physical plausibility—a critical constraint for applications requiring realistic visual outputs. The release of MolmoMotion-1M (1.16M videos) and PointMotionBench (2.7K validated clips) establishes shared evaluation infrastructure, lowering barriers for teams to benchmark their own motion models and accelerates the move from single-frame prediction to multi-step, language-conditioned trajectory reasoning. If the benchmark results hold under independent reproduction, this architecture could become a standard component in vision-language systems handling embodied prediction tasks.