Exclusive by design · every entry verified against the paper

The papers

Not a reading list — a map. A few dozen papers that moved the field, each placed on its pathway with a plain-language account of what it did and why it mattered. If a paper is here, it earned it.

World-Model Foundations (13 papers)

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (NeurIPS 2015) — Learns a latent space from raw pixels in which dynamics are locally linear, so classical optimal control can be run directly on images.

World Models (NeurIPS 2018) — Trains a VAE plus recurrent dynamics model of the environment and evolves a tiny controller entirely inside the model's 'dream', then transfers the policy back to the real environment.

Learning Latent Dynamics for Planning from Pixels (ICML 2019) — PlaNet learns environment dynamics from images and selects actions by fast online planning in latent space, solving continuous control from pixels only.

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (arXiv) — Robots plan manipulation by imagining outcomes with an action-conditioned video prediction model trained from unsupervised interaction, using raw pixels and no reward engineering.

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (Nature 2020) — MuZero reaches superhuman play in Go, chess, shogi and Atari by combining tree search with a learned model that predicts only reward, value and policy — never the rules or observations.

Dream to Control: Learning Behaviors by Latent Imagination (ICLR 2020) — Dreamer learns behaviors by backpropagating analytic value gradients through trajectories imagined inside a learned latent world model, instead of planning at decision time.

Mastering Atari with Discrete World Models (ICLR 2021) — DreamerV2 is the first agent to reach human-level Atari performance by learning entirely inside a world model, using categorical (discrete) latent states.

Temporal Difference Learning for Model Predictive Control (ICML 2022) — TD-MPC plans short horizons with a reconstruction-free latent dynamics model while a TD-learned value function estimates returns beyond the horizon.

DayDreamer: World Models for Physical Robot Learning (CoRL 2022) — Applies Dreamer directly on physical robots — a quadruped learns to walk from scratch in about one hour of real-world experience, with no simulator.

Transformers are Sample-Efficient World Models (ICLR 2023 (notable top 5%)) — IRIS tokenizes frames with a discrete autoencoder and models environment dynamics with an autoregressive Transformer, exceeding human-normalized scores on Atari 100k.

Mastering Diverse Domains through World Models (Nature 2025) — DreamerV3 masters 150+ tasks across wildly different domains with one fixed configuration, and is the first agent to mine diamonds in Minecraft from scratch without human data.

TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024) — A single set of TD-MPC2 hyperparameters works across 104 continuous-control tasks, and one 317M-parameter agent handles 80 tasks spanning multiple embodiments.

Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA) (TMLR 2024) — V-JEPA learns visual representations purely by predicting masked video features in latent space — no pixels reconstructed, no text, no negatives.

Video World Models (13 papers)

Learning Universal Policies via Text-Guided Video Generation (NeurIPS 2023) — Casts robot decision-making as text-conditioned video generation: a diffusion model 'imagines' a video of the task being done, then actions are inferred from the frames.

GAIA-1: A Generative World Model for Autonomous Driving (arXiv) — A ~9B-parameter generative world model that predicts future driving video conditioned on past frames, text, and ego actions, treating world modeling as next-token prediction plus video diffusion decoding.

Learning Interactive Real-World Simulators (UniSim) (ICLR 2024 (Outstanding Paper)) — Learns a universal action-conditioned video simulator of real-world interaction by orchestrating diverse datasets, then trains policies and VLMs entirely inside the learned simulator.

Genie: Generative Interactive Environments (ICML 2024 (Best Paper)) — An 11B foundation world model trained on unlabeled internet gameplay video that turns a single image into a playable 2D world, learning a latent action space without any action labels.

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024) — An open driving world model that predicts high-resolution futures at high frame rate, controllable by commands down to low-level trajectories, and usable as a reward/evaluation function.

Diffusion Models Are Real-Time Game Engines (GameNGen) (ICLR 2025) — Runs DOOM entirely inside a Stable-Diffusion-based neural model at over 20 fps, with an RL agent's gameplay providing the action-labeled training data.

Oasis: A Universe in a Transformer (Technical report) — A diffusion-transformer world model that serves a fully playable, engine-free Minecraft in real time at 20 fps from keyboard/mouse input, with open 500M weights.

Genie 2: A Large-Scale Foundation World Model (DeepMind blog) — Scales the Genie recipe to 3D: from a single prompt image it generates action-controllable, playable 3D environments with emergent physics, object permanence, and NPC interactions.

Cosmos World Foundation Model Platform for Physical AI (arXiv) — An open platform of pre-trained diffusion and autoregressive video world models, tokenizers, and data pipelines that developers post-train into domain-specific world models for robotics and driving.

GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving (arXiv) — A latent diffusion world model that generates spatiotemporally consistent multi-camera driving video conditioned on ego actions, agent layouts, weather, and road structure.

Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition (arXiv) — Generates high-dynamic interactive game videos by unifying keyboard and mouse inputs into a shared camera-representation space, with history-conditioned autoregressive extension.

Matrix-Game: Interactive World Foundation Model (arXiv) — A 17B open-source image-to-world model with fine-grained keyboard/mouse control, trained on 2,700+ hours of Minecraft data, outperforming Oasis and MineWorld on controllability and physics.

Genie 3: A New Frontier for World Models (DeepMind blog) — A general-purpose world model that generates navigable, promptable 720p worlds in real time at 24 fps, staying consistent for minutes and supporting on-the-fly 'promptable world events'.

Vision–Language–Action (16 papers)

RT-1: Robotics Transformer for Real-World Control at Scale (RSS 2023) — A transformer policy trained on 130k real robot demonstrations across 700+ tasks, showing that robot control improves with model and data scale.

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (CoRL 2023) — Co-fine-tunes a large vision-language model on web data and robot trajectories, expressing actions as text tokens so internet knowledge transfers to control.

Octo: An Open-Source Generalist Robot Policy (RSS 2024) — An open transformer policy with a diffusion action head, pretrained on 800k Open X-Embodiment trajectories and cheaply finetunable to new robots and sensors.

OpenVLA: An Open-Source Vision-Language-Action Model (CoRL 2024) — A 7B open-source VLA built on Prismatic/Llama 2 and trained on 970k Open X-Embodiment demos, outperforming the closed 55B RT-2-X.

π0: A Vision-Language-Action Flow Model for General Robot Control (RSS 2025) — Attaches a flow-matching action expert to a pretrained VLM (PaliGemma) to output continuous 50Hz action chunks, enabling dexterous tasks like laundry folding.

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025) — A 1.2B diffusion transformer foundation model for two-armed manipulation, pretrained on 46 multi-robot datasets with a unified physically interpretable action space.

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT) (RSS 2025) — A systematic study showing parallel decoding, action chunking, continuous actions, and an L1 objective dramatically boost VLA fine-tuning speed and success rates.

Gemini Robotics: Bringing AI into the Physical World (arXiv) — A VLA family built on Gemini 2.0, pairing an embodied-reasoning model (Gemini Robotics-ER) with a dexterous action model that adapts to new tasks, objects, and embodiments.

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (arXiv) — A dual-system VLA for humanoids pairing a VLM reasoner with a real-time diffusion-transformer action module, trained on a pyramid of real, synthetic, and web data.

π0.5: a Vision-Language-Action Model with Open-World Generalization (arXiv) — Extends π0 with heterogeneous co-training and high-level subtask inference so mobile manipulators can clean kitchens and bedrooms in entirely unseen homes.

SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (arXiv) — A 450M-parameter VLA trained on community LeRobot datasets that matches models 10x its size and runs on consumer GPUs or even CPUs, with async inference.

Igniting VLMs toward the Embodied Space (WALL-OSS) (arXiv) — An open embodied foundation model that tightly couples a VLM with action synthesis via a Unified Cross-Level chain-of-thought spanning reasoning, subgoal decomposition, and control.

π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities (arXiv) — A 5B steerable generalist VLA conditioned on multimodal context — language, performance metadata, visual subgoals — that matches specialist models out of the box and transfers zero-shot across embodiments.

RoboTTT: Context Scaling for Robot Policies (arXiv) — Integrates test-time training into robot foundation policies, compressing long visuomotor histories into gradient-updated fast weights so a policy can imitate from a single human video and improve on the fly.

RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model (arXiv) — Open family of embodied foundation models for perception, spatial reasoning, localization, and planning, with a VLA variant that maps a shared cross-embodiment action space onto several real robots.

Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories (arXiv) — Pretrains a vision-language-action model on real-world manipulation trajectories collected with handheld UMI devices and auto-labeled with scene-transition language, then post-trains it to follow instructions on robots.

Unified World–Action Models (17 papers)

Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (GR-1) (ICLR 2024) — A GPT-style transformer pretrained on large-scale video generation, then finetuned to jointly predict future frames and robot actions for language-conditioned manipulation.

RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024) — Factorizes text-conditioned video generation into composable language primitives so a robot world model can imagine, then execute, plans for unseen object-action combinations.

GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation (arXiv) — Scales the GR recipe to 38M web video clips, jointly generating future video and action trajectories to hit 97.7% success over 100+ manipulation tasks.

Prediction with Action: Visual Policy Learning via Joint Denoising Process (PAD) (NeurIPS 2024) — Unifies future-image prediction and robot action generation inside a single Diffusion Transformer denoising process, denoising pixels and actions together.

Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (Seer) (ICLR 2025) — An end-to-end transformer that first forecasts the goal visual state and then infers the actions that reach it, tying prediction and control in one network.

EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation (NeurIPS 2025) — A chunk-wise autoregressive video-diffusion foundation model that generates multi-view future embodied spaces from instructions and decodes them into manipulation actions.

Unified Video Action Model (UVA) (RSS 2025) — Jointly optimizes video and action diffusion over a shared latent, with decoupled decoding heads so the policy can act fast without generating video at inference.

CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025) — A 7B VLA that reasons in pixels: it autoregressively predicts subgoal image frames as a visual chain of thought before emitting the action sequence.

Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets (RSS 2025) — Couples video and action diffusion with independent per-modality noise timesteps in one transformer, so a single model can act as policy, forward dynamics, inverse dynamics, or video predictor.

DreamGen: Unlocking Generalization in Robot Learning through Video World Models (arXiv) — Finetunes video world models to dream photorealistic robot trajectories, labels them with latent or inverse-dynamics actions, and trains policies on the resulting 'neural trajectories'.

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (arXiv) — A self-supervised video model trained on 1M+ hours of internet video whose action-conditioned variant V-JEPA 2-AC, post-trained on only 62 hours of robot video, plans pick-and-place zero-shot on new Franka arms.

WorldVLA: Towards Autoregressive Action World Model (arXiv) — Fuses a VLA and a world model into one autoregressive transformer where action generation and future-image prediction mutually improve each other, with an attention mask fixing action-chunk error accumulation.

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025) — Has the policy 'dream' compact world knowledge — dynamic regions, depth, and semantics — instead of full frames, then decodes actions from the dream with a diffusion head.

Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation (arXiv) — A single video-generative world foundation platform (GE-Base/GE-Act/GE-Sim) that unifies policy learning, action execution, and neural simulation for real robots, trained on 1M+ AgiBot trajectories.

Ctrl-World: A Controllable Generative World Model for Robot Manipulation (ICLR 2026) — A multi-view, frame-level action-conditioned world model with pose-conditioned memory that lets generalist VLA policies roll out, be evaluated, and be improved entirely in imagination.

World Action Models are Zero-shot Policies (DreamZero) (arXiv) — Builds a World Action Model on a pretrained video-diffusion backbone that jointly predicts future states and actions, acting as a zero-shot policy on new tasks without task-specific training.

Cosmos 3: Omnimodal World Models for Physical AI (arXiv) — An open Mixture-of-Transformers omnimodel over language, image, video, audio, and action that switches between multimodal reasoning, video generation, and joint action prediction with world simulation.

Data & Evaluation (12 papers)

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (NeurIPS 2023 Datasets & Benchmarks) — A procedurally generated simulation benchmark of 130 language-conditioned manipulation tasks organized into suites that isolate transfer of spatial, object, and goal knowledge.

Open X-Embodiment: Robotic Learning Datasets and RT-X Models (ICRA 2024 (Best Paper Award)) — Pools demonstrations from 22 robot embodiments across 21 institutions into one standardized dataset and shows cross-robot training (RT-X) yields positive transfer.

VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 (Highlight)) — Decomposes video generation quality into 16 disentangled, human-aligned dimensions (subject consistency, motion smoothness, physics-adjacent attributes) with tailored prompts per dimension.

Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER) (CoRL 2024) — Builds visually-matched simulated twins of common real robot setups (Google Robot, WidowX/Bridge) and shows sim success rates strongly correlate with real-world policy performance.

1X World Model Challenge (Open challenge (GitHub)) — An open challenge with 100+ hours of tokenized egocentric video and actions from EVE humanoids, with compression, sampling, and (ultimately) policy-evaluation tracks.

WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025) — Dual evaluation of video models as world simulators: a human-preference evaluator for perceptual quality plus closed-loop tests of whether generated videos translate into correct control signals.

Do Generative Video Models Understand Physical Principles? (Physics-IQ) (arXiv) — A real-video benchmark probing whether video generators grasp fluid dynamics, optics, solid mechanics, magnetism, and thermodynamics, showing visual realism is uncorrelated with physical understanding.

WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 Datasets & Benchmarks) — Evaluates video generators as world models on instruction-following and physics adherence across 7 application domains (robotics, driving, industry, ...), backed by 67K human labels and a fine-tuned 2B automated judge.

AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems (IROS 2025 (Best Paper Award Finalist)) — Over 1 million human-verified teleoperated trajectories (about 3,000 hours) across 217 tasks and 106 scenes from a fleet of 100+ humanoid robots, plus the GO-1 generalist policy.

WorldScore: A Unified Evaluation Benchmark for World Generation (ICCV 2025) — Unifies evaluation of 3D, 4D, and video world generators by casting world generation as camera-controlled next-scene generation, scoring controllability, quality, and dynamics over 3,000 examples.

EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models (arXiv) — A benchmark and toolkit scoring embodied (robot) world models on visual scene consistency, motion/trajectory correctness, and semantic alignment with language instructions.

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies (CoRL 2025 (Oral)) — A Chatbot-Arena-style crowd-sourced network of real-robot evaluators running double-blind A/B comparisons of generalist policies on freely chosen tasks, aggregated into policy rankings.