CVPR 2026
Important
Looking for the data? Start on Hugging Face. All dataset downloads, download tools, and usage guides are available there: Rendered Data · 3D Assets · Leaderboard Evaluation Data. This GitHub repository hosts the code and baseline implementations.
We present PhysInOne, the largest dataset addressing the critical scarcity of physically-grounded training data for AI systems.
- 2 million videos generated from 153,810 dynamic 3D scenes
- Covers 71 fundamental physical phenomena in everyday environments, spanning four major domains: Mechanics, Optics, Fluid Dynamics, Magnetism
- Includes 2,231 common objects tailored to daily physical interactions
- Enriched with 623 materials across five categories: plastic, metal, wood, stone, and fabric
- Features 528 diverse 3D backgrounds to ensure realism and environmental variety
- Each scene involves 1–3 physical phenomena, reflecting real-world activities
- Supports complex multi-object interactions, with increasing scene complexity
- Average number of objects per scene: 3.9 (single-physics), 6.3 (double-physics), 7.8 (triple-physics)
- Each scene is captured from 13 viewpoints: 12 static cameras and 1 moving camera
- 3D geometry
- Semantic labels
- Object motion and dynamics
- Physical properties
- Natural-language scene descriptions
- Physics-aware video generation
- Short- and long-term future frame prediction
- Physical property estimation
- Motion transfer
- And more...
| Component | Progress | Status | Notes |
|---|---|---|---|
| Rendered Data - Train | ██████████ 100%(122988/122988) |
Released | Last updated: Aug 21 |
| Rendered Data - Val | ░░░░░░░░░░ 1%(103/15411) |
In progress | |
| 3D Assets | ██████████ 100% |
Released | 14,847 Val scenes, 120,660 Train scenes, and Train/Val resource libraries |
| Leaderboard Evaluation Inputs | ██████████ 100% |
Released | All public task inputs released; ground truth excluded |
| Baseline code | ███████▌░░ 75% |
In progress | Task-specific implementations; see Baselines below |
| Data processing | ██████████ 100% |
Released | Camera placement, UE rendering, EXR conversion, and point-cloud generation |
| Resource | Link |
|---|---|
| 📄 Paper | arXiv |
| 🌐 Project Page | vlar-group.github.io/PhysInOne |
| 🤗 Dataset | Hugging Face |
| 🎞️ Rendered Data | Scene selection and download guide |
| 🧊 3D Assets | Asset download and project setup guide |
| 🏆 Leaderboard Data | Evaluation input download guide |
| 🛠️ Data Processing | Camera, rendering, and post-processing usage guide |
Due to the large scale of PhysInOne, the rendered data and annotations are split across 16 Hugging Face repositories. Each entry shows the shard size, release status, live all-time downloads, live downloads in the last 30 days, and its repository link.
Combined snapshot (Oct 1, 2026): P01–P16 have 1,233,150 all-time downloads and 408,806 downloads in the last 30 days. Including the main repository, the per-repository sums are 1,280,414 and 442,702.
Download badges query the official Hugging Face API and update automatically. Counts are repository-level download events, not deduplicated users; accessing multiple shards can produce one event in each shard.
Released Train rendered archives are distributed across the 16 repositories listed above; released Val archives are hosted in the main repository. Dependency-free public tools can filter scenes by split, interaction type, and physical phenomenon, then download the selected archives with resume support and per-file logs.
- Read the rendered-data selection and download guide
- Browse the download tools
- Browse the Hugging Face utility mirror
The complete public Unreal Engine 5.5.4 asset release includes the project files and Scene/Trajectory resources for 14,847 validation scenes and 120,660 training scenes. See the dedicated guide for download commands, archive layout, extraction, and project setup.
All user-facing evaluation inputs for the four PhysInOne Leaderboard tasks are publicly available; ground-truth targets remain private. See the dedicated guide for task contents, copyable download commands, manifests, output layout, and resume instructions.
The public toolkit covers static and moving camera generation, Level Sequence creation, unattended Unreal Engine rendering, EXR conversion, and multi-view point-cloud generation. The guides describe the Unreal Engine 5.5.4 setup, required project assets, commands, outputs, and safety behavior.
- Browse the data-processing source
- Read the quick-start guide
- Read the complete camera and rendering guide
The PMF (Physical Motion Fidelity) evaluates video similarity in the frequency domain using 3D FFT-based energy distributions. It is designed for physics-aware video generation, future prediction, and motion transfer tasks in the PhysInOne benchmark.
# Step 1: Install PyTorch first (choose your variant)
# CPU only:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cpu
# CUDA 12.6:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
# Step 2: Install pmf from this repo
pip install git+https://github.com/vLAR-group/PhysInOne.git#subdirectory=pmf#!/usr/bin/env python
"""Test PMF metric with random tensors."""
import torch
from pmf import compute_pmf
def main():
torch.manual_seed(42)
B, T, C, H, W = 1, 81, 3, 128, 128
video_pred = torch.randn(B, T, C, H, W)
video_gt = torch.randn(B, T, C, H, W)
score = compute_pmf(video_pred, video_gt, device='cpu')
# If you want to use gpu, set device='cuda'
# score = compute_pmf(video_pred, video_gt, device='cuda')
if isinstance(score, torch.Tensor):
score = score.item()
print(f"PMF similarity score: {score:.4f}")
if __name__ == "__main__":
main()Baseline implementations are organized by task under baselines/:
Refer to each task or model's README for environment setup, checkpoint requirements, training (where supported), and inference or evaluation instructions. Available implementations and supported workflows vary by task. Public evaluation inputs and download instructions are available on Hugging Face; benchmark ground-truth targets remain private.
📅 Update Schedule: This section is actively being updated throughout Oct. and Nov.
If you find this work useful, please cite:
@article{zhou2026physinone,
title={PhysInOne: Visual Physics Learning and Reasoning in One Suite},
author={Siyuan Zhou and Hejun Wang and Hu Cheng and Jinxi Li and Dongsheng Wang and Junwei Jiang and Yixiao Jin and Jiayue Huang and Shiwei Mao and Shangjia Liu and Yafei Yang and Hongkang Song and Shenxing Wei and Zihui Zhang and Peng Huang and Shijie Liu and Zhengli Hao and Hao Li and Yitian Li and Wenqi Zhou and Zhihan Zhao and Zongqi He and Hongtao Wen and Shouwang Huang and Peng Yun and Bowen Cheng and Pok Kazaf Fu and Wai Kit Lai and Jiahao Chen and Kaiyuan Wang and Zhixuan Sun and Ziqi Li and Haochen Hu and Di Zhang and Chun Ho Yuen and Bing Wang and Zhihua Wang and Chuhang Zou and Bo Yang},
year={2026},
journal={CVPR}
}This project is licensed under the CC BY-NC-SA 4.0 license.
We would like to express our sincere gratitude to all contributors who participated in human evaluations and data collection efforts.