Egocentric action transfer aims to decouple the semantic action from egocentric observations and reproduce them in new visual contexts, enabling scalable world simulation, embodied policy learning, and robotic data generation. Existing approaches to transferring actions rely on condition-guided video generation, which converts the observation video into explicit geometric controls (e.g., hand poses or meshes) to drive synthesis. However, these methods introduce geometric estimation errors, making it difficult to preserve physically plausible interactions when transferred to new scenes. Alternatively, motion transfer methods directly extract motion patterns from the observation video, yet motion-level features alone cannot encode rich contact dynamics, frequently leading to severe hand structural collapse and implausible contact layout. To address both limitations, we present EgoACT, a test-time framework for egocentric action transfer that operates directly on the denoising process of video diffusion models without requiring additional geometric estimators. EgoACT uses Velocity-guided Structure Anchoring to stabilize reference-consistent hand-object structure in the early denoising stage, and Sparse Correspondence Calibration to refine reliable local correspondences in the mid-to-late denoising stages. Together, these two components preserve transferable action semantics while improving temporal coherence and interaction realism. We further establish EgoActionBench, a benchmark for evaluating action preservation, visual quality, and hand-object plausibility across diverse egocentric manipulation scenarios. Experiments show that EgoACT generates more coherent and physically plausible interaction videos than strong baselines.
Motion transfer suffers from hand structural collapse and lacks physical interaction. Condition-guided generation preserves hand shape but produces physically implausible contact. Our method maintains both hand structure and realistic interaction.
Figure 3: Overview of the proposed framework. Given an egocentric manipulation video, our framework uses a reference branch and an optimized generation branch for test-time action transfer. We apply Velocity-guided Structure Anchoring (VSA) in the early denoising steps to preserve hand-object structure, and Sparse Correspondence Calibration (SCC) in the mid-to-late steps to refine local correspondences and improve interaction realism.
Given reference videos with distinct motion dynamics, our model generates videos that faithfully transfer the motion patterns while aligning with different prompts. Hover over the reference thumbnail to preview.
If you find our work useful, please consider citing:
@article{ma2026egoact,
title={Decoupling Action from Egocentric Observation for World Simulation},
author={Ma, Yue and Song, Pengjie and Wang, Xinyu and He, Yi and Long, Zeqian and Zhan, Fangneng and Zhou, Kaichen and Liu, Hongyu and Wang, Hongfa and Li, Peihao and Huang, Haoyang and Duan, Nan and Chen, Qifeng},
journal={NeurIPS},
year={2026}
}