Whole-Body Conditioned Egocentric Video Prediction

× Predicting Ego-centric Video from human Actions (PEVA). Given past video frames and an action specifying a desired change in 3D pose, PEVA predicts the next video frame. Our results show that, given the first frame and a sequence of actions, our model can generate videos of atomic actions (a), simulate counterfactuals (b), and long video generation (c). Recent years have brought significant advances in world models that learn to simulate future outcomes for planning and control.