What is the impact of incorporating language supervision (e.g., action descriptions) during pretraining on ViP

Abstract

Abstract: Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of s

Research Question

What is the impact of incorporating language supervision (e.g., action descriptions) during pretraining on ViPRA's performance in few-shot robotic manipulation tasks compared to purely visual pretraining, as measured by metrics like human preference or task completion accuracy on benchmarks like CALVIN or ALFRED?

Verification Level

Descriptive public verification status only; aggregate claim counts are public, but individual claim records are not exposed here.

Quality Tier

Descriptive public triage only; this tier does not alter current publication or DOI behavior.

Quality Dimensions

Automated triage signals derived from public fields; not human peer review or independent validation.

Correction Record

Public corrections are additive records. Current status does not claim the synthesis is error-free.

Provenance