long-horizon robot autonomy
Physical Intelligence ships multi-scale memory and task conditioning to extend robot autonomy past 10 minutes Stanford OnlineTL;DW
- Memory architecture uses sparse temporal attention in video encoder plus compressed language summaries to span short-horizon (15 frames) and long-horizon (minutes) tasks without excessive latency or distribution shift.
- Multi-scale embodied memory (MEM) combines dense visual observations for recent past with semantic language-based summaries for longer history, enabling robots to track completed steps and avoid failure loops.
- PIO 7 uses rich conditioning—metadata, subtasks, predicted subgoals—to train a single high-performance, generalizable policy instead of separate pretraining and post-training stages.
- Conditioning on quality and speed metadata at inference time lets a single policy checkpoint match performance of previously fine-tuned task-specific models without retraining.
- Cross-robot transfer demonstrated: UR5 arm learned to fold shirts using subgoal conditioning, despite no laundry training data, by generalizing skills learned on other tasks.
- Robots without memory fail repeatedly on identical errors (chopstick grasping, symmetric fridge); memory enables in-context adaptation—e.g., adjusting grasp height after observing failure.
- Long-horizon task failures occur subtly in partial observability scenarios (unpacking grocery bag, washing plate, grilling); memory prevents endless loops when objects are unobserved.
- Rich metadata conditioning enables "coaching" mode: humans guide robots through unseen tasks in language without teleoperation, policies then distill into end-to-end learned behaviors.
- Longer task horizons increase distribution shift burden; robot must generalize more because exact 20-30 minute episodes become increasingly rare in training data.
- Poor-quality training data hurts performance when forcing models to learn full distribution, but with quality-conditioned metadata, bad data improves generalization by teaching the model robustness.
TL;DW
- Memory architecture uses sparse temporal attention in video encoder plus compressed language summaries to span short-horizon (15 frames) and long-horizon (minutes) tasks without excessive latency or distribution shift.
- Multi-scale embodied memory (MEM) combines dense visual observations for recent past with semantic language-based summaries for longer history, enabling robots to track completed steps and avoid failure loops.
- PIO 7 uses rich conditioning—metadata, subtasks, predicted subgoals—to train a single high-performance, generalizable policy instead of separate pretraining and post-training stages.
- Conditioning on quality and speed metadata at inference time lets a single policy checkpoint match performance of previously fine-tuned task-specific models without retraining.
- Cross-robot transfer demonstrated: UR5 arm learned to fold shirts using subgoal conditioning, despite no laundry training data, by generalizing skills learned on other tasks.
- Robots without memory fail repeatedly on identical errors (chopstick grasping, symmetric fridge); memory enables in-context adaptation—e.g., adjusting grasp height after observing failure.
- Long-horizon task failures occur subtly in partial observability scenarios (unpacking grocery bag, washing plate, grilling); memory prevents endless loops when objects are unobserved.
- Rich metadata conditioning enables "coaching" mode: humans guide robots through unseen tasks in language without teleoperation, policies then distill into end-to-end learned behaviors.
- Longer task horizons increase distribution shift burden; robot must generalize more because exact 20-30 minute episodes become increasingly rare in training data.
- Poor-quality training data hurts performance when forcing models to learn full distribution, but with quality-conditioned metadata, bad data improves generalization by teaching the model robustness.
Mem compresses visual tokens for short-horizon tracking and language summaries for long-horizon semantics, keeping inference under 300ms. PIO 7 trains a single policy with task metadata and subgoal conditioning, matching fine-tuned specialists on kitchen, laundry, and recipe tasks without post-training.