{"id":75228,"date":"2026-03-04T12:03:02","date_gmt":"2026-03-04T12:03:02","guid":{"rendered":"https:\/\/youzum.net\/physical-intelligence-team-unveils-mem-for-robots-a-multi-scale-memory-system-giving-gemma-3-4b-vlas-15-minute-context-for-complex-tasks\/"},"modified":"2026-03-04T12:03:02","modified_gmt":"2026-03-04T12:03:02","slug":"physical-intelligence-team-unveils-mem-for-robots-a-multi-scale-memory-system-giving-gemma-3-4b-vlas-15-minute-context-for-complex-tasks","status":"publish","type":"post","link":"https:\/\/youzum.net\/de\/physical-intelligence-team-unveils-mem-for-robots-a-multi-scale-memory-system-giving-gemma-3-4b-vlas-15-minute-context-for-complex-tasks\/","title":{"rendered":"Physical Intelligence Team Unveils MEM for Robots: A Multi-Scale Memory System Giving Gemma 3-4B VLAs 15-Minute Context for Complex Tasks"},"content":{"rendered":"

Current end-to-end robotic policies, specifically Vision-Language-Action (VLA) models, typically operate on a single observation or a very short history. This \u2018lack of memory\u2019 makes long-horizon tasks, such as cleaning a kitchen or following a complex recipe, computationally intractable or prone to failure. To address this, researchers from Physical Intelligence, Stanford, UC Berkeley, and MIT have introduced Multi-Scale Embodied Memory (MEM)<\/strong>.<\/p>\n

\n
\"\"
https:\/\/www.pi.website\/download\/Mem.pdf<\/figcaption><\/figure>\n<\/div>\n

The Dual-Scale Memory Architecture<\/strong><\/h3>\n

MEM factorizes robotic memory into two distinct scales to balance semantic context with real-time control constraints<\/sup><\/sup><\/sup><\/sup>.<\/p>\n

(1) Short-Term Video Memory<\/strong><\/h4>\n

For tasks requiring fine-grained spatial awareness\u2014like resolving self-occlusions or adapting a grasp\u2014dense visual data is required. MEM utilizes an efficient video encoder that extends standard Vision Transformers (ViTs). To maintain real-time inference (the 380ms \u2018real-time barrier\u2019), the architecture avoids joint attention over all patches. Instead, it uses Space-Time Separable Attention<\/strong>, interleaving spatial attention within frames with causal-temporal attention across frames every fourth layer.<\/p>\n

The computational complexity is reduced from O<\/em>(n2<\/sup>K2<\/sup><\/em>) to O<\/em>(Kn2<\/sup>+nK2<\/sup><\/em>), where n<\/em> is the number of spatial patches and K<\/em> is the number of timesteps. By dropping tokens from past timesteps in upper layers, the model passes only the current observation\u2019s representation to the VLA backbone, keeping the token count invariant compared to single-frame models.<\/p>\n

(2) Long-Term Language Memory<\/strong><\/h4>\n

To handle tasks spanning up to 15 minutes, MEM uses a language-based representation for semantic events<\/sup><\/sup><\/sup><\/sup><\/sup><\/sup><\/sup><\/sup><\/sup>. The system decomposes the action prediction as:<\/p>\n

$$pi(a_{t:t+H},l_{t+1},m_{t+1}|o_{t-T:t},m_{t},g) approxpi_{LL}(a_{t:t+H}|o_{t-K:t},l_{t+1},g)pi_{HL}(l_{t+1},m_{t+1}|o_{t},m_{t},g)$$\n<\/div>\n

Here, a high-level policy (\u03c0HL<\/sub><\/em>)<\/sub> maintains a running language summary (mt<\/sub>)<\/em> of past events and generates subtask instructions (lt+1<\/sub>) for a low-level policy (\u03c0<\/em>LL<\/sub>). This language memory is trained using LLM-generated summaries that compress information (e.g., \u2018I placed three bowls\u2019 instead of individual attributes), reducing the risk of training-inference distribution shifts.<\/p>\n

\n
\"\"
https:\/\/www.pi.website\/download\/Mem.pdf<\/figcaption><\/figure>\n<\/div>\n

Implementation and Performance<\/strong><\/h3>\n

The research team integrated MEM into the \u03c0<\/em>0.6<\/sub> VLA<\/strong>, which is initialized from a pre-trained Gemma 3-4B<\/strong> model. The model was pre-trained on a diverse mixture of robot demonstrations, vision-language tasks, and internet video data.<\/p>\n

Key Results:<\/strong><\/h4>\n