Uncategorized

Current end-to-end robotic policies, specifically Vision-Language-Action (VLA) models, typically operate on a single observation or a very short history. This ‘lack of memory’ makes long-horizon tasks, such as cleaning a kitchen or following a complex recipe, computationally intractable or prone to failure. To address this, researchers from Physical Intelligence, Stanford, UC Berkeley, and MIT have introduced Multi-Scale Embodied Memory (MEM). https://www.pi.website/download/Mem.pdf The Dual-Scale Memory Architecture MEM factorizes robotic memory into two distinct scales to balance semantic context with real-time control constraints. (1) Short-Term Video Memory For tasks requiring fine-grained spatial awareness—like resolving self-occlusions or adapting a grasp—dense visual data is required. MEM utilizes an efficient video encoder that extends standard Vision Transformers (ViTs). To maintain real-time inference (the 380ms ‘real-time barrier’), the architecture avoids joint attention over all patches. Instead, it uses Space-Time Separable Attention, interleaving spatial attention within frames with causal-temporal attention across frames every fourth layer. The computational complexity is reduced from O(n2K2) to O(Kn2+nK2), where n is the number of spatial patches and K is the number of timesteps. By dropping tokens from past timesteps in upper layers, the model passes only the current observation’s representation to the VLA backbone, keeping the token count invariant compared to single-frame models. (2) Long-Term Language Memory To handle tasks spanning up to 15 minutes, MEM uses a language-based representation for semantic events. The system decomposes the action prediction as: $$pi(a_{t:t+H},l_{t+1},m_{t+1}|o_{t-T:t},m_{t},g) approxpi_{LL}(a_{t:t+H}|o_{t-K:t},l_{t+1},g)pi_{HL}(l_{t+1},m_{t+1}|o_{t},m_{t},g)$$ Here, a high-level policy (πHL) maintains a running language summary (mt) of past events and generates subtask instructions (lt+1) for a low-level policy (πLL). This language memory is trained using LLM-generated summaries that compress information (e.g., ‘I placed three bowls’ instead of individual attributes), reducing the risk of training-inference distribution shifts. https://www.pi.website/download/Mem.pdf Implementation and Performance The research team integrated MEM into the π0.6 VLA, which is initialized from a pre-trained Gemma 3-4B model. The model was pre-trained on a diverse mixture of robot demonstrations, vision-language tasks, and internet video data. Key Results: In-Context Adaptation: MEM enables robots to adapt manipulation strategies based on recent failures. In evaluation, this led to a +62% success rate increase in opening refrigerators with unknown hinge directions and a +11% increase in picking up chopsticks at variable heights. Long-Horizon Tasks: The model successfully performed 15-minute tasks like ‘Recipe Setup’ (retrieving ingredients from multiple locations) and ‘Kitchen Cleaning’ (washing dishes and wiping counters). Memory-less VLAs failed these tasks significantly more often. Efficiency: The video encoder allows the model to process up to 16 observation frames (spanning ~1 minute) while remaining under critical real-time inference thresholds on a single NVIDIA H100 GPU. MEM demonstrates that combining dense, short-term visual tokens with compressed, long-term language summaries allows VLAs to scale their ‘working memory’ without incurring prohibitive computational costs. Check out the Paper and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Physical Intelligence Team Unveils MEM for Robots: A Multi-Scale Memory System Giving Gemma 3-4B VLAs 15-Minute Context for Complex Tasks appeared first on MarkTechPost.

Alibaba has released OpenSandbox, an open-source tool designed to provide AI agents with secure, isolated environments for code execution, web browsing, and model training. Released under the Apache 2.0 license, the proposed system targets to standardize the ‘execution layer’ of the AI agent stack, offering a unified API that functions across various programming languages and infrastructure providers. The tool is built on the same internal infrastructure Alibaba utilizes for large-scale AI workloads. The Technical Gap in Agentic Workflows Building an autonomous agent typically involves two components: the ‘brain’ (usually a Large Language Model) and the ‘tools’ (code execution, web access, or file manipulation). Providing a safe environment for these tools has required developers to manually configure Docker containers, manage complex network isolation, or rely on third-party APIs. OpenSandbox addresses this by providing a standardized, secure environment where agents can execute arbitrary code or interact with interfaces without risking the host system’s integrity. It abstracts the underlying infrastructure, allowing developers to move from local development to production-scale deployments using a single API. Architecture The architecture of OpenSandbox is built on a modular four-layer stack—comprising the SDKs Layer, Specs Layer, Runtime Layer, and Sandbox Instances Layer—designed to decouple client logic from execution environments. At its core, the system utilizes a FastAPI-based server to manage the lifecycle of sandboxes via the Docker or Kubernetes runtimes, while communication is standardized through OpenAPI specifications (the Sandbox Lifecycle and Execution Specs). Within each isolated container, OpenSandbox injects a high-performance Go-based execution daemon (execd) that interfaces with internal Jupyter kernels to provide stateful code execution, real-time output streaming via Server-Sent Events (SSE), and comprehensive filesystem management, ensuring a ‘protocol-first’ approach that remains consistent across any base container image. https://open-sandbox.ai/overview/architecture Core Technical Capabilities OpenSandbox is designed to be environment-agnostic. It supports Docker for local development and Kubernetes for distributed, production-grade runs. The platform provides four primary types of sandboxes: Coding Agents: Environments optimized for software development tasks, where agents can write, test, and debug code. GUI Agents: Supports full VNC desktops, enabling agents to interact with graphical user interfaces. Code Execution: High-performance runtimes for executing specific scripts or computational tasks. RL Training: Isolated environments tailored for Reinforcement Learning (RL) workloads, allowing for safe iterative training. The system utilizes a Unified API, which ensures that the interaction patterns remain consistent regardless of the underlying language or runtime. Currently, OpenSandbox provides SDKs for Python, TypeScript, and Java/Kotlin, with C# and Go listed on the development roadmap. Integration and Ecosystem Support A significant feature of OpenSandbox is its native compatibility with existing AI frameworks and developer tools. By providing a secure execution layer, it allows agents built on various platforms to perform ‘real-world’ actions. The integrations currently supported include: Model Interfaces: Claude Code, Gemini CLI, and OpenAI Codex. Orchestration Frameworks: LangGraph and Google ADK (Agent Development Kit). Automation Tools: Chrome and Playwright for browser-based tasks. Visualization: Full VNC support for visual monitoring and interaction. This means that an agent can be tasked with ‘scraping a website and training a linear regression model’ within a single, isolated session. The agent uses Playwright to navigate the web, downloads data to the sandbox’s local file system, and executes Python code to process that data—all without leaving the secured OpenSandbox environment. Deployment and Configuration The project prioritizes a streamlined developer experience (DX). Setting up a local execution server requires three primary commands through the command-line interface: pip install opensandbox-server — Installs the server components. opensandbox-server init-config — Generates the necessary configuration files for the environment. opensandbox-server — Launches the server and exposes the API for agent interaction. Once the server is running, developers can use the provided SDKs to create, manage, and terminate sandboxes programmatically. This reduces the operational overhead of ‘stitching together’ multiple tools for file management, process isolation, and network proxying. Key Takeaways Unified, Language-Agnostic Execution: OpenSandbox provides a consistent API for AI agents to execute code, browse the web, and interact with GUIs. While it currently supports Python, TypeScript, and Java/Kotlin, SDKs for C# and Go are on the roadmap. Infrastructure Flexibility (Docker & Kubernetes): The tool is designed to scale seamlessly from a developer’s local machine to enterprise-grade production. It utilizes Docker for local isolation and Kubernetes for distributed, large-scale deployments, eliminating the ‘environment drift’ often found when moving agents from dev to cloud. Broad Ecosystem Integration: It is engineered to plug directly into leading AI frameworks and tools, including LangGraph, Claude Code, Gemini CLI, OpenAI Codex, and Google ADK, as well as automation libraries like Playwright and Chrome. Elimination of ‘Sandbox Dependency’: By providing a free, open-source alternative under the Apache 2.0 license, Alibaba removes the dependency on expensive, managed sandbox services that charge per-minute fees or impose vendor lock-in. High-Fidelity Interaction (VNC & Web): Beyond simple script execution, OpenSandbox supports full VNC desktops and browser automation. This allows agents to perform complex, multi-modal tasks—such as navigating web interfaces or using desktop applications—within a secure, ‘blast-resistant’ environment. Check out the Repo, Docs and Examples. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Alibaba Releases OpenSandbox to Provide Software Developers with a Unified, Secure, and Scalable API for Autonomous AI Agent Execution appeared first on MarkTechPost.

On June 1, 2023, as a sweltering heat wave baked Quebec, thousands of lightning strikes flashed across the province, setting off more than 120 wildfires. The blazes ripped through parched forests and withered grasslands, burned for weeks, and compounded what was rapidly turning into Canada’s worst fire year on record. In the end, nearly 7,000 fires scorched tens of millions of acres across the country, generated nearly 500 millions tons of carbon emissions, and forced hundreds of thousands of people to flee their homes. Lightning sparked almost 60% of the wildfires—and those blazes accounted for 93% of the total area burned. Now a Vancouver-based weather modification startup, Skyward Wildfire, says it can prevent such catastrophic fires in the future—by stopping the lightning strikes that ignite them. It just raised millions of dollars in a funding round that it plans to use to accelerate its product development and expand its operations. Until last week the company, which highlights the role lightning played in the 2023 infernos, stated on its website that it has demonstrated technology capable of preventing “up to 100% of lightning strikes.” It was an eye-catching claim that went well beyond the confidence level of researchers who have studied the potential for humans to suppress lightning—and the company took it down following inquiries from MIT Technology Review. “While the statement reflected an observed result under specific conditions, it was not intended to suggest uniform outcomes and has been removed,” Nicholas Harterre, who oversees government partnerships at Skyward, said in an email. “In complex atmospheric systems, consistent 100% outcomes are not realistic, as the experts you spoke to rightly pointed out.”  The company now states it demonstrated that it “can prevent the majority of cloud-to-ground lightning strikes in targeted storm cells.” So far, Skyward hasn’t publicly revealed how it does so, and in response to our questions Harterre said only that the materials are “inert and selected in accordance with regulatory standards.”  But online documents suggest the company is relying on an approach that US government agencies began evaluating in the early 1960s: seeding clouds with metallic chaff, or narrow fiberglass strands coated with aluminum.  The military uses the material to disrupt radar signals; fighter jets, for example, deploy it during dogfights to throw off guided missile systems. Field trials conducted decades ago by US agencies suggest it could help reduce lightning strikes, at least to some degree and under certain conditions. If Skyward could employ it reliably on significant scales, it might offer a powerful tool for countering rising fire risks as climate change drives up temperatures, dries out forests, and likely increases the frequency of lightning strikes. “Preventing lightning on high-risk days saves lives, billions in wildfire costs, and is one of the highest-leverage and most immediate climate solutions available,” Sam Goldman, Skyward’s founder and chief executive, said in a statement posted on LinkedIn last year. But researchers and environmental observers say there are plenty of remaining uncertainties, including how well the seeding may work under varying weather and climate conditions, how much material would need to be released, how frequently it would have to be done, and what sorts of secondary environmental impacts might result from lighting suppression on commercial scales. Some observers are also concerned that the company appears to have moved ahead with weather modification field trials in parts of Canada without providing wide public notice or openly discussing what materials it’s putting into the clouds. Given the escalating fire dangers, it’s “reasonable” to evaluate the potential for new technologies to mitigate them, says Keith Brooks, programs director at Environmental Defence, a Canadian advocacy organization. “But we should be doing so cautiously and really transparently, with a robust scientific methodology that’s open to scrutiny,” he says. Seeding the clouds Skyward’s website offers few technical details, but the company says it worked with Canadian wildfire agencies in 2024 and 2025 to demonstrate its technology. The company also says it has developed AI tools to predict lightning strikes that could set off fires. Skyward announced last month that it raised $7.9 million in Canadian dollars ($5.7 million), in an extension of a seed round initially closed early last year. Investors included Climate Innovation Capital, Active Impact Investments, and Diagram Ventures. “Our first season demonstrated that prevention is possible at scale,” Goldman said in a statement. “This funding allows us to expand into new regions and support partners who need reliable, operational tools to reduce wildfire risk before emergencies begin.” The company doesn’t use the term “cloud seeding” on its site or in its recent announcements. But a press release highlighting its selection as a finalist last year in a conservation group’s Fire Grand Challenge states that it suppresses lightning “by cloud seeding with safe, non-toxic materials to neutralize storm charges,” as The Narwhal previously reported. In addition, Unorthodox Philanthropy, a foundation that provided a grant to support Skyward’s efforts “to test and deploy” the technology, offered more detail in an awardee write-up about Goldman. It states: “The Skyward team … settled on an inert substance consisting of aluminum covered glass fibers, which is regularly used in military operations to intercept and confuse enemy radar and can also dis-charge clouds.” Additional details were disclosed in a document marked “Proprietary and Confidential,” which the World Bank nonetheless released within a package of materials from companies developing means of addressing fire risks. Skyward’s diagrams show planes dropping particles into clouds to prevent cloud-to-ground lightning strikes in “high risk areas.” The company also notes in the document that it uses artificial intelligence for a number of purposes, including forecasting lightning storms, prioritizing treatments, targeting storm cells, and optimizing flight paths.   Harterre stressed that the company would deploy the technology judiciously and reserve it for storm events with elevated wildfire risk, adding that such storms account for less than 0.1% of lightning activity in a given area. “Our objective is to reduce the probability of ignition on the limited number of extreme-risk days when fires threaten lives, critical infrastructure, and ecosystems,

  You’ve built an AI agent that works well in development.