{"id":103470,"date":"2026-07-11T19:10:55","date_gmt":"2026-07-11T19:10:55","guid":{"rendered":"https:\/\/youzum.net\/ant-groups-robbyant-unveils-lingbot-va-2-0-a-causal-video-action-model-built-natively-for-physical-ai\/"},"modified":"2026-07-11T19:10:55","modified_gmt":"2026-07-11T19:10:55","slug":"ant-groups-robbyant-unveils-lingbot-va-2-0-a-causal-video-action-model-built-natively-for-physical-ai","status":"publish","type":"post","link":"https:\/\/youzum.net\/it\/ant-groups-robbyant-unveils-lingbot-va-2-0-a-causal-video-action-model-built-natively-for-physical-ai\/","title":{"rendered":"Ant Group\u2019s Robbyant Unveils LingBot-VA 2.0: A Causal Video-Action Model Built Natively for Physical AI"},"content":{"rendered":"

Robbyant, the embodied AI unit inside Ant Group, has released the LingBot-VA 2.0<\/a><\/strong>.The first embodied-native foundation model. It describes a video-action foundation model for generalist robot manipulation. The research team pretrains the whole stack for embodiment instead of fine-tuning a video generator.<\/p>\n

What is LingBot-VA 2.0?<\/strong><\/h2>\n

Most video-action models reuse two components built for digital content creation. One is a reconstruction-oriented VAE. The other is a bidirectional video-diffusion backbone, with an action module attached.<\/p>\n

This creates three limitations. Pixel-reconstruction latents preserve appearance but carry limited physical structure. Iterative denoising over video tokens is too slow for closed-loop control. Generic video objectives never teach how actions reshape the world.<\/p>\n

A fourth mismatch is structural. Backbones use bidirectional attention, while control unfolds strictly forward in time. LingBot VA Version 1.0 finetuned that stack into a causal model. Version 2.0 pretrains a causal DiT natively.<\/p>\n

\n
\"\"
https:\/\/github.com\/Robbyant\/lingbot-va\/blob\/main\/LingBot_VA2_paper.pdf<\/figcaption><\/figure>\n<\/div>\n

Version 1: The Semantic Visual-Action Tokenizer<\/strong><\/h2>\n

Building on that motivation, stage one replaces the compression-only VAE. Following RepWAM, the tokenizer adds two objectives to reconstruction.<\/p>\n

Semantic alignment pulls visual latents toward a frozen Perception Encoder teacher. A latent-action objective extracts compact transition variables between consecutive latents. An inverse dynamics model predicts each latent action. A forward dynamics model decodes it into a transport map plus residual.<\/p>\n

World states and actions now share one latent space. Unlabeled web video therefore carries action-relevant supervision.<\/p>\n

Version 2: A Causal DiT With a Sparse MoE Video Stream<\/strong><\/h2>\n

On top of that space, version 2 pretrains a causal DiT. It keeps the Mixture-of-Transformers layout of version 1.0. A video expert and an action expert share one causal self-attention. Each owns a separate feed-forward pathway.<\/p>\n

The two streams scale asymmetrically. The video expert replaces its dense FFN with a sparse MoE routed layer. That layer holds 128 routed SwiGLU experts, top-8 routing, one shared expert. Load balancing follows the auxiliary-loss-free Loss-Free Balancing strategy. The action expert keeps a dense FFN at hidden dimension 768.<\/p>\n

The video backbone is roughly 13.0B parameters, about 1.9B active. With the action expert and MCP heads, training covers about 15.3B parameters. Roughly 2.5B activate per token at inference. Training uses a rectified-flow objective with a hybrid Muon plus AdamW optimizer.<\/p>\n

Where the Training Signal Comes From<\/strong><\/h2>\n

Beyond architecture, two objectives shape what the model learns.<\/p>\n

Multi-chunk prediction (MCP) fixes myopic supervision. Teacher forcing supervises only the next chunk, so the model can cut loss by copying appearance. MCP attaches three lightweight modules predicting the next three chunks. In ablation it matched the baseline\u2019s 45k-step accuracy in 20k steps, a 2.3x training speedup.<\/p>\n

Meanwhile, five objectives are co-trained rather than staged: T2I, T2V, TI2VA, ICL, and human-robot co-training. Sampling follows a coarse-to-fine schedule, from appearance grounding to video-action control. Keeping every objective alive avoids forgetting the earlier priors.<\/p>\n

Hierarchical Planning<\/strong><\/h2>\n

Chunk-level control cannot sequence long-horizon goals. Above the policy therefore sits a VLM planner, LoRA-finetuned with a frozen vision tower. It emits structured JSON: done, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The policy reads it at each chunk boundary, so planner latency never blocks execution.<\/p>\n

Foresight Reasoning<\/strong><\/h2>\n

Even with a sparse backbone, deployment hits a serial bottleneck. If the robot waits, model latency becomes control latency.<\/p>\n

Foresight Reasoning therefore runs prediction and execution as asynchronous streams. While the robot executes chunk a_t, the video expert imagines its outcome. The action expert decodes a_{t+1} from that.<\/p>\n

Running ahead risks drift. So each returning observation is encoded into the true latent z_{t+1}, overwriting the imagined one. A forward-dynamics grounding loss trains the video expert for this role.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).\n# Not runnable: policy, executor and encode() are placeholders.\n\nC = init_kv_cache(encode(obs_0))            # feedback-grounded cache C_t\na = policy.action_expert(C)                 # cold start: first action chunk a_0\n\nwhile not done:\n    executor.start(a)                       # execution stream, non-blocking\n\n    C_tmp  = C + [a]                        # prediction stream: C_t u {a_t}\n    z_hat  = policy.video_expert(C_tmp)     # forward dynamics -> imagined z_{t+1}\n    a_next = policy.action_expert(C_tmp + [z_hat])\n\n    obs = executor.wait()                   # real observation of a_t returns\n    C   = overwrite(C_tmp, z_hat, encode(obs))   # re-ground: z_hat <- true z_{t+1}\n    a   = a_next\n<\/code><\/pre>\n<\/div>\n<\/div>\n
Performance<\/strong><\/h2>\n
Consequently, evaluation covers simulation and real hardware. On RoboTwin 2.0, every model trains on 2,500 clean plus 25,000 randomized demonstrations, across 50 tasks.<\/p>\n
\n
\"\"
https:\/\/technology.robbyant.com\/lingbot-va-v2<\/figcaption><\/figure>\n<\/div>\n
\n\n\n\n\n\n\n\n\n
Method<\/strong><\/td>\nClean<\/strong><\/td>\nRandomized<\/strong><\/td>\nAvg.<\/strong><\/td>\n<\/tr>\n
X-VLA<\/td>\n72.9<\/td>\n72.8<\/td>\n72.9<\/td>\n<\/tr>\n
\u03c00.5<\/td>\n82.7<\/td>\n76.8<\/td>\n79.8<\/td>\n<\/tr>\n
Motus<\/td>\n88.7<\/td>\n87.0<\/td>\n87.9<\/td>\n<\/tr>\n
LingBot-VA<\/td>\n92.9<\/td>\n91.6<\/td>\n92.2<\/td>\n<\/tr>\n
LingBot-VA 2.0<\/strong><\/td>\n93.8<\/strong><\/td>\n93.4<\/strong><\/td>\n93.6<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
\n\n\n\n\n\n\n\n\n
Acceleration technique<\/strong><\/td>\nInference time (ms\/chunk)<\/strong><\/td>\nAsync Hz<\/strong><\/td>\n<\/tr>\n
BF16 PyTorch async rollout baseline<\/td>\n927<\/td>\n35<\/td>\n<\/tr>\n
+ Consistency distillation<\/td>\n466<\/td>\n69<\/td>\n<\/tr>\n
+ Low-precision compiled execution<\/td>\n369<\/td>\n87<\/td>\n<\/tr>\n
+ Long-horizon attention optimization<\/td>\n272<\/td>\n118<\/td>\n<\/tr>\n
+ Runtime overhead reduction<\/td>\n142<\/td>\n225<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
Distillation cuts the video sampler from 5 steps to 2, and the action sampler from 10 to 2. FP8 TensorRT engines, a paged\/ragged KV cache with FlashInfer attention, and host-side overhead removal supply the rest.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
# Reproduces Table 3 of the report exactly. Runnable as-is.\nK = 32  # low-level control steps inside one generated chunk\n\nstack = [(\"BF16 PyTorch async rollout baseline\", 927),\n         (\"+ Consistency distillation\",          466),\n         (\"+ Low-precision compiled execution\",  369),\n         (\"+ Long-horizon attention optimization\", 272),\n         (\"+ Runtime overhead reduction\",        142)]\n\nfor name, ms in stack:\n    print(f\"{name:40s} {ms:4d} ms  {round(1000 \/ ms * K):4d} Hz\")\n\nprint(\"end-to-end speedup:\", round(927 \/ 142, 1), \"x\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
Version 1.0 vs Version 2.0<\/strong><\/h2>\n
\n\n\n\n\n\n\n\n\n\n
Dimension<\/strong><\/td>\nLingBot-VA<\/strong><\/td>\nLingBot-VA 2.0<\/strong><\/td>\n<\/tr>\n
Tokenizer<\/td>\nWan2.2 VAE (reconstruction)<\/td>\nSemantic visual-action tokenizer, 96 latent channels<\/td>\n<\/tr>\n
Backbone origin<\/td>\nFinetuned from a bidirectional generator<\/td>\nCausal DiT pretrained from scratch<\/td>\n<\/tr>\n
Video FFN<\/td>\nDense<\/td>\nSparse MoE, 128 experts, top-8<\/td>\n<\/tr>\n
Extra supervision<\/td>\nNot used<\/td>\nMCP, in-context learning, human-robot co-training<\/td>\n<\/tr>\n
Inference<\/td>\nAsync execution, KV cache<\/td>\nForesight Reasoning with observation re-grounding<\/td>\n<\/tr>\n
Peak async control<\/td>\nNot reported in the version 2.0 report<\/td>\n225 Hz<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
The tokenizer ablation isolates row one. Swapping the WAN2.2 VAE for the semantic tokenizer lifts a 1.3B model from 78.0 to 86.6.<\/p>\n
Use Cases and Examples<\/strong><\/h2>\n
Beyond benchmarks, four deployment shapes stand out.<\/p>\n