Uncategorized

Datalab’s Lift is a focused document extraction tool with a specific promise: give it a PDF or image plus a JSON Schema, and it returns schema-shaped JSON directly. Instead of converting a document to Markdown first and then asking another model to extract fields, Lift reads rendered page images and attempts to emit the final structured object in a single pass. According to Datalab, Lift is a 9B vision model for structured JSON extraction from PDFs and images, supports schema-constrained decoding, and returns JSON that matches the user’s schema. That positioning matters because Lift is not mainly an OCR engine, not mainly a PDF-to-Markdown converter, and not a full enterprise document review platform. It is best understood as a schema-first document extractor: a model for turning visually complex documents into application-ready fields. First, the distinction that organizes everything: parsing vs. extraction Most document AI tools solve one of two different problems: Parsers turn documents into faithful intermediate representations: Markdown, HTML, JSON blocks, layout trees, tables, headings, reading order, and chunks for retrieval. Tools such as Docling, MinerU, Marker, Unstructured, PyMuPDF, OCRmyPDF, and Surya primarily fall into this category. Their output is document-shaped. Extractors turn documents into the fields an application actually needs. You define a schema — for example, invoice_number, vendor_name, total, due_date, or line_items[] — and the system tries to return those values directly. Lift, NuExtract3, LlamaExtract, Reducto Extract, Extend, Azure Content Understanding, and other cloud extraction APIs belong closer to this category. Their output is schema-shaped. That distinction matters because many production systems still follow a parse-then-extract pattern: convert a PDF to Markdown or structured text, then send that representation to an LLM with a schema. Lift’s bet is to collapse that workflow into a single visual extraction pass. That can reduce pipeline complexity, but only when the real goal is field extraction rather than faithful document reconstruction. The competitive map Lift sits at the intersection of several overlapping categories: Open-weight extraction VLMs such as NuExtract3 Frontier multimodal LLMs with structured-output modes Cloud document AI systems such as Azure, Google, and AWS Commercial extraction platforms such as Reducto, Extend, LlamaExtract, and Datalab’s own API Open-source document parsers such as Docling, MinerU, Marker, and Unstructured Structured-generation libraries such as XGrammar, Outlines, Instructor, BAML, and related JSON-output systems The important point is that not all of these tools are direct competitors. Some compete with Lift directly. Others are adjacent infrastructure. A parser like Docling is not trying to solve the same problem as Lift. A constrained-decoding library is not a document model at all. A commercial extraction platform may include extraction models, citations, review workflows, and compliance infrastructure. Lift is narrower: it is the raw schema-first extractor. Lift vs. NuExtract3: the closest open-weight comparison NuExtract3 is probably Lift’s closest open-weight competitor. NuMind describes NuExtract3 as a unified 4B vision-language reasoning model for document understanding, combining structured information extraction with image-to-Markdown conversion for documents such as scans, receipts, forms, invoices, contracts, and tables. Its Hugging Face model card lists it under an Apache-2.0 license. The contrast is straightforward. Lift is larger at 9B and, in Datalab’s own benchmark, reports stronger field accuracy than NuExtract3: 90.2% versus 81.5%. NuExtract3 is smaller, more permissively licensed, and also positioned as a Markdown-conversion model. So the practical decision is not only accuracy. If the priorities are permissive licensing, smaller local deployment, and a single model that can also convert documents to Markdown, NuExtract3 is attractive. If the priority is schema-first field extraction with Datalab’s reported speed-accuracy trade-off, Lift becomes more compelling. Lift vs. frontier multimodal LLMs A common alternative is to send the document to a frontier multimodal LLM and ask for structured output. In Datalab’s benchmark, Gemini Flash 3.5 slightly outperforms Lift on field accuracy and full-document accuracy, while Lift is much faster in the reported setup: 9.5 seconds median latency for Lift versus 28.1 seconds for Gemini Flash 3.5. That does not mean Lift is always better. Frontier models remain attractive when volume is modest, setup time matters more than infrastructure control, and cloud processing is acceptable. Lift’s advantage appears when latency, data residency, repeatable self-hosting, and large-volume cost control matter. Lift vs. cloud document AI platforms Azure AI Document Intelligence, Azure Content Understanding, Google Document AI, and AWS Textract are managed cloud services rather than just models. They provide enterprise infrastructure for document processing, including deployment controls, service reliability, monitoring, procurement processes, and integration with broader cloud ecosystems. Microsoft describes Azure Content Understanding as a way to transform unstructured data into structured, machine-readable information while preserving structural relationships. In Datalab’s benchmark, Azure Content Understanding reports lower field accuracy and higher latency than Lift, but it includes citations, which Lift’s open weights do not. Datalab’s own hosted API also adds per-field verification, citations, and confidence scores beyond the open model. This is the cloud tradeoff. Cloud platforms are usually easier to adopt within companies already standardized on Azure, Google Cloud, or AWS. They may also be stronger choices when enterprise governance matters more than raw speed. Lift’s counterargument is portability: teams can run the extraction model locally or through their own vLLM deployment rather than sending every document to a hosted API. For handwriting-heavy, low-quality scans, clinical forms, annotation-heavy documents, or regulated workflows requiring traceability, the cloud and managed platforms should be benchmarked directly against Lift rather than assumed inferior. Lift vs. commercial extraction platforms Reducto, Extend, LlamaExtract, Mindee, and Datalab’s own hosted API occupy a different layer of the market. They are not only extraction models; they are extraction systems. Their value is not limited to field accuracy. They add provenance, review workflows, schema management, confidence scoring, citations, deployment controls, and enterprise compliance. Reducto’s Extract product is positioned around schema-typed JSON extraction with optional citations, while its Parse product emphasizes typed blocks, bounding boxes, and confidence scores. LlamaExtract similarly advertises custom-schema extraction with granular citations and confidence scores. This is where Lift’s open model is intentionally thinner. The open weights prioritize fast, schema-first extraction. Datalab’s hosted API adds the production features

In this tutorial, we explore NVIDIA’s cosmos-framework from a practical Colab-friendly angle while staying honest about the hardware limits of running real Cosmos 3 checkpoints. We begin by checking the current runtime, GPU capabilities, CUDA availability, memory, and disk space to understand why full Cosmos 3 inference is not realistic on standard Colab hardware. Instead of stopping there, we use the framework’s real structure, CLI surface, input schema, and model modes as the foundation for a hands-on miniature implementation. We then build and train a compact omnimodal Mixture-of-Transformers world model that mirrors the core Cosmos idea: shared cross-modal attention with modality-specific expert routing for text, vision, and action streams. Using synthetic physical-world data, training-loss tracking, and an autoregressive rollout, we show how the model learns relationships across modalities and predicts future latent states in a simplified yet technically meaningful way. Probing Colab Hardware Limits Copy CodeCopiedUse a different Browser import os, sys, json, time, math, textwrap, subprocess, shutil, platform from pathlib import Path def rule(title=””): line = “=” * 86 print(“n” + line + (“n ” + title if title else “”) + “n” + line) def spark(vals, width=60): “””Tiny ASCII sparkline for a 1-D sequence (works with no plotting libs).””” if not vals: return “” blocks = “▁▂▃▄▅▆▇█” lo, hi = min(vals), max(vals) rng = (hi – lo) or 1.0 step = max(1, len(vals) // width) s = “”.join(blocks[min(len(blocks) – 1, int((v – lo) / rng * (len(blocks) – 1)))] for v in vals[::step]) return s rule(“SECTION 0 — Environment probe: what you have vs. what Cosmos 3 actually needs”) IN_COLAB = “google.colab” in sys.modules print(f”Running inside Google Colab : {IN_COLAB}”) print(f”Python : {platform.python_version()} ({platform.system()})”) try: import torch except ModuleNotFoundError: print(“torch not found — installing CPU build (a few seconds)…”) subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, “torch”], check=False) import torch print(f”PyTorch : {torch.__version__}”) CUDA_OK = torch.cuda.is_available() DEVICE = torch.device(“cuda” if CUDA_OK else “cpu”) gpu_name, gpu_mem_gb, cc = “None (CPU)”, 0.0, (0, 0) if CUDA_OK: p = torch.cuda.get_device_properties(0) gpu_name = p.name gpu_mem_gb = p.total_memory / 1024**3 cc = torch.cuda.get_device_capability(0) print(f”CUDA build : {torch.version.cuda}”) print(f”GPU : {gpu_name}”) print(f”GPU memory : {gpu_mem_gb:.1f} GiB”) print(f”Compute capability : sm_{cc[0]}{cc[1]}”) try: free_gb = shutil.disk_usage(‘/’).free / 1024**3 print(f”Free disk : {free_gb:.0f} GiB”) except Exception: free_gb = 0.0 AMPERE = cc[0] >= 8 reqs = [ (“GPU architecture”, “Ampere+ (sm_80+, A100/RTX30xx)”, “OK” if AMPERE else “TOO OLD (T4=sm_75)”), (“GPU memory”, “>=80 GiB for Nano-16B (single H100)”, “OK” if gpu_mem_gb >= 79 else f”{gpu_mem_gb:.0f} GiB — insufficient”), (“CUDA toolkit”, “>=12.8”, “check” ), (“Free disk”, “~150 GiB first run (~1 TB HF cache)”, “OK” if free_gb >= 150 else f”{free_gb:.0f} GiB — insufficient”), (“Attention kernels”,”FlashAttn-3 (Hopper) / FA2 (Ampere)”, “needs Ampere+”), ] print(“n Can this machine run the REAL Cosmos 3 checkpoints?”) print(” ” + “-” * 82) print(f” {‘Requirement’:<18}{‘Cosmos 3 needs’:<38}{‘You have’}”) print(” ” + “-” * 82) for k, need, have in reqs: print(f” {k:<18}{need:<38}{have}”) print(” ” + “-” * 82) VERDICT = AMPERE and gpu_mem_gb >= 79 and free_gb >= 150 print(f” VERDICT: {‘This machine could attempt Nano-16B.’ if VERDICT else ‘NO — real Cosmos 3 inference is not possible here. Educational path below.’}”) We begin by preparing the runtime utilities and checking whether the current machine can realistically support Cosmos 3 inference. We inspect Python, PyTorch, CUDA, GPU memory, compute capability, and available disk space to compare our environment against the actual hardware requirements. We then print a clear verdict explaining why the real 16B+ Cosmos checkpoints cannot usually run on standard Colab hardware. Copy CodeCopiedUse a different Browser rule(“SECTION 1 — Clone & map the real cosmos_framework package (source of truth)”) Mapping The Cosmos-Framework Package Copy CodeCopiedUse a different Browser REPO = “https://github.com/NVIDIA/cosmos-framework.git” DST = Path(“/content/cosmos-framework”) if Path(“/content”).exists() else Path(“cosmos-framework”) cloned = False try: if not DST.exists(): print(f”Shallow-cloning {REPO} …”) subprocess.run([“git”, “clone”, “–depth”, “1”, REPO, str(DST)], check=True, capture_output=True, text=True, timeout=180) cloned = DST.exists() except Exception as e: print(f”(Clone skipped/failed — offline is fine, tutorial continues.) {e}”) if cloned: print(f”Repo at: {DST}n”) pkg = DST / “cosmos_framework” if pkg.exists(): print(“cosmos_framework/ subpackages (the real code layout):”) for child in sorted(pkg.iterdir()): if child.is_dir() and not child.name.startswith((“_”, “.”)): n_py = len(list(child.rglob(“*.py”))) print(f” • {child.name:<20} ({n_py:>3} .py files)”) example = DST / “inputs” / “omni” / “t2v.json” if example.exists(): print(f”nReal example input spec ({example.relative_to(DST)}):”) print(textwrap.indent(example.read_text().strip(), ” “)) else: print(“Proceeding without a local clone (we already extracted the real schema/CLI).”) print(“”” Real CLI surface (docs/inference.md): Single GPU : python -m cosmos_framework.scripts.inference \ –parallelism-preset=latency -i “inputs/omni/t2v.json” \ -o outputs/omni_nano –checkpoint-path Cosmos3-Nano –seed 0 Multi GPU : torchrun –nproc-per-node=8 -m cosmos_framework.scripts.inference \ –parallelism-preset=throughput -i “inputs/omni/*.json” \ -o outputs/omni_super –checkpoint-path Cosmos3-Super –seed 0 Models : Cosmos3-Nano (16B, all modes) | Cosmos3-Super (65B, t2i/t2v/i2v) Modes : text2image · text2video · image2video · video2video · forward_dynamics · inverse_dynamics · policy Parallelism: FSDP dp-shard / dp-replicate · context (cp) · CFG (cfgp) presets {latency, throughput} Guardrails : Cosmos-Guardrail1 + Qwen3Guard-Gen-0.6B + RetinaFace (on by default) “””) rule(“SECTION 2 — Omnimodal Mixture-of-Transformers (MoT) world model — the idea”) print(r””” Cosmos 3 unifies language, image, video, audio and ACTION in ONE model. The key trick is a Mixture-of-Transformers: every modality is turned into tokens placed on a SINGLE interleaved sequence; SELF-ATTENTION is SHARED across all modalities (so vision can be conditioned on text, actions on vision, etc.), but each token is processed by a MODALITY-SPECIFIC expert feed-forward block (“Mixture-of-Transformers” routing). text tokens vision tokens action tokens [t0 t1 t2 …] [v0 v1 v2 …] [a0 a1 …] | / | / +———– one sequence ———–+ | ┌─────────── shared causal self-attention (RoPE) ───────────┐ │ every token attends to all earlier tokens, ANY modality │ └───────────────────────────────────────────────────────────┘ | route each token to its modality’s EXPERT FFN (SwiGLU): text→Expert0 vision→Expert1 action→Expert2 | per-modality heads: next-token / next-latent / next-action Physical-AI modes fall right out of this one model: text2video = generate the vision-token stream from a text prompt image2video = condition vision stream on a first frame + text forward_dynamics= given frames + ACTIONS, roll future frames forward (a world model) inverse_dynamics= given frames, infer the ACTIONS that caused them policy =

arXiv:2607.06054v1 Announce Type: cross Abstract: Off-the-shelf TTS systems are poorly adapted to Taiwanese Mandarin. Their accent defaults to other Mandarin variants, their tokenizers over-segment common Taiwanese text, and their pronunciation degrades at code-switching boundaries where Chinese and English alternate within one utterance. These problems share one root: the text side lacks adaptation to the Taiwanese context. We address the text side from the bottom up. PangolinTokenizer, a byte-level BPE tokenizer trained on Taiwan-context data, reaches the lowest token rate (0.485 tokens/character) with the smallest vocabulary among nine tokenizers. Barbet, a billion-parameter Traditional-Chinese language model trained on PangolinTokenizer, serves as the text-semantic frontend and ranks first among comparable public models on a 14-task evaluation. BlueMagpie-TTS attaches Barbet to the pretrained acoustic stack of VoxCPM2 through a learned bridge, keeping the acoustic stack fixed. On a 1000-sentence Taiwan-localized test set, it lowers CER from 11.45% to 4.81% and WER from 14.83% to 5.36%, relative reductions of 58.0% and 63.9%. In a blind listening study on 500 of these sentences with ten listeners, 65.6% of majority votes prefer BlueMagpie-TTS.

Today, OpenAI released GPT-Live. It is a new generation of voice models. GPT-Live now powers the ChatGPT Voice experience. The stated goal is natural, real-time conversation with AI. Two versions ship first: GPT-Live-1 and GPT-Live-1 mini. Both roll out to ChatGPT users globally today. TL;DR GPT-Live is a full-duplex voice model family that listens and speaks at once. It delegates search and reasoning to GPT-5.5 while keeping the conversation flowing. GPT-Live-1 and mini were strongly preferred over Advanced Voice Mode in human tests. It ships today to ChatGPT users globally; the API is planned soon. Video, screen sharing, and full multilingual parity are not available at launch. What is GPT-Live? GPT-Live is built on a full-duplex architecture. Full-duplex means the model can listen and speak at the same time. During a conversation, it can add short cues like ‘mhmm’ or ‘yeah.’ It can engage in quick back-and-forth, or stay quiet when you think. For questions needing web search, deeper reasoning, or complex work, GPT-Live delegates. It hands the task to a frontier model behind the scenes. The result returns to the conversation when it is ready. At launch, that background model is GPT-5.5. While the frontier model works, GPT-Live keeps the conversation going. Why Cascaded and Turn-Based Voice Fell Short Earlier voice systems moved toward natural conversation, but with tradeoffs. Cascaded voice systems chained three separate models per turn. A speech-to-text model transcribed your speech first. A large language model then produced a response. A text-to-speech model converted that text back into audio. This let people talk to frontier models for the first time. But information could be lost across models, and responses were slow and stilted. Turn-based voice models, like ChatGPT Advanced Voice Mode, processed audio inside one model. That reduced latency and made conversations smoother. They still operated through discrete turns, waiting for the user to stop speaking. Turn detection was based on silence. A brief pause or background noise could be mistaken for the end of a turn. This caused the model to interrupt at unnatural times. Dimension Cascaded (original ChatGPT Voice) Turn-based (Advanced Voice Mode) Full-duplex (GPT-Live) Pipeline STT → LLM → TTS, three models Single model handling audio Single model, continuous processing Turn handling Discrete turns Discrete turns, silence-based Continuous, decisions many times/sec Listen while speaking No No Yes Backchannels (“mhmm”) No No Yes Latency feel Slow, stilted, long pauses Faster, smoother, still rigid Fast, natural, expressive Interrupt handling Not supported Can misfire on pauses/noise Can pause, interrupt, resume Deeper work In-line LLM In-line model Delegates to GPT-5.5 in background The Two Architectural Changes GPT-Live addresses these limits with two changes: Continuous interaction using full-duplex processing: The model processes input while generating output at the same time. It can make interaction decisions many times per second. Those decisions include whether to speak, continue listening, pause, interrupt, or invoke a tool. This supports more natural back-and-forth and a better sense of time. It also enables live translation. Delegation for deeper work: OpenAI decoupled continuous interaction from heavier reasoning. When a task needs search, reasoning, or more agentic capabilities, GPT-Live delegates it. Another model, such as GPT-5.5, handles that work in the background. Meanwhile, GPT-Live keeps the conversation flowing. This design also lets GPT-Live adopt newer frontier models as they ship. What OpenAI’s Evaluations Show OpenAI built new human evaluations for pleasantness and conversational flow. Evaluators compared models in matched five-to-ten-minute conversations. In these head-to-head tests, GPT-Live-1 and GPT-Live-1 mini were strongly preferred over Advanced Voice Mode. The comparisons measured overall preference, turn-taking, interruptions, flow, and how natural each interaction felt. On automated benchmarks, GPT-Live-1 also showed gains over Advanced Voice Mode: GPQA: GPT-Live-1 substantially outperforms it on expert-level science reasoning. BrowseComp: GPT-Live-1 shows strong gains on agentic web search. τ³-Voice Telecom (internal variant): GPT-Live-1 outperforms it on multi-turn telecom support tasks. OpenAI team notes it used a customized user model for the τ³-Voice Telecom eval. That user model was powered by its latest reasoning models. GPT-Live-1 (instant) and GPT-Live-1 mini use GPT-5.5 Instant in the background. GPT-Live-1 Medium and GPT-Live-1 High use GPT-5.5 Thinking with medium and high reasoning effort. GPT-Live: Full-Duplex Voice Explorer Use Cases With Examples Hands-free help: ask for cooking steps or directions without touching a screen. Language practice: hold a back-and-forth chat with gentle corrections. Live translation: full-duplex timing supports translating speech during a conversation. Research on the go: ask a hard question on your commute; GPT-5.5 searches in the background. Support workflows: multi-turn telecom-style tasks map to the τ³-Voice Telecom evaluation. Visual lookups: see weather, stocks, or sports as cards while you talk. A Conceptual Look at the Full-Duplex Loop The API is not available yet. So this is a runnable, illustrative simulation of the decision loop. It is a teaching model of the architecture, not the actual GPT-Live API. You can run it as plain Python to see the flow. Copy CodeCopiedUse a different Browser “””Illustrative simulation of the GPT-Live full-duplex decision loop. Teaching model of the described architecture, NOT the real API.””” import random random.seed(7) # reproducible output class BackgroundModel: # stands in for GPT-5.5 def run(self, query): return f”answer to ‘{query}'” class GPTLive: def __init__(self, background): self.background = background self.pending = None # a delegated task, if one is running def decide(self, user_speaking, needs_deep_work): # A real model makes this choice many times per second. if self.pending is not None: return “await_delegate” if needs_deep_work: return “delegate” if user_speaking: return random.choice([“listen”, “backchannel”]) return “speak” def step(self, frame): action = self.decide(frame[“user_speaking”], frame[“needs_deep_work”]) if action == “delegate”: self.pending = frame[“query”] # hand off, keep talking return ‘speak -> “one sec, still with you”‘ if action == “await_delegate”: result = self.background.run(self.pending) # background result self.pending = None return f’speak -> “{result}”‘ if action == “backchannel”: return ‘backchannel-> “mhmm” (while user talks)’ if action == “listen”: return “listen -> (quiet, attending)” return “speak -> (normal reply)” # A short scripted stream of audio frames the loop consumes in order. stream = [ {“user_speaking”: True, “needs_deep_work”: False, “query”: None}, {“user_speaking”: True, “needs_deep_work”: False,