Uncategorized

Residual connections are one of the least questioned parts of modern Transformer design. In PreNorm architectures, each layer adds its output back into a running hidden state, which keeps optimization stable and allows deep models to train. Moonshot AI researchers argue that this standard mechanism also introduces a structural problem: all prior layer outputs are accumulated with fixed unit weights, which causes hidden-state magnitude to grow with depth and progressively weakens the contribution of any single layer. The research team proposes Attention Residuals (AttnRes) as a drop-in replacement for standard residual accumulation. Instead of forcing every layer to consume the same uniformly mixed residual stream, AttnRes lets each layer aggregate earlier representations using softmax attention over depth. The input to layer (l) is a weighted sum of the token embedding and previous layer outputs, where the weights are computed over prior depth positions rather than over sequence positions. The core idea is simple: if attention improved sequence modeling by replacing fixed recurrence over time, a similar idea can be applied to the depth dimension of a network. https://github.com/MoonshotAI/Attention-Residuals/tree/master?tab=readme-ov-file Why Standard Residuals Become a Bottleneck The research team identified three issues with standard residual accumulation. First, there is no selective access: all layers receive the same aggregated state even though attention layers and feed-forward or MoE layers may benefit from different mixtures of earlier information. Second, there is irreversible loss: once information is blended into a single residual stream, later layers cannot selectively recover specific earlier representations. Third, there is output growth: deeper layers tend to produce larger outputs to remain influential inside an ever-growing accumulated state, which can destabilize training. This is the research team’s main framing: standard residuals behave like a compressed recurrence over layers. AttnRes replaces that fixed recurrence with explicit attention over previous layer outputs. Full AttnRes: Attention Over All Previous Layers In Full AttnRes, each layer computes attention weights over all preceding depth sources. The default design does not use an input-conditioned query. Instead, each layer has a learned layer-specific pseudo-query vector wl ∈ Rd, while keys and values come from the token embedding and previous layer outputs after RMSNorm. The RMSNorm step is important because it prevents large-magnitude layer outputs from dominating the depth-wise attention weights. Full AttnRes is straightforward, but it increases cost. Per token, it requires O(L2 d) arithmetic and (O(Ld)) memory to store layer outputs. In standard training this memory largely overlaps with activations already needed for backpropagation, but under activation re-computation and pipeline parallelism the overhead becomes more significant because those earlier outputs must remain available and may need to be transmitted across stages. Block AttnRes: A Practical Variant for Large Models To make the method usable at scale, Moonshot AI research team introduces Block AttnRes. Instead of attending over every earlier layer output, the model partitions layers into N blocks. Within each block, outputs are accumulated into a single block representation, and attention is applied only over those block-level representations plus the token embedding. This reduces memory and communication overhead from O(Ld) to O(Nd). The research team describes cache-based pipeline communication and a two-phase computation strategy that make Block AttnRes practical in distributed training and inference. This results in less than 4% training overhead under pipeline parallelism, while the repository reports less than 2% inference latency overhead on typical workloads. Scaling Results The research team evaluates five model sizes and compares three variants at each size: a PreNorm baseline, Full AttnRes, and Block AttnRes with about eight blocks. All variants within each size group share the same hyperparameters chosen under the baseline, which the research team note makes the comparison conservative. The fitted scaling laws are reported as: Baseline: L = 1.891 x C-0.057Block AttnRes: L = 1.870 x C-0.058Full AttnRes: L = 1.865 x C-0.057 The practical implication is that AttnRes achieves lower validation loss across the tested compute range, and the Block AttnRes matches the loss of a baseline trained with about 1.25× more compute. Integration into Kimi Linear Moonshot AI also integrates AttnRes into Kimi Linear, its MoE architecture with 48B total parameters and 3B activated parameters, and pre-trains it on 1.4T tokens. According to the research paper, AttnRes mitigates PreNorm dilution by keeping output magnitudes more bounded across depth and distributing gradients more uniformly across layers. Another implementation detail is that all pseudo-query vectors are initialized to zero so the initial attention weights are uniform across source layers, effectively reducing AttnRes to equal-weight averaging at the start of training and avoiding early instability. On downstream evaluation, the reported gains are consistent across all listed tasks. It reports improvements from 73.5 to 74.6 on MMLU, 36.9 to 44.4 on GPQA-Diamond, 76.3 to 78.0 on BBH, 53.5 to 57.1 on Math, 59.1 to 62.2 on HumanEval, 72.0 to 73.9 on MBPP, 82.0 to 82.9 on CMMLU, and 79.6 to 82.5 on C-Eval. Key Takeaways Attention Residuals replaces fixed residual accumulation with softmax attention over previous layers. The default AttnRes design uses a learned layer-specific pseudo-query, not an input-conditioned query. Block AttnRes makes the method practical by reducing depth-wise memory and communication from O(Ld) to O(Nd). Moonshot research teamreports lower scaling loss than the PreNorm baseline, with Block AttnRes matching about 1.25× more baseline compute. In Kimi Linear, AttnRes improves results across reasoning, coding, and evaluation benchmarks with limited overhead. Check out Paper and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Moonshot AI Releases 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔 to Replace Fixed Residual Mixing with Depth-Wise Attention for Better Scaling in Transformers appeared first on MarkTechPost.

Why Document OCR Still Remains a Hard Engineering Problem? What does it take to make OCR useful for real documents instead of clean demo images? And can a compact multimodal model handle parsing, tables, formulas, and structured extraction without turning inference into a resource bonfire? That is the problem targeted by GLM-OCR, introduced by researchers from Zhipu AI and Tsinghua University. The research team presents GLM-OCR as a 0.9B-parameter compact multimodal model for document understanding. It combines a 0.4B CogViT visual encoder, a lightweight cross-modal connector, and a 0.5B GLM language decoder. The stated goal is to balance document recognition quality with lower latency and lower computational cost than larger multimodal systems. Traditional OCR systems are often good at plain text transcription, but they struggle when documents contain mixed layouts, tables, formulas, code blocks, seals, and structured fields. Recent multimodal large language models improve document understanding, but the research team argue that their size and standard autoregressive decoding make them expensive for edge deployment and large-scale production. GLM-OCR is positioned as a smaller system built for these deployment constraints rather than as a general-purpose vision-language model adapted to OCR as an afterthought. A Compact Architecture Built for OCR Workloads A key technical point for this research is the use of Multi-Token Prediction (MTP). Standard autoregressive decoding predicts one token at a time, which is not ideal for OCR-style tasks where outputs are often deterministic and locally structured. GLM-OCR instead predicts multiple tokens per step. The model is trained to predict 10 tokens per step and generates 5.2 tokens per decoding step on average at inference time, yielding about 50% throughput improvement. To keep memory overhead manageable, the implementation uses a parameter-sharing scheme across the draft models. Two-Stage Layout Parsing Instead of Flat Page Reading At the system level, GLM-OCR adopts a two-stage pipeline. The first stage uses PP-DocLayout-V3 for layout analysis, which detects structured regions on the page. The second stage performs parallel region-level recognition over those detected areas. This is important because the model is not simply reading a whole page left-to-right as a generic vision-language model might. It first breaks down the page into semantically meaningful regions, which improves efficiency and makes the system more robust on documents with complicated layouts. Document Parsing and KIE Use Different Output Paths The architecture also separates two related document tasks. For document parsing, the pipeline uses layout detection and region processing to produce structured outputs such as Markdown and JSON. For Key Information Extraction (KIE), the research team describes a different path: the full document image is fed to the model with a task prompt, and the model directly generates JSON containing the extracted fields. That distinction matters because GLM-OCR is not presented as a single monolithic page-to-text model. It is a structured generation system with different operating modes depending on the task. A Four-Stage Training Pipeline with Task-Specific Rewards The training recipe is split into 4 stages. Stage 1 trains the vision encoder on image-text pairs and grounding or retrieval data. Stage 2.1 performs multimodal pretraining on image-text, document parsing, grounding, and VQA data. Stage 2.2 adds the MTP objective. Stage 3 is supervised fine-tuning on OCR-specific tasks including text recognition, formula transcription, table structure recovery, and KIE. Stage 4 applies reinforcement learning using GRPO. The reward design is task-specific: Normalized Edit Distance for text recognition, CDM score for formula recognition, TEDS score for table recognition, and field-level F1 for KIE, along with structural penalties such as repetition penalties, malformed structure penalties, and JSON validation constraints. Benchmark Results Show Strong Performance, With Important Caveats On public benchmarks, GLM-OCR reports strong results across several document tasks. It scores 94.6 on OmniDocBench v1.5, 94.0 on OCRBench (Text), 96.5 on UniMERNet, 85.2 on PubTabNet, and 86.0 on TEDS_TEST. For KIE, it reports 93.7 on Nanonets-KIE and 86.1 on Handwritten-KIE. The research team notes that results for Gemini-3-Pro and GPT-5.2-2025-12-11 are shown only for reference and are excluded from the best-score ranking, which is an important detail when interpreting claims about model leadership. https://arxiv.org/pdf/2603.10910 The benchmark story is strong, but it needs careful phrasing. GLM-OCR achieves the highest reported scores among the evaluated non-reference models on OmniDocBench v1.5, OCRBench (Text), UniMERNet, and TEDS_TEST. On PubTabNet, however, it does not lead overall; MinerU 2.5 reports 88.4 versus GLM-OCR’s 85.2. For KIE, GLM-OCR outperforms the listed open-source competitors in the above table, but Gemini-3-Pro scores higher on both Nanonets-KIE and Handwritten-KIE in the reference column. So the reserach team supports a strong competitive claim, but not a blanket ‘best at everything’ claim. Deployment Details The research team state that GLM-OCR supports vLLM, SGLang, and Ollama, and can be fine-tuned through LLaMA-Factory. They also report throughput of 0.67 images/s and 1.86 PDF pages/s under their evaluation setup. In addition, they describe a MaaS API priced at 0.2 RMB per million tokens, with example cost estimates for scanned images and simple-layout PDFs. These details suggest that GLM-OCR is being framed as both a research model and a deployable system. Key Takeaways GLM-OCR is a compact 0.9B multimodal OCR model built with a 0.4B CogViT encoder and 0.5B GLM decoder. It uses Multi-Token Prediction (MTP) to improve decoding efficiency, reaching 5.2 tokens per step on average and about 50% higher throughput. The model uses a two-stage pipeline: PP-DocLayout-V3 handles layout analysis, then GLM-OCR performs parallel region-level recognition. It supports both document parsing and KIE: parsing outputs Markdown/JSON, while KIE directly generates JSON from the full document image. Benchmark results are strong but not universal wins: GLM-OCR leads several reported non-reference benchmarks, but MinerU 2.5 is higher on PubTabNet, and Gemini-3-Pro is higher on the reference-only KIE scores. Check out Paper, Repo and Model Page. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Zhipu AI Introduces GLM-OCR: A 0.9B Multimodal OCR Model for Document Parsing and Key Information Extraction (KIE) appeared first on MarkTechPost.

For decades, manufacturers have pursued automation to drive efficiency, reduce costs, and stabilize operations. That approach delivered meaningful gains, but it is no longer enough. Today’s manufacturing leaders face a different challenge: how to grow amid labor constraints, rising complexity, and increasing pressure to innovate faster without sacrificing safety, quality, or trust. The next phase of transformation will not be defined by isolated AI tools or individual robots, but by intelligence that can operate reliably in the physical world. This is where physical AI—intelligence that can sense, reason, and act in the real world—marks a decisive shift. And it is why Microsoft and NVIDIA are working together to help manufacturers move from experimentation to production at industrial scale. The industrial frontier: Intelligence and trust, not just automation Most early AI adoption focused on narrow optimization: automating tasks, improving utilization, and cutting costs. While valuable, that phase often created new friction, including skills gaps, governance concerns, and uncertainty about long‑term impact. Furthermore, the use cases were plentiful but not as strategic. The industrial frontier represents a different approach. Rather than asking how much work machines can replace, frontier manufacturers ask how AI can expand human capability, accelerate innovation, and unlock new forms of value while remaining trustworthy and controllable. Across industries, companies that successfully move into this frontier phase share two non‑negotiables: Intelligence: AI systems must understand how the business actually handles its data, workflows, and institutional knowledge. Trust: As AI begins to act in high‑stakes environments, organizations must retain security, governance, and observability at every layer. Without intelligence, AI becomes generic. Without trust, adoption stalls. Why manufacturing is the proving ground for physical AI Manufacturing is uniquely positioned at the center of this shift. AI is no longer confined to planning or analytics. It is moving into physical execution: coordinating machines, adapting to real‑world variability, and working alongside people on the factory floor. Robotics, autonomous systems, and AI agents must now perceive, reason, and act in dynamic environments. This transition exposes a critical gap. Traditional automation excels at repetition but struggles with adaptability. Human workers bring judgment and context but are constrained by scale. Physical AI closes that gap by enabling human‑led, AI‑operated systems, where people set intent and intelligent systems execute, learn, and improve over time. Humans are essential for scaled success. Microsoft and NVIDIA: Accelerating physical AI at scale Physical AI cannot be delivered through point solutions. It requires agentic-driven, enterprise-grade development, deployment, and operations toolchains and workflows that connect simulation, data, AI models, robotics, and governance into a coherent system. NVIDIA is building the AI infrastructure that makes physical AI possible, including accelerated computing, open models, simulation libraries, and robotics frameworks and blueprints that enable the ecosystem to build autonomous robotics systems that can perceive, reason, plan, and take action in the physical world. Microsoft complements this with a cloud and data platform designed to operate physical AI securely, at scale, and across the enterprise. Together, Microsoft and NVIDIA are enabling manufacturers to move beyond pilots toward production‑ready physical AI systems that can be developed, tested, deployed, and continuously improved across heterogeneous environments spanning the product lifecycle, factory operations, and supply chain. From intelligence to action: Human-agent teams in the factory At the industrial frontier, AI is not a standalone system, but a digital teammate. When AI agents are grounded in the proper operational data, embedded in human workflows, and governed end to end, they can assist with tasks such as: Optimizing production lines in real time Coordinating maintenance and quality decisions Adapting operations to supply or demand disruptions Accelerating engineering and product lifecycle decisions For example, manufacturers are beginning to use simulation‑grounded AI agents to evaluate production changes virtually before deploying them on the factory floor, reducing risk while accelerating decision‑making. Crucially, frontier manufacturers design these systems so humans remain in control. AI executes, monitors, and recommends, while people provide intent, oversight, and judgment. This balance allows organizations to move faster without losing confidence or control. The role of trust in scaling physical AI As physical AI systems scale, trust becomes the limiting factor. Manufacturers must ensure that AI systems are secure, observable, and operating within policy, especially when they influence safety‑critical or mission‑critical processes. Governance cannot be an afterthought; It must be engineered into the platform itself. This is why frontier manufacturers treat trust as a first‑class requirement, pairing innovation with visibility, compliance, and accountability. Only then can physical AI move from promising demonstrations to enterprise‑wide deployment. Why this moment matters—and what’s next The convergence of AI agents, robotics, simulation, and real‑time data marks an inflection point for manufacturing. What was once experimental is becoming operational. What was once siloed is becoming connected. At NVIDIA GTC 2026, Microsoft and NVIDIA will demonstrate how this collaboration supports physical AI systems that manufacturers can deploy today and scale responsibly tomorrow. From simulation‑driven development to real‑world execution, the focus is on helping manufacturers cross the industrial frontier with confidence. For manufacturing leaders, the question is no longer whether physical AI will reshape operations, but how quickly they can adopt it responsibly, at scale, and with trust built in from the start. Discover more with Microsoft at NVIDIA GTC 2026. This content was produced by Microsoft. It was not written by MIT Technology Review’s editorial staff.

Google AI Research team recently released Groundsource, a new methodology that uses Gemini model to extract structured historical data from unstructured public news reports. The project addresses the lack of historical data for rapid-onset natural disasters. Its first output is an open-source dataset containing 2.6 million historical urban flash flood events across more than 150 countries. The Hydro-Meteorological Data Gap Machine learning models for early warning systems (EWS) require extensive historical baselines for training and validation. However, hydro-meteorological hazards like flash floods lack standardized, global observation networks. The Impact of Flash Floods: According to the World Meteorological Organization (WMO), flash floods cause approximately 85% of flood-related fatalities, resulting in over 5,000 deaths annually. Limitations of Existing Data: Satellite-based databases, such as the Global Flood Database (GFD) and the Dartmouth Flood Observatory (DFO), are limited by cloud cover, satellite revisit times, and a bias toward long-lasting events. Scale of the Deficit: The Global Disaster Alert and Coordination System (GDACS) provides an inventory of roughly 10,000 high-impact events. This volume is insufficient for training global-scale predictive models. The Groundsource Methodology To build a larger training corpus, Google’s research team developed a pipeline that processes decades of localized news reports to synthesize a historical baseline. Semantic Parsing with Gemini: The LLM is deployed for entity extraction. It processes unstructured, multilingual text to identify specific hazard events, classify their severity, and filter out irrelevant noise. Geospatial Mapping: The extracted text descriptions of flood locations are integrated with Google Maps APIs to assign precise geographic coordinates and polygonal boundaries to each event. This pipeline successfully converts qualitative journalistic reporting into a highly structured, machine-readable dataset. https://research.google/blog/introducing-groundsource-turning-news-reports-into-data-with-gemini/ Application: Flash Flood Forecasting Historically, Google’s Flood Forecasting Initiative focused on riverine floods, which develop slowly and are easier to track. Flash floods require distinct predictive approaches due to their rapid onset. Using the 2.6-million-record Groundsource dataset, the research team trained a new AI model to predict urban flash flood risks up to 24 hours in advance. Empirical studies note that even a 12-hour lead time can reduce flash flood damage by 60%. These forecasts are now live on Google’s Flood Hub platform. The underlying dataset has been open-sourced to allow the broader data science community to train their own localized predictive models. Key Takeaways LLM-Driven Data Pipeline: Groundsource uses the Gemini model for semantic parsing to extract structured historical disaster data from unstructured, multilingual public news reports. Massive Dataset Generation: The pipeline successfully produced an open-source dataset containing 2.6 million historical urban flash flood records across more than 150 countries. Overcoming Sensor Limitations: This NLP-based approach addresses the historical ‘data desert,’ bypassing the physical constraints of remote sensing (such as cloud cover or satellite revisit times) and the limited volume of existing traditional databases like GDACS. Geospatial Integration: Extracted natural language descriptions of hazard locations are integrated with Google Maps APIs to assign precise geographic coordinates and polygonal boundaries to each event. Predictive Model Deployment: The resulting dataset was utilized to train a new AI model capable of predicting urban flash flood risks up to 24 hours in advance, which is now actively deployed on Google’s Flood Hub platform. Check out Dataset, Pre-Print Paper and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Google AI Introduces ‘Groundsource’: A New Methodology that Uses Gemini Model to Transform Unstructured Global News into Actionable, Historical Data appeared first on MarkTechPost.