Committee

Transformers use attention and Mixture-of-Experts to scale computation, but they still lack a native way to perform knowledge lookup. They re-compute the same local patterns again and again, which wastes depth and FLOPs. DeepSeek’s new Engram module targets exactly this gap by adding a conditional memory axis that works alongside MoE rather than replacing it. At a high level, Engram modernizes classic N gram embeddings and turns them into a scalable, O(1) lookup memory that plugs directly into the Transformer backbone. The result is a parametric memory that stores static patterns such as common phrases and entities, while the backbone focuses on harder reasoning and long range interactions. https://github.com/deepseek-ai/Engram/tree/main How Engram Fits Into A DeepSeek Transformer The proposed approach use the DeepSeek V3 tokenizer with a 128k vocabulary and pre-train on 262B tokens. The backbone is a 30 block Transformer with hidden size 2560. Each block uses Multi head Latent Attention with 32 heads and connects to feed forward networks through Manifold Constrained Hyper Connections with expansion rate 4. Optimization uses the Muon optimizer. Engram attaches to this backbone as a sparse embedding module. It is built from hashed N gram tables, with multi head hashing into prime sized buckets, a small depthwise convolution over the N gram context and a context aware gating scalar in the range 0 to 1 that controls how much of the retrieved embedding is injected into each branch. In the large scale models, Engram-27B and Engram-40B share the same Transformer backbone as MoE-27B. MoE-27B replaces the dense feed forward with DeepSeekMoE, using 72 routed experts and 2 shared experts. Engram-27B reduces routed experts from 72 to 55 and reallocates those parameters into a 5.7B Engram memory while keeping total parameters at 26.7B. The Engram module uses N equal to {2,3}, 8 Engram heads, dimension 1280 and is inserted at layers 2 and 15. Engram 40B increases the Engram memory to 18.5B parameters while keeping activated parameters fixed. https://github.com/deepseek-ai/Engram/tree/main Sparsity Allocation, A Second Scaling Knob Beside MoE The core design question is how to split the sparse parameter budget between routed experts and conditional memory. The research team formalize this as the Sparsity Allocation problem, with allocation ratio ρ defined as the fraction of inactive parameters assigned to MoE experts. A pure MoE model has ρ equal to 1. Reducing ρ reallocates parameters from experts into Engram slots. On mid scale 5.7B and 9.9B models, sweeping ρ gives a clear U shaped curve of validation loss versus allocation ratio. Engram models match the pure MoE baseline even when ρ drops to about 0.25, which corresponds to roughly half as many routed experts. The optimum appears when around 20 to 25 percent of the sparse budget is given to Engram. This optimum is stable across both compute regimes, which suggests a robust split between conditional computation and conditional memory under fixed sparsity. The research team also studied an infinite memory regime on a fixed 3B MoE backbone trained for 100B tokens. They scale the Engram table from roughly 2.58e5 to 1e7 slots. Validation loss follows an almost perfect power law in log space, meaning that more conditional memory keeps paying off without extra compute. Engram also outperforms OverEncoding, another N gram embedding method that averages into the vocabulary embedding, under the same memory budget. Large Scale Pre Training Results The main comparison involves four models trained on the same 262B token curriculum, with 3.8B activated parameters in all cases. These are Dense 4B with 4.1B total parameters, MoE 27B and Engram 27B at 26.7B total parameters, and Engram 40B at 39.5B total parameters. On The Pile test set, language modeling loss is 2.091 for MoE 27B, 1.960 for Engram 27B, 1.950 for the Engram 27B variant and 1.942 for Engram 40B. The Dense 4B Pile loss is not reported. Validation loss on the internal held out set drops from 1.768 for MoE 27B to 1.634 for Engram 27B and to 1.622 and 1.610 for the Engram variants. Across knowledge and reasoning benchmarks, Engram-27B consistently improves over MoE-27B. MMLU increases from 57.4 to 60.4, CMMLU from 57.9 to 61.9 and C-Eval from 58.0 to 62.7. ARC Challenge rises from 70.1 to 73.8, BBH from 50.9 to 55.9 and DROP F1 from 55.7 to 59.0. Code and math tasks also improve, for example HumanEval from 37.8 to 40.8 and GSM8K from 58.4 to 60.6. Engram 40B typically pushes these numbers further even though the authors note that it is likely under trained at 262B tokens because its training loss continues to diverge from the baselines near the end of pre training. https://github.com/deepseek-ai/Engram/tree/main Long Context Behavior And Mechanistic Effects After pre-training, the research team extend the context window using YaRN to 32768 tokens for 5000 steps, using 30B high quality long context tokens. They compare MoE-27B and Engram-27B at checkpoints corresponding to 41k, 46k and 50k pre training steps. On LongPPL and RULER at 32k context, Engram-27B matches or exceeds MoE-27B under three conditions. With about 82 percent of the pre training FLOPs, Engram-27B at 41k steps matches LongPPL while improving RULER accuracy, for example Multi Query NIAH 99.6 versus 73.0 and QA 44.0 versus 34.5. Under iso loss at 46k and iso FLOPs at 50k, Engram 27B improves both perplexity and all RULER categories including VT and QA. Mechanistic analysis uses LogitLens and Centered Kernel Alignment. Engram variants show lower layer wise KL divergence between intermediate logits and the final prediction, especially in early blocks, which means representations become prediction ready sooner. CKA similarity maps show that shallow Engram layers align best with much deeper MoE layers. For example, layer 5 in Engram-27B aligns with around layer 12 in the MoE baseline. Taken together, this supports the view that Engram effectively increases model depth by offloading static reconstruction to memory. Ablation studies on a 12 layer 3B MoE model with 0.56B activated parameters add a 1.6B Engram memory as a reference configuration, using N equal to {2,3} and inserting Engram at layers 2 and 6. Sweeping a

Google Research has expanded its Health AI Developer Foundations program (HAI-DEF) with the release of MedGemma-1.5. The model is released as open starting points for developers who want to build medical imaging, text and speech systems and then adapt them to local workflows and regulations. https://research.google/blog/next-generation-medical-image-interpretation-with-medgemma-15-and-medical-speech-to-text-with-medasr/ MedGemma 1.5, small multimodal model for real clinical data MedGemma is a family of medical generative models built on Gemma. The new release, MedGemma-1.5-4B, targets developers who need a compact model that can still handle real clinical data. The previous MedGemma-1-27B model remains available for more demanding text heavy use cases. MedGemma-1.5-4B is multimodal. It accepts text, two dimensional images, high dimensional volumes and whole slide pathology images. The model is part of the Health AI Developer Foundations program so it is intended as a base to fine tune, not a ready made diagnostic device. https://research.google/blog/next-generation-medical-image-interpretation-with-medgemma-15-and-medical-speech-to-text-with-medasr/ Support for high dimensional CT, MRI and pathology A major change in MedGemma-1.5 is support for high dimensional imaging. The model can process three dimensional CT and MRI volumes as sets of slices together with a natural language prompt. It can also process large histopathology slides by working over patches extracted from the slide. On internal benchmarks, MedGemma-1.5 improves disease related CT findings from 58% to 61% accuracy and MRI disease findings from 51% to 65% accuracy when averaged over findings. For histopathology, the ROUGE L score on single slide cases increases from 0.02 to 0.49. This matches the 0.498 ROUGE L score of the task specific PolyPath model. https://research.google/blog/next-generation-medical-image-interpretation-with-medgemma-15-and-medical-speech-to-text-with-medasr/ Imaging and report extraction benchmarks MedGemma-1.5 also improves several benchmarks that are closer to production workflows. On the Chest ImaGenome benchmark for anatomical localization in chest X rays, it improves intersection over union from 3% to 38%. On the MS-CXR-T benchmark for longitudinal chest X-ray comparison, macro-accuracy increases from 61% to 66%. Across internal single image benchmarks that cover chest radiography, dermatology, histopathology and ophthalmology, average accuracy goes from 59% to 62%t. These are simple single image tasks, useful as sanity checks during domain adaptation. MedGemma-1.5 also targets document extraction. On medical laboratory reports, the model improves macro F1 from 60% to 78% when extracting lab type, value and units. For developers this means less custom rule based parsing for semi structured PDF or text reports. Applications deployed on Google Cloud can now work directly with DICOM, which is the standard file format used in radiology. This removes the need for a custom preprocessor for many hospital systems. https://research.google/blog/next-generation-medical-image-interpretation-with-medgemma-15-and-medical-speech-to-text-with-medasr/ Medical text reasoning with MedQA and EHRQA MedGemma-1.5 is not only an imaging model. It also improves baseline performance on medical text tasks. On MedQA, a multiple choice benchmark for medical question answering, the 4B model improves accuracy from 64% to 69% relative to the previous MedGemma-1. On EHRQA, a text based electronic health record question answering benchmark, accuracy increases from 68% to 90%. These numbers matter if you plan to use MedGemma-1.5 as a backbone for tools such as chart summarization, guideline grounding or retrieval augmented generation over clinical notes. The 4B size keeps fine tuning and serving cost at a practical level. MedASR, a domain tuned speech recognition model Clinical workflows contain a large amount of dictated speech. MedASR is the new medical automated speech recognition model released together with MedGemma-1.5. MedASR uses a Conformer based architecture that is pre trained and fine tuned for clinical audio. It targets tasks such as chest X-ray dictation, radiology reports and general medical notes. The model is available through the same Health AI Developer Foundations channel on Vertex AI and on Hugging Face. In evaluations against Whisper-large-v3, a general ASR model, MedASR reduces word error rate for chest X-ray dictation from 12.5% to 5.2%. That corresponds to 58% fewer transcription errors. On a broader internal medical dictation benchmark, MedASR reaches 5.2% word error rate while Whisper-large-v3 has 28.2%, which corresponds to 82% fewer errors. Key Takeaways MedGemma-1.5-4B is a compact multimodal medical model that handles text, 2D images, 3D CT and MRI volumes and whole slide pathology, released as part of the Health AI Developer Foundations program for adaptation to local use cases. On imaging benchmarks, MedGemma-1.5 improves CT disease findings from 58% to 61%, MRI disease findings from 51% to 65%, and histopathology ROUGE-L from 0.02 to 0.49, matching the PolyPath model performance. For downstream clinical style tasks, MedGemma-1.5 increases Chest ImaGenome intersection over union from 3% to 38%, MS-CXR-T macro accuracy from 61%t to 66% and lab report extraction macro F1 from 60% to 78% while keeping model size at 4B parameters. MedGemma-1.5 also strengthens text reasoning, raising MedQA accuracy from 64% to 69% and EHRQA accuracy from 68% to 90%, which makes it suitable as a backbone for chart summarization and EHR question answering systems. MedASR, a Conformer based medical ASR model in the same program, cuts word error rate on chest X-ray dictation from 12.5% to 5.2% and on a broad medical dictation benchmark from 28.2% to 5.2% compared to Whisper-large-v3, providing a domain tuned speech front end for MedGemma centered workflows. Check out the Model Weights and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Google AI Releases MedGemma-1.5: The Latest Update to their Open Medical AI Models for Developers appeared first on MarkTechPost.