AI

In this article, you will learn how static, dynamic, and continuous batching work in LLM inference, and why the differences between them matter at production…

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Trump’s AI protectionism has come for robotics   —James O’Donnell  Humanoid robots usually elicit more cringe than awe: They stumble, kick children, and despite advances are still worse at using their hands than my toddler. It’s a nascent industry, and such robots are more commonly seen in viral videos than real workplaces or homes.  It was a surprise, then, when last week the Federal Trade Commission issued a sweeping ban on foreign imports of advanced robots, including humanoids, quadrupeds, and wheeled robots.  The decision should be understood not as another chapter in the old China trade playbook, but as evidence that the Trump administration is expanding its protection of the AI industry beyond today’s leading labs. It is now willing to step in on behalf of an emerging robotics sector that is still barely finding its footing. Read our analysis to understand the ban’s potential impact. This story is from The Algorithm, our weekly newsletter all about the latest goings-on in the world of AI. Sign up to receive it in your inbox every Monday. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 ICE collected nearly one million people’s DNA last year Most of them have never been convicted of any crime. (Wired $) 2 China is poised to win the technologies of the futureIt’s already a high-tech hardware powerhouse, but it increasingly has its sights set on software too. (New Yorker $)+ US tech and political leaders are freaking out, but few seem to agree how to respond. (Vox $) + What’s next for Chinese open-source AI. (MIT Technology Review) 3 AI “tokenomics” is a burgeoning new field Businesses are pouring a lot of money into AI. Now they want to see what they’re getting in return. (NYT $)+ Why it’s proving so hard to make AI pay. (BBC)+ The US economy is becoming more and more reliant on the AI boom. (WSJ $) 4 Eli Lilly is letting people apply to try an unapproved obesity drugRetatrutide is still in clinical trials, but certain patients can gain early access. (STAT $) + Montana’s plan to become an experimental medical hub just pushed forward. (MIT Technology Review) 5 Inside the one US town that wants a data center Jay in Maine is a reminder that politics is all about the local. (The Atlantic $)+ How data centers broke US politics. (Wired $) 6 Flock license plate readers can have a shockingly high error rateIn one California town, Flock misread license plates in 71% of the alerts it sent to police. (BI $)+ A leaked guide shows how Flock teaches cops to promote its tech. (404 Media $) 7 A drone explosion on a beach in Russia killed seven people It seems to have been caused by Russian forces shooting down a Ukrainian drone. (CNN)+ A US company won a $100 million deal to give Ukrainian drones an AI upgrade. (Ars Technica)+ Europe’s drone-filled vision for the future of war. (MIT Technology Review) 8 How car headlights became so brightLots of modern cars still blind other drivers on the road. That could change soon though, thanks to new tech. (Ars Technica) 9 A $2 million crime novel deal collapsed over AI use concernsAnd the agent and the writer involved have, erm…rather differing accounts of what happened. (Guardian) 10 AI matchmaking services are on the rise Online daters hope it might prove better at finding them love than fruitless swiping. (WSJ $) Quote of the day “Just a little too pleasant to be human.” —North Carolina resident Kristen Charpentier tells Wired how she could tell she was talking to AI when ordering at a Dairy Queen drive-thru. One More Thing GETTY IMAGES How to have a child in the digital age  Before journalist and culture critic Amanda Hess even got pregnant with her first child, in 2020, the internet knew she was trying. She saw pregnancy ads way before a doctor.    Hess’s experience is pretty typical these days, but still raises some big questions. How do we retain control over our bodies when corporations have access to our most personal information? What happens when people stop relying on friends and family for advice on having a kid and instead go online?  Read our interview with Hess to learn what she has to say.  —Alison Arieff We can still have nice things A place for comfort, fun, and distraction to brighten up your day. (Got any ideas? Drop me a line.) + Brighten your day with this eight-minute video of the world’s cutest baby animals.+ Intertapes has assembled a remarkable treasure trove of cassette tapes found around the world.+ Armchair basketball GMs can finally put their talents to the test in this game that lets you draft an elite NBA team.+ Here’s a hilarious look at the lost recordings of Robin Williams voicing the Genie in Disney’s “Aladdin.”

This chapter is divided into eight parts; they are: • Metrics for LLM Inference • Measuring a Single Request • Warmup and Synchronization • Measuring GPU Work with CUDA Events • Measuring Memory Usage • Measuring Concurrent Requests • Multiple GPUs and Multiple Machines • Cost per Token The most common inference metrics are: • Latency:  How long a request takes from start to finish.

Cursor Research has open-sourced Mixture-of-Kittens (MoK), the mixture-of-experts training megakernel behind its Composer models. MoK fuses every MoE communication and computation step into a single deterministic kernel. Cursor team reports up to 2.37x higher throughput than the strongest public baseline. It already powers Composer training across tens of thousands of GPUs. Is it deployable Yes, but the hardware floor is high. MoK is on GitHub under Apache-2.0. It requires NVIDIA Blackwell SM100 or SM103 GPUs, which means GB200 NVL72 or GB300 NVL72 racks. It also needs Python 3.12+, PyTorch 2.10+, and CUDA toolkit 13.0+. Inter-GPU buffers rely on PyTorch symmetric memory. That limits realistic adopters to organizations that own or rent NVL72 capacity. Frontier labs, funded model startups, GPU neoclouds, and national computing centers fit. Single-node teams and 8-GPU shops do not. Applications are narrow but high-value. They include pretraining and post-training of DeepSeek-V3-style MoE models. Determinism also makes it useful for on-policy RL post-training and internal ablations. Relevant industries are AI model development, cloud GPU infrastructure, code-generation tooling, and quantitative research. MoE layer as the bottleneck Cursor’s earlier work covered the compute side. The research team wrote its own MXFP8 and NVFP4 training kernels and a ‘warp decode’ path for MoE inference. Those assumed inter-GPU communication was handled separately. In production, communication became the limiting factor. The MoE layer can consume more than half of end-to-end training time. Moving to GB300 NVL72s changed the problem again. A rack is 72 GPUs inside one NVLink domain, which allows fine-grained overlap. But the integrated Grace CPUs are slow relative to the GPUs. CPU-GPU synchronization therefore has to be minimized aggressively. Three design decisions that matter Communication direction is chosen per operation: Existing approaches such as DeepEP lean on push-based transfers. Cursor’s microbenchmarks show push moves fewer total bytes in one direction. That leaves the reverse NVLink lane mostly idle. Pull-based dispatch delivers up to 29% higher NVLink bandwidth utilization under expert imbalance. It also eliminates cross-GPU completion signals. Push dispatch signalling measured 103 µs against 18 µs for pull, roughly 5.8x. MoK therefore uses pull-based forward dispatch and push-based forward combine. The backward pass mirrors this with pull reverse-combine and push reverse-dispatch. One schedule table serves all four, costing under 3% of MoE runtime. Overlap granularity sits between the extremes: Comet is fine-grained; DeepEP is coarse-grained. Cursor team argues the optimum is in the middle and workload-dependent. The heuristic targets at least two full SM waves per expert-grouped GEMM. For Kimi 2.5 shapes, the base model for Composer 2.5, the floor is 2,368 tokens. Measured latency matches that estimate closely. A ring token buffer removes the CPU from the loop: The alternatives are dropping tokens or asking the CPU to size buffers. MoK instead cycles a fixed ring buffer of a few hundred megabytes. It does so at minibatch granularity, interleaving dispatch and combine at macrobatch boundaries. The ring is walked in reverse to minimize forward activation replay during backward. MoK is built as a megakernel and is fully deterministic. It supports BF16 and MXFP8 precision modes. Scheduling runs through Blackwell’s Cluster Launch Control, so inter-rack RDMA does not serialize behind it. Router weight gradients use a SonicMoE-style calculation fused into the SwiGLU backward. https://cursor.com/blog/mixture-of-kittens Results Layer benchmarks ran in a single NVL72 rack at EP degree 64. Each GPU held 2,048 tokens before routing. Baselines were NCCL+PyTorch, DeepEP+PyTorch, DeepEP+TransformerEngine, and HybridEP+Megatron. Shapes covered Kimi K2.7 Code, GLM-5.2, Qwen3.5-397B-A17B, and DeepSeek-V4-Pro. Against the fastest baseline, MoK is up to 2.37x faster for MXFP8 forward. The other figures are 1.78x MXFP8 backward, 1.92x BF16 forward, and 1.58x BF16 backward. End-to-end testing used 512 GPUs across several GB300 NVL72 racks. Tokens per second per GPU rose from 760.9 to 1,070.2, a 1.41x gain. Key Takeaways MoK fuses all MoE communication and computation into one deterministic megakernel for NVL72 racks. Pull dispatch plus push combine cuts signalling from 103 µs to 18 µs. A ring token buffer drops zero tokens and removes CPU-GPU synchronization entirely. Up to 2.37x over the fastest public baseline; 1.41x end-to-end on 512 GPUs. Apache-2.0, but it demands Blackwell SM100/SM103, CUDA 13.0+, and PyTorch 2.10+. Check out the GitHub Repo and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Cursor Open-Sources Mixture-of-Kittens (MoK): A Deterministic MoE Training Megakernel for GB300 NVL72 Racks appeared first on MarkTechPost.

In this article, you will learn how static, dynamic, and continuous batching work in LLM inference, and why the differences between them matter at production…

Alibaba’s Qwen team has made Qwen3.8-Max broadly available and confirmed that its open weights ship next week. A second checkpoint, Qwen3.8-27B, is also going open-weights. Qwen3.8-Max is a 2.4-trillion-parameter mixture-of-experts model. It accepts text, image and video as input and returns text. Is it deployable Yes, but the deployable surface depends on which artifact you are applying. The hosted API is deployable today by any company size. It is OpenAI- and DashScope-compatible, so integration is a base-URL and model-ID change. The open weights are a different matter. At 2.4T total parameters, the checkpoint is a multi-node datacenter artifact. Alibaba has not disclosed the activated-parameter count. Serving cost therefore cannot yet be modeled. Qwen3.8-27B is the checkpoint that fits ordinary on-premise GPU hardware. The published feature set maps cleanly onto four industries. Those are software engineering, legal and financial document review, media and e-commerce operations, and design. Applications include repository-scale coding agents and long-document knowledge bases. Long-video indexing, structured data extraction and multi-step research assistants also fit. Interactive explainer What is Technically Available The model page lists a 1M-token context window. Maximum input is 991K tokens, dropping to 983K when thinking is enabled. Maximum output is 131K tokens in both modes, and the maximum reasoning budget is 262K tokens. Rate limits are 2M tokens per minute and 15K requests per minute. Pricing is $2.00 per 1M input tokens and $6.00 per 1M output tokens. Implicit cache reads cost $0.25 per 1M tokens. Explicit cache creation is $2.50 and explicit cache reads are $0.17 per 1M tokens. Cached input is eight times cheaper than fresh input. Prefix stability therefore drives cost more than prompt length does. Supported capabilities include function calling, structured outputs, batches, prefix completion and fine-tuning. Five built-in tools ship on the Responses API: code_interpreter, web_search, web_extractor, t2i_search and i2i_search. https://qwen.ai/blog?id=qwen3.8 Performance Alibaba published a full benchmark table with this release. Qwen3.8-Max scores 86.6 on Terminal-Bench 2.1, ahead of Claude Opus 4.8 and Claude Fable 5 at 84.6, behind GPT-5.6 Sol (max) at 88.8. It reports 67.7 on SWE-bench Pro against Fable 5’s 80.0, and 73.5 on FrontierSWE against Fable 5’s 88.8. It leads PaperBench at 93.0 and IFBench at 82.8. GPQA Diamond lands at 92.6, up marginally from Qwen3.7-Max’s 92.4. The clearest gains are multimodal and agentic, not reasoning. It tops most vision rows, including OSWorld-Verified 86.1, Parametric CAD Bench 91.5, and OmniDocBench 1.5 at 92.1. Against its own predecessor the jump is large: DeepSWE 1.1 moves from 21.6 to 56.6, FrontierSWE from 40.7 to 73.5, JobBench from 31.3 to 53.4. Two caveats belong in any honest read. The multimodal table benchmarks against Qwen3.7-Plus, not Qwen3.7-Max, which flatters the generational delta. And Alibaba’s own RL scaling curve peaks at 0.725 near 4,000 training environments, then declines to 0.719 and 0.689. Key Takeaways Qwen3.8-Max is a 2.4T-parameter MoE model with 1M context, now generally available. Pricing is $2 input, $6 output and $0.25 cached input per 1M tokens. Open weights for Qwen3.8-Max and Qwen3.8-27B are promised next week. No benchmark table, license, or activated-parameter count has been published. The 27B checkpoint, not the flagship, is the realistic on-premise deployment path. Check out the Technical details, API and Qwen Studio. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Alibaba Qwen Releases Qwen3.8-Max: A 2.4 Trillion Parameter MoE Model and the Most Capable One in the Qwen Family to Date appeared first on MarkTechPost.

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Here’s why AI agents lie and cheat to reach their goals When two OpenAI models hacked into Hugging Face last month, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question.   According to OpenAI, the models decided to solve a cybersecurity exercise by hacking out of the environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored. The incident has attracted intense attention over the past couple of weeks. It’s a dramatic illustration of just how good AI models have gotten at hacking. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat.  Read our story explaining why AI engages in this sort of behavior—known as ”reward hacking.” —Grace Huckins This story is from our ‘Explains’ series, where our writers untangle the complex, messy world of technology to help you understand what’s coming next. Read more from the collection. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 It looks like Iran is conducting cyberattacks on US water systemsThat’s according to preliminary investigations on hacks in at least seven states. (NYT $)+ Will this be a wake-up call? (Forbes) 2 Google briefly made it easy to fake satellite imagesLiterally the last thing the world needs right now. (NPR)+ AI companies keep moving fast and breaking things. (The Atlantic $)+ Apple is struggling to keep pace with incoming AI-assisted software bug reports. (FT $) 3 Why wildfires have got so bad in Europe this summerIt’s a mix of climate change, land abandonment, and outdated firefighting tactics. (New Yorker $)+ How Europe can become more fire-resilient. (New Scientist $)+ How much wildfire prevention is too much? (MIT Technology Review) 4 Law enforcement officers are using license-plate cameras for stalkingThere are at least 50 examples of officers being charged with or accused of misusing them. (WP $)+ Inside Chicago’s surveillance panopticon. (MIT Technology Review) 5 China may impose more controls on its homegrown AI modelsThey’re winning influence overseas—but create new security and political risks. (NYT $)+ Silicon Valley is deeply divided over how to respond. (Rest of World)+ China’s AI models have Trump’s AI world at war with itself. (MIT Technology Review) 6 The vast majority of Australian teens are still on social mediaA lack of effective age checks means the country’s under-16s ban simply isn’t enforceable. (Reuters $) 7 Is it possible to make smart glasses that aren’t creepy? It doesn’t really look like it right now! (Wired $) 8 The US ban on robot vacuum cleaners isn’t workable It’s going to leave Americans with less choice and way higher prices. (The Verge $) 9 YouTube just banned a bunch of ASMR artistsThey say they’re being unfairly caught up in rules against “sexually gratifying” content. (404 Media) 10 Why Pokémon is still popular all over the worldIt seems to have a rare ability to both cheer us up, and bring us together. (The Guardian) Quote of the day “Trump knows exactly who is responsible for this attack, and knows that other states were hit too. This is what modern warfare looks like, and it further illustrates there’s no plan to win a war with Iran.” —Governor Tim Walz responds to Trump blaming Minnesota for cyberattacks on its own water systems, the Washington Post reports. One More Thing RANDY MONTOYA/SANDIA NATIONAL LABORATORY Meet the researchers testing the “Armageddon” approach to asteroid defense  One day a big asteroid will find itself on a collision course with Earth. If we are lucky, it’d land in the middle of the vast ocean, creating a good-size but innocuous tsunami, or in an uninhabited patch of desert. But if it has a city in its crosshairs, one of the worst natural disasters in modern times would unfold. Homes dozens of miles away would fold like cardboard. Millions of people would die. Fortunately for all 8 billion of us, planetary defense—the science of preventing asteroid impacts—is a highly active field of research.  We already know that we could ram a rock with an uncrewed spacecraft to push it away from Earth. But if that’s not enough, we could need another method, one that is notoriously difficult to test in real life: a nuclear explosion.  Read our story about the scientists who, despite the odds, are trying to do exactly that.  —Robin George Andrews We can still have nice things A place for comfort, fun, and distraction to brighten up your day. (Got any ideas? Drop me a line.) + There’s a quiet power to this photo of 118 swimmers. + Matt Damon’s biceps in the Odyssey actually belong to a stunt woman called Devyn Dalton. + A newly retired doctor and his filmmaker daughter drove 600 miles with a baby cow in the back seat to save the animal’s life.+ 400 years after a collector cut apart Leonardo da Vinci’s notebooks, a digital archive has reunited them.

This chapter is divided into nine parts; they are: • Reading Logits from a Model • Greedy Decoding • Temperature Sampling • Top-$k$ Sampling • Nucleus Sampling • Repetition Penalties • Beam Search • Stop Conditions • Structured Output Constraints The model returns a vector of logits for every position in the input sequence.