Uncategorized

Audio AI has had a breakout year. Automatic speech recognition has gotten dramatically better with models like OpenAI’s Whisper variants, NVIDIA’s Parakeet, and Mistral’s Voxtral. Audio understanding stepped forward with models like NVIDIA’s Audio Flamingo 3. Dialogue-grade text-to-speech arrived via Nari Labs’ Dia-1.6B. And Meta shipped the Perception Encoder Audiovisual (PE-AV), a multimodal encoder capable of learning a shared embedding space across audio, video, and text. The frontier has never moved faster. The catch? The practical knowledge required to actually work with these models — how to fine-tune them, adapt them to new languages, or run efficient inference — is scattered across GitHub issues, research blogs, and private notebooks that never see the light of day. If you are an ML engineer who just wants to fine-tune Whisper on a new domain or run zero-shot video classification with PE-AV, you are often starting from scratch. That is the gap smol-audio is designed to close. What is smol-audio ? Released under the Apache-2.0 license by the Deep-unlearning team, smol-audio is a flat repository of self-contained Jupyter notebooks, each focused on a single practical audio AI task. Every notebook is designed to be opened directly in Google Colab, requires no local GPU setup, and is built entirely on the Hugging Face ecosystem — specifically transformers, datasets, peft, and accelerate. Most recipes fit within a 16 GB Colab runtime, which means a free or standard Colab tier is sufficient for the majority of tasks. The “flat repo” design is a deliberate choice. Rather than wrapping recipes inside a framework or hiding complexity behind convenience functions, smol-audio exposes every step. You can read the training loop, understand the data pipeline, and modify the configuration without reverse-engineering a library. For early-career engineers, that transparency is genuinely educational. ASR Fine-Tuning: Whisper, Parakeet, Voxtral, and Granite Speech The largest category in the repo today covers ASR fine-tuning across four distinct model families. Each requires meaningfully different handling. The Whisper notebook covers fine-tuning using transformers and datasets, making it straightforward to adapt the encoder-decoder architecture to a custom language or narrow domain. Whisper uses a sequence-to-sequence approach, generating transcripts token by token — familiar territory for anyone who has worked with language models. NVIDIA’s Parakeet uses a CTC (Connectionist Temporal Classification) architecture rather than a sequence-to-sequence setup. CTC is faster and lighter for inference but requires alignment between audio frames and output tokens rather than autoregressive decoding. The smol-audio notebook covers both full fine-tuning and LoRA (Low-Rank Adaptation) for Parakeet, which is important because full fine-tuning large CTC models can be memory-intensive. Mistral’s Voxtral is architecturally distinct from both Whisper and Parakeet. Rather than a traditional ASR encoder-decoder, Voxtral is built on a large language model backbone — Ministral 3B for Voxtral Mini and Mistral Small 3.1 24B for Voxtral Small — making it an LLM-based speech understanding model. The smol-audio notebook handles fine-tuning for ASR with prompt masking, supporting both full fine-tuning and LoRA. Prompt masking is important here precisely because of this LLM architecture: when a model accepts text prompts alongside audio input, you typically do not want to compute loss on the prompt tokens themselves — only on the generated transcription. Getting this wrong leads to degraded training dynamics, so having a working reference implementation saves significant debugging time. IBM’s Granite Speech gets its own notebook focused on Italian ASR using the YODAS-Granary dataset. This is a useful example beyond just the model: it demonstrates domain- and language-specific fine-tuning on a real multilingual speech corpus, a common production scenario. Audio Understanding with NVIDIA’s Audio Flamingo 3 Audio Flamingo 3, developed by NVIDIA, is a Large Audio Language Model (LALM) for reasoning and understanding across speech, sound, and music. The smol-audio notebook fine-tunes it specifically for the audio captioning task — generating a natural language description of an audio clip, which is useful for accessibility tooling, content indexing, and retrieval systems. The notebook covers both full fine-tuning and LoRA-based fine-tuning, giving practitioners the choice between maximum performance and memory efficiency. LoRA, for those newer to parameter-efficient fine-tuning, works by freezing the original model weights and injecting small trainable rank-decomposition matrices into specific layers. For large multimodal models like Audio Flamingo 3, LoRA can reduce GPU memory requirements by an order of magnitude compared to full fine-tuning, enabling iteration on commodity hardware. Dialogue TTS with Dia-1.6B The Dia-1.6B notebook covers dialogue-style text-to-speech, where the goal is not just synthesizing a single speaker but generating natural conversational exchanges. Dia is a 1.6-billion-parameter TTS model by Nari Labs capable of producing multi-speaker dialogue, making it relevant for anyone building voice agents, podcast generation tools, or conversational interfaces. Multimodal Inference with Meta’s PE-AV Perhaps the most forward-looking notebook in the current release covers inference with Meta’s Perception Encoder Audiovisual (PE-AV). PE-AV is a multimodal encoder that learns a single shared embedding space across audio, video, and text — enabling zero-shot video classification without any task-specific fine-tuning, and audiotext retrieval on benchmarks like AudioCaps. Because all three modalities map into the same embedding space, cross-modal queries such as retrieving an audio clip from a text description work via simple dot-product similarity. The notebook demonstrates how to run these inference pipelines directly, which is valuable because multimodal models with joint audio-visual-text encoders are architecturally more complex than single-modality models and typically require careful preprocessing of multiple input modalities. Check out the Repo here. Also, feel free to follow us on Twitter and don’t forget to join our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post smol-audio: A Colab-Friendly Notebook Collection for Fine-Tuning Whisper, Parakeet, Voxtral, Granite Speech, and Audio Flamingo 3 appeared first on MarkTechPost.

Today, nuclear energy enjoys a rare moment of support across the political spectrum in the US. Interest from tech companies that are scrambling to meet demand for massive data centers has sparked a resurgence of money and attention in the industry. That newfound interest is exactly why it’s time to talk about an old problem: nuclear waste.  In the US alone, nuclear reactors produce about 2,000 metric tons of high-level waste each year. And there’s nowhere to put it. Though newly popular, the nuclear program in the US is nothing new. The US hosts more reactors and production capacity than any other country in the world. And yet nearly seven decades after the first permanent nuclear facility in the US went online, there’s still not a long-term solution for nuclear waste.  Used fuel is largely stored onsite at operating and shut-down reactors, in pools and casks made of steel and concrete. Experts generally agree that these methods are safe, but they’re not designed to be permanent. The leading strategy around the world for long-term storage of this high-level radioactive waste is to house it in a deep geological repository—dig a hole, put radioactive material down there, and fill it up with concrete. These holes, hundreds of meters underground, are designed to be a permanent home. There aren’t any operating geological repositories for spent fuel yet, but some countries are well on their way. Finland is the furthest along; as of 2026, the country is testing its facility. Final approvals are expected soon, and operations could start later this year. Some other countries aren’t far behind. France is home to over 50 nuclear reactors, and its grid gets more of its power from nuclear than any other. The country also has the world’s most established program for reprocessing spent fuel. The process separates out the plutonium and uranium to create a type of fuel known as mixed oxide (MOX) fuel. But reprocessing isn’t a perfect recycling loop, so the leftovers from this process still need somewhere to go. The country currently stores waste onsite at the La Hague reprocessing plant, but it plans to build a repository. Initial approvals could come later this decade, and pilot operations could start up by 2035. Technically, the US also has a destination for its spent fuel: Yucca Mountain in Nevada. The site, which is on federal land, was designated by Congress in 1987. However, progress has entirely stalled out because of political opposition. In 2011, the federal government stopped providing funding for the site, and for roughly a decade, there’s been no activity to speak of. In the meantime, waste continues to pile up. The nuclear industry is kicking into a new gear around the world. China is home to the world’s fastest–growing nuclear energy program, and countries including Bangladesh and Turkey are building their first reactors. Even the long-established US program is seeing growth: Interest in and approval for nuclear energy have spiked, and Big Tech is throwing money around to meet rising electricity demand. Companies are proposing (and beginning to receive regulatory approval for) next-generation reactors, which employ different coolants, fuels, and designs. Given all this new interest, and the impending arrival of new types of nuclear waste, it’s time for nuclear companies, as well as their powerful customers, to push for progress on building geological storage facilities. As the richest country on the planet and home to a large chunk of the activity in next-generation reactors, the US should aim to join the leaders rather than continue to lag behind.  Directing even a small fraction of the recent surge in funding and attention to progress on waste could make a difference. Some experts are calling for a new organization in the US to manage nuclear waste rather than leaving it to the Department of Energy. This organization would mirror programs in Finland, Canada, and France. The process of planning, building, and commissioning a permanent solution for nuclear waste is a long one. Finland started planning in the 1980s and selected its site in the early 2000s, and it’s nearly ready to start accepting waste. For countries that don’t have a permanent storage solution sorted, the best time to start was decades ago. But the second-best time is now.  This article is from The Spark, MIT Technology Review’s weekly climate newsletter. To receive it in your inbox every Wednesday, sign up here. 

In this tutorial, we build an embodied simulation vision agent that learns to perceive, plan, predict, and replan directly from pixel observations. We create a fully NumPy-rendered grid world in which the agent observes RGB frames rather than symbolic state variables, enabling us to simulate a simplified Vision-Language-Action-style pipeline. We train a lightweight world model that encodes visual input into a latent representation, predicts future states conditioned on actions and goals, and reconstructs the next frame. Using model predictive control in latent space, we enable the agent to sample possible action sequences, evaluate predicted outcomes, and execute the best action in a closed loop. Copy CodeCopiedUse a different Browser import random, numpy as np, torch, torch.nn as nn, torch.nn.functional as F import matplotlib.pyplot as plt from dataclasses import dataclass from typing import Tuple, Dict, List from torch.utils.data import Dataset, DataLoader try: from tqdm.auto import tqdm except Exception: def tqdm(x, **kwargs): return x SEED = 7 random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED) if device.type == “cuda”: torch.backends.cudnn.benchmark = True @dataclass class WorldConfig: grid_size: int = 8 cell_px: int = 14 max_steps: int = 45 n_obstacles: int = 8 spawn_margin: int = 1 class GridWorldRGBNoPIL: ACTIONS = {0:(0,-1),1:(0,1),2:(-1,0),3:(1,0),4:(0,0)} ACTION_NAMES = {0:”UP”,1:”DOWN”,2:”LEFT”,3:”RIGHT”,4:”STAY”} def __init__(self, cfg: WorldConfig): self.cfg = cfg self.reset() def reset(self) -> Dict: g = self.cfg.grid_size self.steps = 0 def sample_empty(exclude=set()): while True: x = random.randint(self.cfg.spawn_margin, g-1-self.cfg.spawn_margin) y = random.randint(self.cfg.spawn_margin, g-1-self.cfg.spawn_margin) if (x,y) not in exclude: return (x,y) self.obstacles = set() ax, ay = sample_empty() gx, gy = sample_empty(exclude={(ax,ay)}) used = {(ax,ay),(gx,gy)} for _ in range(self.cfg.n_obstacles): ox, oy = sample_empty(exclude=used) self.obstacles.add((ox,oy)) used.add((ox,oy)) self.agent = (ax,ay) self.goal = (gx,gy) return {“image”: self._render_u8()} def _in_bounds(self, x, y): return 0 <= x < self.cfg.grid_size and 0 <= y < self.cfg.grid_size def _dist_to_goal(self, pos: Tuple[int,int]) -> float: x,y = pos; gx,gy = self.goal return abs(x-gx)+abs(y-gy) def _state_vector(self) -> np.ndarray: g = self.cfg.grid_size – 1 ax,ay = self.agent; gx,gy = self.goal return np.array([ax/g, ay/g, gx/g, gy/g], dtype=np.float32) def step(self, action: int): self.steps += 1 dx, dy = self.ACTIONS[int(action)] x,y = self.agent nx, ny = x+dx, y+dy if self._in_bounds(nx,ny) and (nx,ny) not in self.obstacles: self.agent = (nx,ny) done = (self.agent == self.goal) or (self.steps >= self.cfg.max_steps) d_prev = self._dist_to_goal((x,y)) d_now = self._dist_to_goal(self.agent) reward = 0.1*(d_prev – d_now) + (1.0 if self.agent == self.goal else 0.0) obs = {“image”: self._render_u8()} info = {“state”: self._state_vector()} return obs, float(reward), bool(done), info def _render_u8(self) -> np.ndarray: g, s = self.cfg.grid_size, self.cfg.cell_px H = W = g*s bg = np.array([245,245,245], np.uint8) gridline = np.array([220,220,220], np.uint8) obstacle_c = np.array([220,70,70], np.uint8) goal_c = np.array([60,180,75], np.uint8) agent_c = np.array([65,105,225], np.uint8) img = np.empty((H,W,3), np.uint8); img[…] = bg img[::s,:,:] = gridline img[:,::s,:] = gridline def paint_cell(x,y,color): y0,y1 = y*s,(y+1)*s x0,x1 = x*s,(x+1)*s img[y0+1:y1-1, x0+1:x1-1] = color for (ox,oy) in self.obstacles: paint_cell(ox,oy, obstacle_c) gx,gy = self.goal; paint_cell(gx,gy, goal_c) ax,ay = self.agent; paint_cell(ax,ay, agent_c) return img cfg = WorldConfig() env = GridWorldRGBNoPIL(cfg) plt.figure(figsize=(3,3)) plt.imshow(env.reset()[“image”]); plt.axis(“off”); plt.title(“No-Pillow observation”); plt.show() def to_tensor_img_u8(img_u8: np.ndarray) -> torch.Tensor: return torch.from_numpy(img_u8).permute(2,0,1).float() / 255.0 We initialize the environment, set deterministic seeds, and define the lightweight grid-world configuration. We implement a fully NumPy-based RGB renderer so that the agent perceives raw pixel observations without relying on external libraries. We also define the state transition dynamics and prepare image-to-tensor conversion for model training. Copy CodeCopiedUse a different Browser class TransitionDataset(Dataset): def __init__(self, items): self.items = items def __len__(self): return len(self.items) def __getitem__(self, i): return self.items[i] def collect_transitions(n_episodes=120): items = [] e = GridWorldRGBNoPIL(cfg) for _ in tqdm(range(n_episodes), desc=”Collect”): obs = e.reset() img_t = to_tensor_img_u8(obs[“image”]) for _ in range(cfg.max_steps): a = random.randint(0,4) obs2, r, done, info = e.step(a) img_tp1 = to_tensor_img_u8(obs2[“image”]) st = torch.from_numpy(info[“state”]).float() goal = st[2:4].clone() items.append({ “img_t”: img_t, “action”: torch.tensor(a, dtype=torch.long), “img_tp1”: img_tp1, “state_tp1”: st, “goal”: goal }) img_t = img_tp1 if done: break return items items = collect_transitions(n_episodes=120) print(“Transitions:”, len(items)) H, W = items[0][“img_t”].shape[1], items[0][“img_t”].shape[2] dl = DataLoader(TransitionDataset(items), batch_size=64, shuffle=True, num_workers=0, drop_last=True) We collect rollout data by allowing the agent to interact randomly with the environment. We construct transitions that map the current image and action to the next image and state representation. We then wrap this data into a PyTorch Dataset and DataLoader to enable efficient mini-batch training. Copy CodeCopiedUse a different Browser class Encoder(nn.Module): def __init__(self, H, W, zdim=64): super().__init__() self.net = nn.Sequential( nn.Conv2d(3, 24, 5, stride=2, padding=2), nn.ReLU(), nn.Conv2d(24, 48, 5, stride=2, padding=2), nn.ReLU(), nn.Conv2d(48, 64, 3, stride=2, padding=1), nn.ReLU(), ) with torch.no_grad(): f = self.net(torch.zeros(1,3,H,W)) self.feat_shape = f.shape[1:] self.fc = nn.Linear(int(np.prod(self.feat_shape)), zdim) def forward(self, x): return self.fc(self.net(x).flatten(1)) class Decoder(nn.Module): def __init__(self, feat_shape, zdim=64): super().__init__() C,h,w = feat_shape self.C,self.h,self.w = C,h,w self.fc = nn.Linear(zdim, C*h*w) self.net = nn.Sequential( nn.ConvTranspose2d(C, 48, 4, stride=2, padding=1), nn.ReLU(), nn.ConvTranspose2d(48, 24, 4, stride=2, padding=1), nn.ReLU(), nn.ConvTranspose2d(24, 16, 4, stride=2, padding=1), nn.ReLU(), nn.Conv2d(16, 3, 3, padding=1), nn.Sigmoid() ) def forward(self, z): x = self.fc(z).view(z.size(0), self.C, self.h, self.w) return self.net(x) class VLASimLite(nn.Module): def __init__(self, H, W, zdim=64, adim=5): super().__init__() self.enc = Encoder(H,W,zdim) self.dec = Decoder(self.enc.feat_shape, zdim) self.aemb = nn.Embedding(adim, 16) self.gnet = nn.Sequential(nn.Linear(2,16), nn.ReLU(), nn.Linear(16,16)) self.dyn = nn.Sequential( nn.Linear(zdim+16+16, 128), nn.ReLU(), nn.Linear(128, zdim) ) self.state = nn.Sequential( nn.Linear(zdim, 64), nn.ReLU(), nn.Linear(64, 4), nn.Sigmoid() ) def encode(self, img): return self.enc(img) def predict_next_latent(self, z, a, goal): return self.dyn(torch.cat([z, self.aemb(a), self.gnet(goal)], dim=-1)) def decode(self, z): return self.dec(z) def forward(self, img_t, a, goal): z = self.encode(img_t) z_next = self.predict_next_latent(z, a, goal) return z_next, self.decode(z_next), self.state(z_next) model = VLASimLite(H,W,zdim=64,adim=5).to(device) opt = torch.optim.Adam(model.parameters(), lr=2e-3) We define the compact Vision-Language-Action-inspired world model. We build a CNN encoder to compress visual input into a latent space and condition latent dynamics on actions and goals. We also add a decoder and a state-prediction head so the model can reconstruct future frames and predict structured state variables. Copy CodeCopiedUse a different Browser def train(epochs=4): model.train() for ep in range(1, epochs+1): losses = [] for b in tqdm(dl, desc=f”Train {ep}/{epochs}”): img_t = b[“img_t”].to(device) a = b[“action”].to(device) img_tp1 = b[“img_tp1”].to(device) st_tp1 = b[“state_tp1”].to(device) goal = b[“goal”].to(device) z_next, img_pred, st_pred = model(img_t, a, goal) loss = F.l1_loss(img_pred, img_tp1) + 3.0*F.mse_loss(st_pred, st_tp1) + 1e-4*z_next.pow(2).mean() opt.zero_grad(set_to_none=True) loss.backward() nn.utils.clip_grad_norm_(model.parameters(), 2.0) opt.step() losses.append(loss.item()) print(“Epoch”, ep, “loss”, float(np.mean(losses))) train(epochs=4) We train the world model using a combination of image reconstruction

arXiv:2603.24231v2 Announce Type: replace Abstract: Stance detection is nearly always formulated as classifying text into Favor, Against, or Neutral. This convention was inherited from debate analysis and has been applied without modification to social media since SemEval-2016. However, attitudes toward complex targets are not unitary. A person can accept climate science while opposing carbon taxes, expressing support on one dimension and opposition on another. When annotators must compress such multi-dimensional attitudes into a single label, different annotators may weight different dimensions, producing disagreement that reflects different compression choices rather than confusion. We call this the projection problem. We conduct an annotation study across five targets from three stance benchmarks (SemEval-2016, P-Stance, COVID-19-Stance), with the same three annotators labeling all targets. For each target, annotators assign both a standard stance label and per-dimension judgments along target-specific dimensions discovered through bottom-up analysis, using the same number of categories for both. Across all fifteen target–dimension pairs, dimensional agreement consistently exceeds label agreement. The gap appears to scale with target complexity: modest for a single-entity target like Joe Biden (AC1: 0.87 vs. 0.95), but large for a multi-faceted policy target like school closures (AC1: 0.21 vs. 0.71).

arXiv:2512.09972v5 Announce Type: replace-cross Abstract: Navigating the capability–efficiency trade-off in Large Language Models (LLMs) requires approximating a high-quality Pareto set. Existing model merging research has focused predominantly on coarse model-level operators, which are easy to apply but offer limited control over the trade-off geometry. Layer-wise merging is more expressive, yet current methods still suffer from two bottlenecks: they treat the high-dimensional fusion space as an unstructured black box, and they rely on synchronous optimization despite highly uneven LLM evaluation latency. We propose Asynchronous Prior-guided Bayesian Model Merging (AP-BMM), which addresses these issues with a discrepancy-derived importance prior that initializes the surrogate geometry and an event-driven optimization loop built on pending-aware hypervolume improvement. Under a common evaluation budget, AP-BMM yields stronger Pareto-set approximations than both synchronous layer-wise baselines and representative model-level merging methods, with higher hypervolume and broader coverage of the trade-off frontier. Against the synchronous Bayesian baseline, it also achieves substantially shorter wall-clock time. Code: https://github.com/MiLab-HITSZ/AP-BMM.