YouZum

新闻

AI, Committee, 新闻, Uncategorized

AP-BMM: Approximating Capability-Efficiency Pareto Sets of LLMs via Asynchronous Prior-guided Bayesian Model Merging

arXiv:2512.09972v5 Announce Type: replace-cross Abstract: Navigating the capability–efficiency trade-off in Large Language Models (LLMs) requires approximating a high-quality Pareto set. Existing model merging research has focused predominantly on coarse model-level operators, which are easy to apply but offer limited control over the trade-off geometry. Layer-wise merging is more expressive, yet current methods still suffer from two bottlenecks: they treat the high-dimensional fusion space as an unstructured black box, and they rely on synchronous optimization despite highly uneven LLM evaluation latency. We propose Asynchronous Prior-guided Bayesian Model Merging (AP-BMM), which addresses these issues with a discrepancy-derived importance prior that initializes the surrogate geometry and an event-driven optimization loop built on pending-aware hypervolume improvement. Under a common evaluation budget, AP-BMM yields stronger Pareto-set approximations than both synchronous layer-wise baselines and representative model-level merging methods, with higher hypervolume and broader coverage of the trade-off frontier. Against the synchronous Bayesian baseline, it also achieves substantially shorter wall-clock time. Code: https://github.com/MiLab-HITSZ/AP-BMM.

AP-BMM: Approximating Capability-Efficiency Pareto Sets of LLMs via Asynchronous Prior-guided Bayesian Model Merging Read Post »

AI, Committee, 新闻, Uncategorized

How to Build a Lightweight Vision-Language-Action-Inspired Embodied Agent with Latent World Modeling and Model Predictive Control

In this tutorial, we build an embodied simulation vision agent that learns to perceive, plan, predict, and replan directly from pixel observations. We create a fully NumPy-rendered grid world in which the agent observes RGB frames rather than symbolic state variables, enabling us to simulate a simplified Vision-Language-Action-style pipeline. We train a lightweight world model that encodes visual input into a latent representation, predicts future states conditioned on actions and goals, and reconstructs the next frame. Using model predictive control in latent space, we enable the agent to sample possible action sequences, evaluate predicted outcomes, and execute the best action in a closed loop. Copy CodeCopiedUse a different Browser import random, numpy as np, torch, torch.nn as nn, torch.nn.functional as F import matplotlib.pyplot as plt from dataclasses import dataclass from typing import Tuple, Dict, List from torch.utils.data import Dataset, DataLoader try: from tqdm.auto import tqdm except Exception: def tqdm(x, **kwargs): return x SEED = 7 random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED) if device.type == “cuda”: torch.backends.cudnn.benchmark = True @dataclass class WorldConfig: grid_size: int = 8 cell_px: int = 14 max_steps: int = 45 n_obstacles: int = 8 spawn_margin: int = 1 class GridWorldRGBNoPIL: ACTIONS = {0:(0,-1),1:(0,1),2:(-1,0),3:(1,0),4:(0,0)} ACTION_NAMES = {0:”UP”,1:”DOWN”,2:”LEFT”,3:”RIGHT”,4:”STAY”} def __init__(self, cfg: WorldConfig): self.cfg = cfg self.reset() def reset(self) -> Dict: g = self.cfg.grid_size self.steps = 0 def sample_empty(exclude=set()): while True: x = random.randint(self.cfg.spawn_margin, g-1-self.cfg.spawn_margin) y = random.randint(self.cfg.spawn_margin, g-1-self.cfg.spawn_margin) if (x,y) not in exclude: return (x,y) self.obstacles = set() ax, ay = sample_empty() gx, gy = sample_empty(exclude={(ax,ay)}) used = {(ax,ay),(gx,gy)} for _ in range(self.cfg.n_obstacles): ox, oy = sample_empty(exclude=used) self.obstacles.add((ox,oy)) used.add((ox,oy)) self.agent = (ax,ay) self.goal = (gx,gy) return {“image”: self._render_u8()} def _in_bounds(self, x, y): return 0 <= x < self.cfg.grid_size and 0 <= y < self.cfg.grid_size def _dist_to_goal(self, pos: Tuple[int,int]) -> float: x,y = pos; gx,gy = self.goal return abs(x-gx)+abs(y-gy) def _state_vector(self) -> np.ndarray: g = self.cfg.grid_size – 1 ax,ay = self.agent; gx,gy = self.goal return np.array([ax/g, ay/g, gx/g, gy/g], dtype=np.float32) def step(self, action: int): self.steps += 1 dx, dy = self.ACTIONS[int(action)] x,y = self.agent nx, ny = x+dx, y+dy if self._in_bounds(nx,ny) and (nx,ny) not in self.obstacles: self.agent = (nx,ny) done = (self.agent == self.goal) or (self.steps >= self.cfg.max_steps) d_prev = self._dist_to_goal((x,y)) d_now = self._dist_to_goal(self.agent) reward = 0.1*(d_prev – d_now) + (1.0 if self.agent == self.goal else 0.0) obs = {“image”: self._render_u8()} info = {“state”: self._state_vector()} return obs, float(reward), bool(done), info def _render_u8(self) -> np.ndarray: g, s = self.cfg.grid_size, self.cfg.cell_px H = W = g*s bg = np.array([245,245,245], np.uint8) gridline = np.array([220,220,220], np.uint8) obstacle_c = np.array([220,70,70], np.uint8) goal_c = np.array([60,180,75], np.uint8) agent_c = np.array([65,105,225], np.uint8) img = np.empty((H,W,3), np.uint8); img[…] = bg img[::s,:,:] = gridline img[:,::s,:] = gridline def paint_cell(x,y,color): y0,y1 = y*s,(y+1)*s x0,x1 = x*s,(x+1)*s img[y0+1:y1-1, x0+1:x1-1] = color for (ox,oy) in self.obstacles: paint_cell(ox,oy, obstacle_c) gx,gy = self.goal; paint_cell(gx,gy, goal_c) ax,ay = self.agent; paint_cell(ax,ay, agent_c) return img cfg = WorldConfig() env = GridWorldRGBNoPIL(cfg) plt.figure(figsize=(3,3)) plt.imshow(env.reset()[“image”]); plt.axis(“off”); plt.title(“No-Pillow observation”); plt.show() def to_tensor_img_u8(img_u8: np.ndarray) -> torch.Tensor: return torch.from_numpy(img_u8).permute(2,0,1).float() / 255.0 We initialize the environment, set deterministic seeds, and define the lightweight grid-world configuration. We implement a fully NumPy-based RGB renderer so that the agent perceives raw pixel observations without relying on external libraries. We also define the state transition dynamics and prepare image-to-tensor conversion for model training. Copy CodeCopiedUse a different Browser class TransitionDataset(Dataset): def __init__(self, items): self.items = items def __len__(self): return len(self.items) def __getitem__(self, i): return self.items[i] def collect_transitions(n_episodes=120): items = [] e = GridWorldRGBNoPIL(cfg) for _ in tqdm(range(n_episodes), desc=”Collect”): obs = e.reset() img_t = to_tensor_img_u8(obs[“image”]) for _ in range(cfg.max_steps): a = random.randint(0,4) obs2, r, done, info = e.step(a) img_tp1 = to_tensor_img_u8(obs2[“image”]) st = torch.from_numpy(info[“state”]).float() goal = st[2:4].clone() items.append({ “img_t”: img_t, “action”: torch.tensor(a, dtype=torch.long), “img_tp1”: img_tp1, “state_tp1”: st, “goal”: goal }) img_t = img_tp1 if done: break return items items = collect_transitions(n_episodes=120) print(“Transitions:”, len(items)) H, W = items[0][“img_t”].shape[1], items[0][“img_t”].shape[2] dl = DataLoader(TransitionDataset(items), batch_size=64, shuffle=True, num_workers=0, drop_last=True) We collect rollout data by allowing the agent to interact randomly with the environment. We construct transitions that map the current image and action to the next image and state representation. We then wrap this data into a PyTorch Dataset and DataLoader to enable efficient mini-batch training. Copy CodeCopiedUse a different Browser class Encoder(nn.Module): def __init__(self, H, W, zdim=64): super().__init__() self.net = nn.Sequential( nn.Conv2d(3, 24, 5, stride=2, padding=2), nn.ReLU(), nn.Conv2d(24, 48, 5, stride=2, padding=2), nn.ReLU(), nn.Conv2d(48, 64, 3, stride=2, padding=1), nn.ReLU(), ) with torch.no_grad(): f = self.net(torch.zeros(1,3,H,W)) self.feat_shape = f.shape[1:] self.fc = nn.Linear(int(np.prod(self.feat_shape)), zdim) def forward(self, x): return self.fc(self.net(x).flatten(1)) class Decoder(nn.Module): def __init__(self, feat_shape, zdim=64): super().__init__() C,h,w = feat_shape self.C,self.h,self.w = C,h,w self.fc = nn.Linear(zdim, C*h*w) self.net = nn.Sequential( nn.ConvTranspose2d(C, 48, 4, stride=2, padding=1), nn.ReLU(), nn.ConvTranspose2d(48, 24, 4, stride=2, padding=1), nn.ReLU(), nn.ConvTranspose2d(24, 16, 4, stride=2, padding=1), nn.ReLU(), nn.Conv2d(16, 3, 3, padding=1), nn.Sigmoid() ) def forward(self, z): x = self.fc(z).view(z.size(0), self.C, self.h, self.w) return self.net(x) class VLASimLite(nn.Module): def __init__(self, H, W, zdim=64, adim=5): super().__init__() self.enc = Encoder(H,W,zdim) self.dec = Decoder(self.enc.feat_shape, zdim) self.aemb = nn.Embedding(adim, 16) self.gnet = nn.Sequential(nn.Linear(2,16), nn.ReLU(), nn.Linear(16,16)) self.dyn = nn.Sequential( nn.Linear(zdim+16+16, 128), nn.ReLU(), nn.Linear(128, zdim) ) self.state = nn.Sequential( nn.Linear(zdim, 64), nn.ReLU(), nn.Linear(64, 4), nn.Sigmoid() ) def encode(self, img): return self.enc(img) def predict_next_latent(self, z, a, goal): return self.dyn(torch.cat([z, self.aemb(a), self.gnet(goal)], dim=-1)) def decode(self, z): return self.dec(z) def forward(self, img_t, a, goal): z = self.encode(img_t) z_next = self.predict_next_latent(z, a, goal) return z_next, self.decode(z_next), self.state(z_next) model = VLASimLite(H,W,zdim=64,adim=5).to(device) opt = torch.optim.Adam(model.parameters(), lr=2e-3) We define the compact Vision-Language-Action-inspired world model. We build a CNN encoder to compress visual input into a latent space and condition latent dynamics on actions and goals. We also add a decoder and a state-prediction head so the model can reconstruct future frames and predict structured state variables. Copy CodeCopiedUse a different Browser def train(epochs=4): model.train() for ep in range(1, epochs+1): losses = [] for b in tqdm(dl, desc=f”Train {ep}/{epochs}”): img_t = b[“img_t”].to(device) a = b[“action”].to(device) img_tp1 = b[“img_tp1”].to(device) st_tp1 = b[“state_tp1”].to(device) goal = b[“goal”].to(device) z_next, img_pred, st_pred = model(img_t, a, goal) loss = F.l1_loss(img_pred, img_tp1) + 3.0*F.mse_loss(st_pred, st_tp1) + 1e-4*z_next.pow(2).mean() opt.zero_grad(set_to_none=True) loss.backward() nn.utils.clip_grad_norm_(model.parameters(), 2.0) opt.step() losses.append(loss.item()) print(“Epoch”, ep, “loss”, float(np.mean(losses))) train(epochs=4) We train the world model using a combination of image reconstruction

How to Build a Lightweight Vision-Language-Action-Inspired Embodied Agent with Latent World Modeling and Model Predictive Control Read Post »

AI, Committee, 新闻, Uncategorized

The Download: Musk and Altman’s legal showdown, and AI’s profit problem

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Elon Musk and Sam Altman are going to court over OpenAI’s future Elon Musk and OpenAI CEO Sam Altman head to trial this week in a case with sweeping consequences. Ahead of OpenAI’s IPO, the court could rule on whether the company can exist as a for-profit enterprise. It could even oust its leadership. Musk, an OpenAI co-founder, claims he was deceived into bankrolling the firm under false pretenses. He’s seeking $134 billion in damages, the removal of Altman and president Greg Brockman, and the company’s restoration to a non-profit. Find out how the trial could upend the global AI race. —Michelle Kim The missing step between hype and profit In a celebrated South Park episode, a community of gnomes sneak out at night to steal underpants. Why? The gnomes present their pitch deck. “Phase 1: Collect underpants. Phase 2: ? Phase 3: Profit.” It’s a business plan that captures the current state of AI.  Companies have built the tech (Step 1) and promised transformation (Step 3). But how they get there is still a big question mark. Read about the potential paths forward. —Will Douglas Heaven This story originally appeared in The Algorithm, our weekly newsletter giving you the inside track on all things AI. Sign up to receive it in your inbox every Monday. Welcome to the era of weaponized deepfakes For years, experts have warned that deepfakes could be deployed in malicious ways. These dangers are now here. Cheap, accessible models now produce weaponized deepfakes—from sexually explicit images to political propaganda—that look startlingly real. They’re already inciting violence, changing minds, and sowing mistrust, with women and marginalized groups disproportionately affected. Experts fear that they’re cratering trust and critical thinking. Here’s why they’re alarmed. —Eileen Guo Weaponized deepfakes are on our list of the 10 Things That Matter in AI Right Now, MIT Technology Review’s guide to what’s really worth your attention in the busy, buzzy world of AI.  The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 OpenAI has ended its exclusive partnership with MicrosoftThe new deal allows OpenAI to court rivals such as Amazon. (Reuters $)+ Microsoft will still license OpenAI’s tech, but no longer exclusively. (NYT $)+ OpenAI is missing key growth targets ahead of its IPO. (WSJ $) 2 Google has signed a classified AI deal with the PentagonIt permits AI use for “any lawful government purpose.” (The Information $)+ Over 600 Google workers had called for a block on the deal. (QZ)+ AI firms are set to train military versions of their models on classified data. (MIT Technology Review) 3 The EU has told Google to open Android to AI rivalsIt wants to end Gemini’s built-in advantage. (Ars Technica)+ Google calls the move an “unwarranted intervention.” (WSJ $)+ A final decision is expected by the end of July. (Reuters $) 4 OpenAI is reportedly developing an AI-first smartphoneIt would replace apps with agents. (TechCrunch)+ Qualcomm and MediaTek may be developing its processors. (Gizmodo) 5 A brain implant for depression is moving into human testingThe FDA has approved a human study of the device. (Wired $)+ BCIs have thus far struggled to reach the market. (MIT Technology Review) 6 A populist backlash against AI is gaining momentum in rural AmericaFrom Indiana to Idaho, voters are pushing back against the technology. (NYT $)+ Anti-AI protests are expanding worldwide. (MIT Technology Review) 7 DeepSeek has priced its new model 97% below OpenAI’s GPT-5.5It aims to attract more enterprises, developers, and agent-based users. (SCMP)+ Here are three reasons why DeepSeek V4 matters. (MIT Technology Review) 8 AI now generates a third of new websitesA study found it’s making the web more cheery and less verbose. (404 Media) 9 Top talent is leaving Big Tech to launch their own AI startupsMeta, Google, and OpenAI are facing a brain drain. (CNBC) 10 Taylor Swift is trademarking her voice and imageThe Grammy winner has been the target of numerous deepfakes. (NBC News)+ A growing number of celebrities are fighting AI with trademarks. (BBC) Quote of the day “The reality is people don’t like him.” —Judge Yvonne Gonzalez Rogers reacts to prospective jurors confessing their negative views of Elon Musk ahead of his legal battle with Sam Altman, The Verge reports. One More Thing How covid conspiracy theories led to an alarming resurgence in AIDS denialism When Joe Rogan falsely declared that “party drugs” were an “important factor in AIDS,” several million people were listening. He also asserted that AZT, the earliest drug used to treat AIDS, killed people “quicker” than the disease itself—another claim that has been disproven. Such comments illustrate an unmistakable resurgence in AIDS denialism: a false collection of theories arguing either that HIV does not cause AIDS or that there is no such thing as HIV at all. By the dawn of the millennium, these claims had largely fallen out of favour. That changed when the coronavirus arrived. Follow the digital path from Covid skepticism to the return of a deadly conspiracy theory. —Anna Merlan We can still have nice things A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line.) + Explore the planets from your laptop with this live sky map.+ This marathon DJ set from Daphni is an incredible journey through electronic music.+ NASA’s stunning Artemis II wallpapers bring a high-res piece of deep space to your phone.+ This fascinating GPS explainer breaks down how your phone figures out exactly where you are.

The Download: Musk and Altman’s legal showdown, and AI’s profit problem Read Post »

AI, Committee, 新闻, Uncategorized

The Download: DeepSeek’s latest AI breakthrough, and the race to build world models

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Three reasons why DeepSeek’s new model matters On Friday, Chinese AI firm DeepSeek released a preview of V4, its long-awaited new flagship model. Notably, the model can process much longer prompts than its last generation, thanks to a new design that handles large amounts of text more efficiently. While the model remains open source, its performance matches leading closed-source rivals from Anthropic, OpenAI, and Google. It is also DeepSeek’s first release optimized Huawei’s Ascend chips—a key test of China’s dependence on Nvidia. Here are three ways V4 could shake up AI. —Caiwei Chen The rise of world models AI systems have already gained impressive mastery over the digital world, but the physical world remains humanity’s domain. As it turns out, building an AI that composes novels or code apps is far easier than developing one to fold laundry or navigate city streets. To bridge this gap, many researchers believe you need something called a world model. Proponents like Stanford professor Fei-Fei Li and AMI Labs founder Yann LeCun argue these models can overcome the well-known limitations of LLMs—and realize AI’s promise for robotics. Find out why they’ve brought world models to the forefront of the field. —Grace Huckins World models are on our list of the 10 Things That Matter in AI Right Now, our essential guide to what’s really worth your attention in the field. Subscribers can watch an exclusive roundtable unveiling the technologies and trends on the list, with analysis from MIT Technology Review’s AI reporter Grace Huckins and executive editors Amy Nordrum and Niall Firth. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 China has blocked Meta’s $2 billion acquisition of AI startup ManusRegulators cited national security grounds. (WSJ $)+ Beijing called the deal a “conspiratorial” attempt to hollow out its tech base. (FT $)+ The country is tightening its grip on AI firms that try to leave. (TechCrunch)+ The decision escalates China’s AI rivalry with the US. (Bloomberg $)+ But there will be no winners in their competition. (MIT Technology Review) 2 Google is investing up to $40 billion in AnthropicIn a deal valuing the AI firm at $350 billion. (CNBC)+ The funding will support the firm’s growing computing needs. (TechCrunch)+ Anthropic and OpenAI are fighting for compute capacity. (Axios) 3 President Trump just fired the entire National Science BoardThe NSF has played a crucial role in developing technology. (The Verge)+ The move heightens fears over political interference in US science. (Nature) 4 Conspiracy theories about the Washington shooting are proliferating onlineOver 300,000 posts appeared on X using the keyword “staged.” (NYT $)+ The theories are also swirling on Bluesky and Instagram. (Wired) 5 The AI compute crunch is starting to hit the broader economy.It’s affecting jobs, gadgets, and electricity prices. (404 Media)+ The AI compute explosion is the tech story of our time. (MIT Technology Review) 6 Elon Musk says a new banking tool brings X close to a “super app”He’s pledged to launch the tool this month. (Bloomberg) 7 AI optimism is surging across Asia while US sentiment coolsThe divide could shape where adoption happens fastest. (Rest of World) 8 Apple is tying its new CEO’s ascent to its first foldable iPhoneIt wants to build the buzz around John Ternus. (Gizmodo)  9 Twelve firms are developing the Golden Dome’s space-based interceptorsThey’ve won contracts worth up to $3.2 billion. (Ars Technica) 10 NASA has shared promising results from Artemis IIThe spacecraft and rocket fared well. (Engadget) Quote of the day “Getting out the truth and establishing facts and reliable information takes time. But our audiences really don’t have that kind of patience.” —Amanda Crawford, associate professor at the University of Connecticut, tells the NYT why conspiracy theories are gaining traction online. One More Thing MIRIAM MARTINCIC Welcome to Kenya’s Great Carbon Valley: a bold new gamble to fight climate change Kenya’s Great Rift Valley is home to five geothermal power stations, which harness clouds of steam to generate about a quarter of the country’s electricity. But some of the energy escapes into the atmosphere, while even more remains underground for lack of demand. That’s what brought Octavia Carbon here. Last year, the startup began harnessing some of that excess energy to remove CO2 from the air. The company says the method is efficient, affordable, and—crucially—scalable. But the project also faces fierce opposition.  Read the full story on the future of Kenya’s “Great Carbon Valley.” —Diana Kruzman We can still have nice things A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line.) + Fred Again’s Tiny Desk Concert is a masterclass in intimate performance.+ Here’s a delightful look at how we’re all linked through geography and shared heritage.+ Take a short, peaceful break to watch Tokyo’s cherry blossoms from a bird’s eye view.+ There’s something oddly satisfying about watching an industrial shredder turn everyday items into confetti.

The Download: DeepSeek’s latest AI breakthrough, and the race to build world models Read Post »

AI, Committee, 新闻, Uncategorized

Meta AI Releases Sapiens2: A High-Resolution Human-Centric Vision Model for Pose, Segmentation, Normals, Pointmap, and Albedo

If you’ve ever watched a motion capture system struggle with a person’s fingers, or seen a segmentation model fail to distinguish teeth from gums, you already understand why human-centric computer vision is hard. Humans are not just objects, they come with articulated structure, fine surface details, and enormous variation in pose, clothing, lighting, and ethnicity. Getting a model to understand all of that, at once, across arbitrary real-world images, is genuinely difficult. Meta AI research team introduced Sapiens2, the second generation of its foundation model family for human-centric vision. Trained on a newly curated dataset of 1 billion human images, spanning model sizes from 0.4B to 5B parameters, and designed to operate at native 1K resolution with hierarchical variants supporting 4K, Sapiens2 is a substantial leap over its predecessor across every benchmark the team evaluated. https://arxiv.org/pdf/2604.21681 What Sapiens2 is Trying to Solve The original Sapiens model relied primarily on Masked Autoencoder (MAE) pretraining. MAE works by masking a large portion of input image patches, 75% in this case, and training the model to reconstruct the missing pixels. This forces the model to learn spatial details and textures, which is useful for dense prediction tasks like segmentation or depth estimation. The problem is that MAE, as a form of masked image modeling (MIM), learns largely through compression. It doesn’t naturally learn high-level semantics. It can tell you what something looks like, but not necessarily what it means in the context of a human body. That’s where contrastive learning (CL) methods like DINO and SimCLR shine: they organize representations semantically by training the model to treat different views of the same image as similar and views of different images as distinct. But CL has its own tradeoff. Its aggressive augmentation strategies like color jitter, blurring, can strip away appearance cues like skin tone or lighting conditions that are critical for tasks like albedo estimation (recovering the true color of a surface independent of lighting). This is what the research team calls representation drift. Sapiens2 addresses this problem directly by combining both objectives: a masked image reconstruction loss (LMAE) to preserve low-level fidelity, and a global contrastive loss (LCL) on the [CLS] token using a student-teacher framework based on DINOv3, where the teacher’s parameters are an exponential moving average (EMA) of the student. Crucially, color augmentations are not applied to global views used for the MAE objective, preserving the appearance cues needed for photorealistic tasks. The joint objective is L = LMAE + λLCL. https://arxiv.org/pdf/2604.21681 The Data: Humans-1B Getting 1 billion training images right required a multi-stage filtering pipeline. Starting from a web-scale pool of approximately 4 billion images, Meta team applied bounding box detection, head-pose estimation, aesthetic and realism scoring, CLIP-based feature filtering, and text-overlay detection. The result is a curated corpus where every image contains at least one prominent person with a minimum short-side resolution of 384 pixels. To ensure diversity, the research team used perceptual hashing and deep-feature nearest-neighbor pruning for deduplication, then clustered visual embeddings and applied selective sampling to balance the dataset across poses, viewpoints, occlusion levels, clothing types, and lighting conditions. No task labels or human-specific priors were injected during pretraining — just images. The Architecture: Scaling to 5B and 4K Sapiens2 introduces four model sizes: 0.4B, 0.8B, 1B, and 5B parameters, each at native 1K resolution. The 5B model is the highest-FLOPs vision transformer reported to date at 15.722 TFLOPs. For 4K resolution, the research team adopted a hierarchical windowed attention design. The first K layers apply windowed self-attention locally to capture fine texture and boundaries within spatial windows. A [CLS]-guided pooling step then downsamples the 2D token grid by a spatial stride √ω, and the subsequent L layers apply global self-attention over this reduced sequence. This layout is compatible with MAE-style pretraining because masked tokens can be dropped after the local stage, preventing information from leaking across masked regions — a problem that convolutional backbones typically need masked convolutions to avoid. The masking strategy itself is also carefully designed: Sapiens2 uses mixed blockwise/patchwise masking (blockwise probability 0.4) at a 75% mask ratio with patch size 16. At 1024×768 resolution (64×48 = 3072 patches), this masks approximately 2304 patches per image which is enough to create coarse occlusions that regularize MAE while preserving sufficient context for the contrastive objective. For stability at scale, the architecture incorporates several improvements: RMSNorm replacing LayerNorm, Grouped-Query Attention (GQA) in mid-depth blocks for higher throughput, QK-Norm for robust high-resolution training, and SwiGLU feed-forward layers. The decoder uses pixel-shuffle upsampling for sub-pixel reasoning. Decoder output resolution was also increased from 0.5K to 1K for base backbones, and to 2K for 4K backbones. Post-Training: Five Human Tasks, 10× More Supervision A critical improvement over the original Sapiens is the scale and quality of task-specific supervision. Relative to the first generation, Sapiens2 scales task-specific labels by 10×, typically reaching around 1 million labels per task. After pretraining, the backbone is fine-tuned for five downstream tasks using lightweight task-specific heads while leaving the backbone unchanged: Pose Estimation: A 308-keypoint full-body skeleton with dense face (243 keypoints) and hand (40 keypoints) coverage. The research team newly annotated 100K in-the-wild images to complement studio capture data, improving generalization significantly. Body-Part Segmentation: 29 semantic classes (extended from 28 by adding eyeglasses), trained with per-pixel weighted cross-entropy combined with Dice loss for sharper boundaries. Pointmap Estimation: Rather than predicting relative depth, Sapiens2 regresses a per-pixel 3D pointmap P̂(u) ∈ ℝ³ in the camera frame — a harder task that requires reasoning about camera intrinsics. Normal Estimation: Per-pixel surface unit normals, decoded using multiple PixelShuffle layers for artifact-free upsampling. Albedo Estimation: Per-pixel diffuse albedo Â(u) ∈ [0,1]³, trained purely on synthetic high-fidelity data and designed to recover true skin tone and clothing color under varying illumination. Results The numbers are difficult to argue with. On the 11K-image in-the-wild pose test set, Sapiens2-5B achieves 82.3 mAP compared to 78.3 mAP for Sapiens-2B — a +4 mAP improvement. On body-part segmentation, even the smallest model, Sapiens2-0.4B, scores 79.5 mIoU (+21.3 over

Meta AI Releases Sapiens2: A High-Resolution Human-Centric Vision Model for Pose, Segmentation, Normals, Pointmap, and Albedo Read Post »

AI, Committee, 新闻, Uncategorized

Rebuilding the data stack for AI

Artificial intelligence may be dominating boardroom agendas, but many enterprises are discovering that the biggest obstacle to meaningful adoption is the state of their data. While consumer-facing AI tools have dazzled users with speed and ease, enterprise leaders are discovering that deploying AI at scale requires something far less glamorous but far more consequential: data infrastructure that is unified, governed, and fit for purpose. That gap between AI ambition and enterprise readiness is becoming one of the defining challenges of this next phase of digital transformation. As Bavesh Patel, senior vice president of Databricks, puts it, “the quality of that AI and how effective that AI is, is really dependent on information in your organization.” Yet in many companies, that information remains fragmented across legacy systems, siloed applications, and disconnected formats, making it nearly impossible for AI systems to generate trustworthy, context-rich outputs. “Really, the big competitive differentiator for most organizations is their own data and then their third-party data that they can add to it,” says Patel. For enterprise AI to deliver value, data must be consolidated into open formats, governed with precision, and made accessible across functions. Without that foundation, businesses risk “terrible AI,” as Patel bluntly describes it. That means moving beyond siloed SaaS platforms and disconnected dashboards toward a unified, open data architecture capable of combining structured and unstructured data, preserving real-time context, and enforcing rigorous access controls. When the groundwork is laid correctly, organizations can move toward measurable outcomes, unlocking efficiencies, automating complex workflows, and even launching entirely new lines of business. That value focus is critical, says Rajan Padmanabhan, unit technology officer at Infosys, especially as enterprises seek precision in the outputs driving business decisions. Rather than treating AI initiatives as isolated innovation projects, leading companies are tying AI deployment directly to business metrics, using governance frameworks to determine what delivers results and what should be abandoned quickly. “We see this big opportunity just with AI literacy with business users, where they’re very eager to understand how they should be thinking about AI,” adds Patel. “What does AI mean when you peel the covers? What are the pieces and the building blocks that you need to put in place, both from a technology and a training and an enablement standpoint?” The possibilities ahead are substantial. As AI agents evolve from copilots into autonomous operators capable of managing workflows and transactions, the organizations that win will be those that build the right foundation now. “What we are seeing as a new way of thinking is moving from a system of execution or a system of engagement to a system of action,” notes  Padmanabhan. “That is the new way we see the road ahead.” The future of AI in the enterprise will be determined by whether businesses can turn fragmented information into a strategic asset capable of powering both smarter decisions and entirely new ways of operating. This episode of Business Lab is produced in partnership with Infosys Topaz. Full Transcript: Megan Tatum: From MIT Technology Review, I’m Megan Tatum, and this is Business Lab, the show that helps business leaders make sense of new technologies coming out of the lab and into the marketplace. This episode is produced in partnership with Infosys Topaz. Now, recent advancements in AI may have unlocked some compelling new industrial applications, but a reliance on inadequate data models means that many enterprises are hitting a brick wall. AI and agentic AI in particular place a whole new set of demands on data. The technology requires greater access, context, and guardrails to operate effectively. Existing data models often fall short. They’re too fragmented or siloed. Data itself often lacks quality. To bridge the gap, they require an AI-ready upgrade. Two words for you: data reconfigured. My guest today, are Bavesh Patel, senior vice president for Go-to-Market at Databricks, and Rajan Padmanabhan, unit technology officer for data analytics and AI at Infosys. Welcome, Bavesh and Rajan. Rajan Padmanabhan: Thank you. Thanks for having us. Bavesh Patel: Thanks for having us. Megan: Fantastic. Thank you both so much for joining us today. Bavesh, if I could come to you first, when we talk about AI-ready data, what exactly do we mean? What new demands does AI place on data, and how does this impact the way it needs to be structured and used? Bavesh: Yeah. Great question. Appreciate you hosting us today. I think that obviously the whole world is enamored with AI because of all of the power that we can all see as users. AI is now democratized across hundreds of millions of users. And when we think about enterprises and businesses using AI, the quality of that AI and how effective that AI is really dependent on information in your organization, and that’s data. And what we found is that most enterprises, their data is kind of locked away in these different applications and different systems. And it’s very difficult to get a good view of, what is all my data? How trustworthy is it? How recent and fresh is it? And all of that is being injected into the AI. Unless you have a proper understanding of your data, the ability to ensure that it’s data that’s accurate and that can be used so that the AI can take advantage of it, you’re actually going to end up having terrible AI. We see a lot of customers spend time on cleansing their data, organizing their data, making sure it’s access controlled correctly, and that tends to be the fuel of good AI. Megan: Yeah. It’s such a foundational thing, isn’t it? But it can be missed, I think, quite easily. Rajan, what difference can having AI-ready data really make for enterprises as they unlock that full potential of AI and its applications? Rajan: First and foremost, thanks for having us. It’s a pleasure. I think in continuation of what Bavesh talked about, see, data and AI is pretty synonymous. And similarly, the consumer AI and enterprise AI and

Rebuilding the data stack for AI Read Post »

We use cookies to improve your experience and performance on our website. You can learn more at 隱私權政策 and manage your privacy settings by clicking Settings.

Privacy Preferences

You can choose your cookie settings by turning on/off each type of cookie as you wish, except for essential cookies.

Allow All
Manage Consent Preferences
  • Always Active

Save
zh_CN