{"id":86446,"date":"2026-04-27T15:36:25","date_gmt":"2026-04-27T15:36:25","guid":{"rendered":"https:\/\/youzum.net\/meta-ai-releases-sapiens2-a-high-resolution-human-centric-vision-model-for-pose-segmentation-normals-pointmap-and-albedo\/"},"modified":"2026-04-27T15:36:25","modified_gmt":"2026-04-27T15:36:25","slug":"meta-ai-releases-sapiens2-a-high-resolution-human-centric-vision-model-for-pose-segmentation-normals-pointmap-and-albedo","status":"publish","type":"post","link":"https:\/\/youzum.net\/zh\/meta-ai-releases-sapiens2-a-high-resolution-human-centric-vision-model-for-pose-segmentation-normals-pointmap-and-albedo\/","title":{"rendered":"Meta AI Releases Sapiens2: A High-Resolution Human-Centric Vision Model for Pose, Segmentation, Normals, Pointmap, and Albedo"},"content":{"rendered":"

If you\u2019ve ever watched a motion capture system struggle with a person\u2019s fingers, or seen a segmentation model fail to distinguish teeth from gums, you already understand why human-centric computer vision is hard. Humans are not just objects, they come with articulated structure, fine surface details, and enormous variation in pose, clothing, lighting, and ethnicity. Getting a model to understand all of that, at once, across arbitrary real-world images, is genuinely difficult.<\/p>\n

Meta AI research team introduced Sapiens2<\/strong>, the second generation of its foundation model family for human-centric vision. Trained on a newly curated dataset of 1 billion human images<\/strong>, spanning model sizes from 0.4B to 5B parameters, and designed to operate at native 1K resolution<\/strong> with hierarchical variants supporting 4K<\/strong>, Sapiens2 is a substantial leap over its predecessor across every benchmark the team evaluated.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.21681<\/figcaption><\/figure>\n<\/div>\n

What Sapiens2 is Trying to Solve<\/strong><\/h3>\n

The original Sapiens model relied primarily on Masked Autoencoder (MAE)<\/strong> pretraining. MAE works by masking a large portion of input image patches, 75% in this case, and training the model to reconstruct the missing pixels. This forces the model to learn spatial details and textures, which is useful for dense prediction tasks like segmentation or depth estimation.<\/p>\n

The problem is that MAE, as a form of masked image modeling (MIM), learns largely through compression. It doesn\u2019t naturally learn high-level semantics. It can tell you what something looks<\/em> like, but not necessarily what it means<\/em> in the context of a human body. That\u2019s where contrastive learning (CL) methods like DINO and SimCLR shine: they organize representations semantically by training the model to treat different views of the same image as similar and views of different images as distinct.<\/p>\n

But CL has its own tradeoff. Its aggressive augmentation strategies like color jitter, blurring, can strip away appearance cues like skin tone or lighting conditions that are critical for tasks like albedo estimation (recovering the true color of a surface independent of lighting). This is what the research team calls representation drift<\/strong>.<\/p>\n

Sapiens2 addresses this problem directly by combining both objectives: a masked image reconstruction loss (LMAE)<\/strong> to preserve low-level fidelity, and a global contrastive loss (LCL)<\/strong> on the [CLS] token using a student-teacher framework based on DINOv3, where the teacher\u2019s parameters are an exponential moving average (EMA) of the student. Crucially, color augmentations are not applied to global views<\/strong> used for the MAE objective, preserving the appearance cues needed for photorealistic tasks. The joint objective is L = LMAE<\/sub> + \u03bbLCL<\/sub><\/strong>.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.21681<\/figcaption><\/figure>\n<\/div>\n

The Data: Humans-1B<\/strong><\/h3>\n

Getting 1 billion training images right required a multi-stage filtering pipeline. Starting from a web-scale pool of approximately 4 billion images<\/strong>, Meta team applied bounding box detection, head-pose estimation, aesthetic and realism scoring, CLIP-based feature filtering, and text-overlay detection. The result is a curated corpus where every image contains at least one prominent person with a minimum short-side resolution of 384 pixels<\/strong>.<\/p>\n

To ensure diversity, the research team used perceptual hashing and deep-feature nearest-neighbor pruning for deduplication, then clustered visual embeddings and applied selective sampling to balance the dataset across poses, viewpoints, occlusion levels, clothing types, and lighting conditions. No task labels or human-specific priors were injected during pretraining \u2014 just images.<\/p>\n

The Architecture: Scaling to 5B and 4K<\/strong><\/h3>\n

Sapiens2 introduces four model sizes: 0.4B, 0.8B, 1B, and 5B parameters<\/strong>, each at native 1K resolution. The 5B model is the highest-FLOPs vision transformer reported to date at 15.722 TFLOPs<\/strong>.<\/p>\n

For 4K resolution, the research team adopted a hierarchical windowed attention design<\/strong>. The first K layers apply windowed self-attention locally to capture fine texture and boundaries within spatial windows. A [CLS]-guided pooling step then downsamples the 2D token grid by a spatial stride \u221a\u03c9, and the subsequent L layers apply global self-attention over this reduced sequence. This layout is compatible with MAE-style pretraining because masked tokens can be dropped after the local stage, preventing information from leaking across masked regions \u2014 a problem that convolutional backbones typically need masked convolutions to avoid.<\/p>\n

The masking strategy itself is also carefully designed: Sapiens2 uses mixed blockwise\/patchwise masking<\/strong> (blockwise probability 0.4) at a 75% mask ratio<\/strong> with patch size 16. At 1024\u00d7768 resolution (64\u00d748 = 3072 patches), this masks approximately 2304 patches per image which is enough to create coarse occlusions that regularize MAE while preserving sufficient context for the contrastive objective.<\/p>\n

For stability at scale, the architecture incorporates several improvements: RMSNorm<\/strong> replacing LayerNorm, Grouped-Query Attention (GQA)<\/strong> in mid-depth blocks for higher throughput, QK-Norm<\/strong> for robust high-resolution training, and SwiGLU feed-forward<\/strong> layers. The decoder uses pixel-shuffle<\/strong> upsampling for sub-pixel reasoning. Decoder output resolution was also increased from 0.5K to 1K for base backbones<\/strong>, and to 2K for 4K backbones<\/strong>.<\/p>\n

Post-Training: Five Human Tasks, 10\u00d7 More Supervision<\/strong><\/h3>\n

A critical improvement over the original Sapiens is the scale and quality of task-specific supervision. Relative to the first generation, Sapiens2 scales task-specific labels by 10\u00d7<\/strong>, typically reaching around 1 million labels per task<\/strong>. After pretraining, the backbone is fine-tuned for five downstream tasks using lightweight task-specific heads while leaving the backbone unchanged:<\/p>\n