Committee

Language model users often ask questions without enough detail, making it hard to understand what they want. For example, a question like “What book should I read next?” depends heavily on personal taste. At the same time, “How do antibiotics work?” should be answered differently depending on the user’s background knowledge. Current evaluation methods often overlook this missing context, resulting in inconsistent judgments. For instance, a response praising coffee might seem fine, but could be unhelpful or even harmful for someone with a health condition. Without knowing the user’s intent or needs, it’s difficult to fairly assess a model’s response quality.  Prior research has focused on generating clarification questions to address ambiguity or missing information in tasks such as Q&A, dialogue systems, and information retrieval. These methods aim to improve the understanding of user intent. Similarly, studies on instruction-following and personalization emphasize the importance of tailoring responses to user attributes, such as expertise, age, or style preferences. Some works have also examined how well models adapt to diverse contexts and proposed training methods to enhance this adaptability. Additionally, language model-based evaluators have gained traction due to their efficiency, although they can be biased, prompting efforts to improve their fairness through clearer evaluation criteria.  Researchers from the University of Pennsylvania, the Allen Institute for AI, and the University of Maryland, College Park have proposed contextualized evaluations. This method adds synthetic context (in the form of follow-up question-answer pairs) to clarify underspecified queries during language model evaluation. Their study reveals that including context can significantly impact evaluation outcomes, sometimes even reversing model rankings, while also improving agreement between evaluators. It reduces reliance on superficial features, such as style, and uncovers potential biases in default model responses, particularly toward WEIRD (Western, Educated, Industrialized, Rich, Democratic) contexts. The work also demonstrates that models exhibit varying sensitivities to different user contexts.  The researchers developed a simple framework to evaluate how language models perform when given clearer, contextualized queries. First, they selected underspecified queries from popular benchmark datasets and enriched them by adding follow-up question-answer pairs that simulate user-specific contexts. They then collected responses from different language models. They had both human and model-based evaluators compare responses in two settings: one with only the original query, and another with the added context. This allowed them to measure how context affects model rankings, evaluator agreement, and the criteria used for judgment. Their setup offers a practical way to test how models handle real-world ambiguity.  Adding context, such as user intent or audience, greatly improves model evaluation, boosting inter-rater agreement by 3–10% and even reversing model rankings in some cases. For instance, GPT-4 outperformed Gemini-1.5-Flash only when context was provided. Without it, evaluations focus on tone or fluency, while context shifts attention to accuracy and helpfulness. Default generations often reflect Western, formal, and general-audience biases, making them less effective for diverse users. Current benchmarks that ignore context risk produce unreliable results. To ensure fairness and real-world relevance, evaluations must pair context-rich prompts with matching scoring rubrics that reflect the actual needs of users.  In conclusion, Many user queries to language models are vague, lacking key context like user intent or expertise. This makes evaluations subjective and unreliable. To address this, the study proposes contextualized evaluations, where queries are enriched with relevant follow-up questions and answers. This added context helps shift the focus from surface-level traits to meaningful criteria, such as helpfulness, and can even reverse model rankings. It also reveals underlying biases; models often default to WEIRD (Western, Educated, Industrialized, Rich, Democratic) assumptions. While the study uses a limited set of context types and relies partly on automated scoring, it offers a strong case for more context-aware evaluations in future work.  Check out the Paper, Code, Dataset and Blog. All credit for this research goes to the researchers of this project. SUBSCRIBE NOW to our AI Newsletter The post Why Context Matters: Transforming AI Model Evaluation with Contextualized Queries appeared first on MarkTechPost.

Researchers at the University of Pennsylvania and the Allen Institute for Artificial Intelligence have developed a groundbreaking tool that allows open-source AI systems to match or surpass the visual understanding capabilities of proprietary models like GPT-4V and Gemini 1.5 Flash, potentially reshaping the competitive landscape between open and c…Read More

Hierarchical Reasoning Models (HRM) tackle complex reasoning tasks while being smaller, faster, and more data-efficient than large AI models.Read More

LLMs have demonstrated exceptional performance across multiple tasks by utilizing few-shot inference, also known as in-context learning (ICL). The main problem lies in selecting the most representative demonstrations from large training datasets. Early methods selected demonstrations based on relevance using similarity scores between each example and the input question. Current methods suggest using additional selection rules, along with similarity, to enhance the efficiency of demonstration selection. These improvements introduce significant computational overhead when the number of shots increases. The effectiveness of selected demonstrations should also consider the specific LLM in use, as different LLMs exhibit varying capabilities and knowledge domains. Researchers from Shanghai Jiao Tong University, Xiaohongshu Inc., Carnegie Mellon University, Peking University, No Affiliation, University College London, and University of Bristol have proposed FEEDER (FEw yet Essential Demonstration prE-selectoR), a method to identify a core subset of demonstrations containing the most representative examples in training data, adjusted to specific LLMs. To construct this subset, “sufficiency” and “necessity” metrics are introduced in the pre-selection stage, along with a tree-based algorithm. Moreover, FEEDER reduces training data size by 20% while maintaining performance and seamlessly integrating with various downstream demonstration selection techniques in ICL across LLMs ranging from 300M to 8B parameters. FEEDER is evaluated on 6 text classification datasets: SST-2, SST-5, COLA, TREC, SUBJ, and FPB, covering tasks from sentiment classification and linguistic analysis to textual entailment. It is also evaluated on the reasoning dataset GSM8K, the semantic-parsing dataset SMCALFlow, and the scientific question-answering dataset GPQA. The official splits for each dataset are directly followed to get the training and test data. Moreover, multiple LLM variants are utilized to evaluate the performance of the method, including two GPT-2 variants, GPT-neo with 1.3B parameters, GPT-3 with 6B parameters, Gemma-2 with 2B parameters, Llama-2 with 7B parameters, Llama-3 with 8B parameters, and Qwen-2.5 with 32B parameters as the LLM base. Results regarding in-context learning performance show that FEEDER enables retention of almost half the training samples while achieving superior or comparable performance. Evaluation of few-shot performance on complex tasks using LLMs like Gemma-2 shows that FEEDER improves performance even when LLMs struggle with challenging tasks. It performs effectively with large numbers of shots, handling situations where LLM performance usually drops when the number of examples increases from 5 to 10 due to noisy or repeated demonstrations. Moreover, FEEDER minimizes negative impact on LLM performance by evaluating the sufficiency and necessity of each demonstration, and helps in the performance stability of LLMs On bi-level optimization, FEEDER achieves improved performance by utilizing a small yet high-quality dataset for fine-tuning while simultaneously reducing computational expenses, aligning with the core-set selection principle. Results indicate that fine-tuning LLMs provides greater performance improvements compared to augmenting LLMs with contexts, with FEEDER achieving even better performance gains in fine-tuning settings. Performance analysis reveals that FEEDER’s effectiveness first rises and then drops with increasing number of runs or rounds (R and K, respectively), confirming that identifying representative subsets from training datasets enhances LLM performance. However, overly narrow subsets may limit potential performance gains. In conclusion, researchers introduced FEEDER, a demonstration pre-selector designed to use LLM capabilities and domain knowledge to identify high-quality demonstrations through an efficient discovery approach. It reduces training data requirements while maintaining comparable performance, offering a practical solution for efficient LLM deployment. Future research directions include exploring applications with larger LLMs and extending FEEDER’s capabilities to areas such as data safety and data management. FEEDER makes a valuable contribution to demonstration selection, providing researchers and practitioners with an effective tool for optimizing LLM performance while reducing computational overhead. Check out the Paper. All credit for this research goes to the researchers of this project. Meet the AI Dev Newsletter read by 40k+ Devs and Researchers from NVIDIA, OpenAI, DeepMind, Meta, Microsoft, JP Morgan Chase, Amgen, Aflac, Wells Fargo and 100s more [SUBSCRIBE NOW] The post FEEDER: A Pre-Selection Framework for Efficient Demonstration Selection in LLMs appeared first on MarkTechPost.

The move underscores Meta’s strategy of spending aggressively now to secure a dominant position in what it views as the next foundational technology platform.Read More

Advancements in artificial intelligence are rapidly closing the gap between digital reasoning and real-world interaction. At the forefront of this progress is embodied AI—the field focused on enabling robots to perceive, reason, and act effectively in physical environments. As industries look to automate complex spatial and temporal tasks—from household assistance to logistics—having AI systems that truly understand their surroundings and plan actions becomes critical. Introducing RoboBrain 2.0: A Breakthrough in Embodied Vision-Language AI Developed by the Beijing Academy of Artificial Intelligence (BAAI), RoboBrain 2.0 marks a major milestone in the design of foundation models for robotics and embodied artificial intelligence. Unlike conventional AI models, RoboBrain 2.0 unifies spatial perception, high-level reasoning, and long-horizon planning within a single architecture. Its versatility supports a diverse set of embodied tasks, such as affordance prediction, spatial object localization, trajectory planning, and multi-agent collaboration. Key Highlights of RoboBrain 2.0 Two Scalable Versions: Offers both a fast, resource-efficient 7-billion-parameter (7B) variant and a powerful 32-billion-parameter (32B) model for more demanding tasks. Unified Multi-Modal Architecture: Couples a high-resolution vision encoder with a decoder-only language model, enabling seamless integration of images, video, text instructions, and scene graphs. Advanced Spatial and Temporal Reasoning: Excels at tasks requiring an understanding of object relationships, motion forecasting, and complex, multi-step planning. Open-Source Foundation: Built using the FlagScale framework, RoboBrain 2.0 is designed for easy research adoption, reproducibility, and practical deployment. How RoboBrain 2.0 Works: Architecture and Training Multi-Modal Input Pipeline RoboBrain 2.0 ingests a diverse mix of sensory and symbolic data: Multi-View Images & Videos: Supports high-resolution, egocentric, and third-person visual streams for rich spatial context. Natural Language Instructions: Interprets a wide range of commands, from simple navigation to intricate manipulation instructions. Scene Graphs: Processes structured representations of objects, their relationships, and environmental layouts. The system’s tokenizer encodes language and scene graphs, while a specialized vision encoder utilizes adaptive positional encoding and windowed attention to process visual data effectively. Visual features are projected into the language model’s space via a multi-layer perceptron, enabling unified, multimodal token sequences. Three-Stage Training Process RoboBrain 2.0 achieves its embodied intelligence through a progressive, three-phase training curriculum: Foundational Spatiotemporal Learning: Builds core visual and language capabilities, grounding spatial perception and basic temporal understanding. Embodied Task Enhancement: Refines the model with real-world, multi-view video and high-resolution datasets, optimizing for tasks like 3D affordance detection and robot-centric scene analysis. Chain-of-Thought Reasoning: Integrates explainable step-by-step reasoning using diverse activity traces and task decompositions, underpinning robust decision-making for long-horizon, multi-agent scenarios. Scalable Infrastructure for Research and Deployment RoboBrain 2.0 leverages the FlagScale platform, offering: Hybrid parallelism for efficient use of compute resources Pre-allocated memory and high-throughput data pipelines to reduce training costs and latency Automatic fault tolerance to ensure stability across large-scale distributed systems This infrastructure allows for rapid model training, easy experimentation, and scalable deployment in real-world robotic applications. Real-World Applications and Performance RoboBrain 2.0 is evaluated on a broad suite of embodied AI benchmarks, consistently surpassing both open-source and proprietary models in spatial and temporal reasoning. Key capabilities include: Affordance Prediction: Identifying functional object regions for grasping, pushing, or interacting Precise Object Localization & Pointing: Accurately following textual instructions to find and point to objects or vacant spaces in complex scenes Trajectory Forecasting: Planning efficient, obstacle-aware end-effector movements Multi-Agent Planning: Decomposing tasks and coordinating multiple robots for collaborative goals Its robust, open-access design makes RoboBrain 2.0 immediately useful for applications in household robotics, industrial automation, logistics, and beyond. Potential in Embodied AI and Robotics By unifying vision-language understanding, interactive reasoning, and robust planning, RoboBrain 2.0 sets a new standard for embodied AI. Its modular, scalable architecture and open-source training recipes facilitate innovation across the robotics and AI research community. Whether you are a developer building intelligent assistants, a researcher advancing AI planning, or an engineer automating real-world tasks, RoboBrain 2.0 offers a powerful foundation for tackling the most complex spatial and temporal challenges. Check out the Paper and Codes. All credit for this research goes to the researchers of this project | Meet the AI Dev Newsletter read by 40k+ Devs and Researchers from NVIDIA, OpenAI, DeepMind, Meta, Microsoft, JP Morgan Chase, Amgen, Aflac, Wells Fargo and 100s more [SUBSCRIBE NOW] The post RoboBrain 2.0: The Next-Generation Vision-Language Model Unifying Embodied AI for Advanced Robotics appeared first on MarkTechPost.