Uncategorized

In this tutorial, we build an advanced meta-cognitive control agent that learns how to regulate its own depth of thinking. We treat reasoning as a spectrum, ranging from fast heuristics to deep chain-of-thought to precise tool-like solving, and we train a neural meta-controller to decide which mode to use for each task. By optimizing the trade-off between accuracy, computation cost, and a limited reasoning budget, we explore how an agent can monitor its internal state and adapt its reasoning strategy in real time. Through each snippet, we experiment, observe patterns, and understand how meta-cognition emerges when an agent learns to think about its own thinking. Check out the FULL CODE NOTEBOOK. Copy CodeCopiedUse a different Browser import random import numpy as np import torch import torch.nn as nn import torch.optim as optim OPS = [‘+’, ‘*’] def make_task(): op = random.choice(OPS) if op == ‘+’: a, b = random.randint(1, 99), random.randint(1, 99) else: a, b = random.randint(2, 19), random.randint(2, 19) return a, b, op def true_answer(a, b, op): return a + b if op == ‘+’ else a * b def true_difficulty(a, b, op): if op == ‘+’ and a <= 30 and b <= 30: return 0 if op == ‘*’ and a <= 10 and b <= 10: return 1 return 2 def heuristic_difficulty(a, b, op): score = 0 if op == ‘*’: score += 0.6 score += max(a, b) / 100.0 return min(score, 1.0) def fast_heuristic(a, b, op): if op == ‘+’: base = a + b noise = random.choice([-2, -1, 0, 0, 0, 1, 2, 3]) else: base = int(0.8 * a * b) noise = random.choice([-5, -3, 0, 0, 2, 5, 8]) return base + noise, 0.5 def deep_chain_of_thought(a, b, op, verbose=False): if op == ‘+’: x, y = a, b carry = 0 pos = 1 result = 0 step = 0 while x > 0 or y > 0 or carry: dx, dy = x % 10, y % 10 s = dx + dy + carry carry, digit = divmod(s, 10) result += digit * pos x //= 10; y //= 10; pos *= 10 step += 1 else: result = 0 step = 0 for i, d in enumerate(reversed(str(b))): row = a * int(d) * (10 ** i) result += row step += 1 return result, max(2.0, 0.4 * step) def tool_solver(a, b, op): return eval(f”{a}{op}{b}”), 1.2 ACTION_NAMES = [“fast”, “deep”, “tool”] We set up the world our meta-agent operates in. We generate arithmetic tasks, define ground-truth answers, estimate difficulty, and implement three different reasoning modes. As we run it, we observe how each solver behaves differently in terms of accuracy and computational cost, which form the foundation of the agent’s decision space. Check out the FULL CODE NOTEBOOK. Copy CodeCopiedUse a different Browser def encode_state(a, b, op, rem_budget, error_ema, last_action): a_n = a / 100.0 b_n = b / 100.0 op_plus = 1.0 if op == ‘+’ else 0.0 op_mul = 1.0 – op_plus diff_hat = heuristic_difficulty(a, b, op) rem_n = rem_budget / MAX_BUDGET last_onehot = [0.0, 0.0, 0.0] if last_action is not None: last_onehot[last_action] = 1.0 feats = [ a_n, b_n, op_plus, op_mul, diff_hat, rem_n, error_ema ] + last_onehot return torch.tensor(feats, dtype=torch.float32, device=device) STATE_DIM = 10 N_ACTIONS = 3 class PolicyNet(nn.Module): def __init__(self, state_dim, hidden=48, n_actions=3): super().__init__() self.net = nn.Sequential( nn.Linear(state_dim, hidden), nn.Tanh(), nn.Linear(hidden, hidden), nn.Tanh(), nn.Linear(hidden, n_actions) ) def forward(self, x): return self.net(x) policy = PolicyNet(STATE_DIM, hidden=48, n_actions=N_ACTIONS).to(device) optimizer = optim.Adam(policy.parameters(), lr=3e-3) We encode each task into a structured state that captures operands, operation type, predicted difficulty, remaining budget, and recent performance. We then define a neural policy network that maps this state to a probability distribution over actions. As we work through it, we see how the policy becomes the core mechanism through which the agent learns to regulate its thinking. Check out the FULL CODE NOTEBOOK. Copy CodeCopiedUse a different Browser GAMMA = 0.98 COST_PENALTY = 0.25 MAX_BUDGET = 25.0 EPISODES = 600 STEPS_PER_EP = 20 ERROR_EMA_DECAY = 0.9 def run_episode(train=True): log_probs = [] rewards = [] info = [] rem_budget = MAX_BUDGET error_ema = 0.0 last_action = None for _ in range(STEPS_PER_EP): a, b, op = make_task() state = encode_state(a, b, op, rem_budget, error_ema, last_action) logits = policy(state) dist = torch.distributions.Categorical(logits=logits) action = dist.sample() if train else torch.argmax(logits) act_idx = int(action.item()) if act_idx == 0: pred, cost = fast_heuristic(a, b, op) elif act_idx == 1: pred, cost = deep_chain_of_thought(a, b, op, verbose=False) else: pred, cost = tool_solver(a, b, op) correct = (pred == true_answer(a, b, op)) acc_reward = 1.0 if correct else 0.0 budget_penalty = 0.0 rem_budget -= cost if rem_budget < 0: budget_penalty = -1.5 * (abs(rem_budget) / MAX_BUDGET) step_reward = acc_reward – COST_PENALTY * cost + budget_penalty rewards.append(step_reward) if train: log_probs.append(dist.log_prob(action)) err = 0.0 if correct else 1.0 error_ema = ERROR_EMA_DECAY * error_ema + (1 – ERROR_EMA_DECAY) * err last_action = act_idx info.append({ “correct”: correct, “cost”: cost, “difficulty”: true_difficulty(a, b, op), “action”: act_idx }) if train: returns = [] G = 0.0 for r in reversed(rewards): G = r + GAMMA * G returns.append(G) returns = list(reversed(returns)) returns_t = torch.tensor(returns, dtype=torch.float32, device=device) baseline = returns_t.mean() adv = returns_t – baseline loss = -(torch.stack(log_probs) * adv).mean() optimizer.zero_grad() loss.backward() optimizer.step() return rewards, info We implement the heart of learning using the REINFORCE policy gradient algorithm. We run multi-step episodes, collect log-probabilities, accumulate rewards, and compute returns. As we execute this part, we watch the meta-controller adjust its strategy by reinforcing decisions that balance accuracy with cost. Check out the FULL CODE NOTEBOOK. Copy CodeCopiedUse a different Browser print(“Training meta-cognitive controller…”) for ep in range(EPISODES): rewards, _ = run_episode(train=True) if (ep + 1) % 100 == 0: print(f” episode {ep+1:4d} | avg reward {np.mean(rewards):.3f}”) def evaluate(n_episodes=50): all_actions = {0: [0,0,0], 1: [0,0,0], 2: [0,0,0]} stats = {0: {“n”:0,”acc”:0,”cost”:0}, 1: {“n”:0,”acc”:0,”cost”:0}, 2: {“n”:0,”acc”:0,”cost”:0}} for _ in range(n_episodes): _, info = run_episode(train=False) for step in info: d = step[“difficulty”] a_idx = step[“action”] all_actions[d][a_idx] += 1 stats[d][“n”] += 1 stats[d][“acc”] += 1 if step[“correct”] else 0 stats[d][“cost”] += step[“cost”] for d in

arXiv:2505.21265v2 Announce Type: replace Abstract: Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstream cross-lingual transfer, multilingual pretraining remains underexplored. We introduce PIXEL-M4, a model pretrained on four visually and linguistically diverse languages: English, Hindi, Ukrainian, and Simplified Chinese. Multilingual evaluations on semantic and syntactic tasks show that PIXEL-M4 outperforms an English-only counterpart on non-Latin scripts. Word-level probing analyses confirm that PIXEL-M4 captures rich linguistic features, even in languages not seen during pretraining. Furthermore, an analysis of its hidden representations shows that multilingual pretraining yields a semantic embedding space closely aligned across the languages used for pretraining. This work demonstrates that multilingual pretraining substantially enhances the capability of pixel language models to effectively support a diverse set of languages.

arXiv:2512.02807v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective, reward models are vulnerable to reward hacking, and self-evaluation methods suffer from prompt sensitivity and biases. In this work, we propose stable rank, an intrinsic, annotation-free quality signal derived from model representations. Stable rank measures the effective dimensionality of hidden states by computing the ratio of total variance to dominant-direction variance, capturing quality through how information distributes across representation dimensions. Empirically, stable rank achieves 84.04% accuracy on RewardBench and improves task accuracy by an average of 11.3 percentage points over greedy decoding via Best-of-N sampling. Leveraging this insight, we introduce Stable Rank Group Relative Policy Optimization (SR-GRPO), which uses stable rank as a reward signal for reinforcement learning. Without external supervision, SR-GRPO improves Qwen2.5-1.5B-Instruct by 10% on STEM and 19% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Our findings demonstrate that quality signals can be extracted from internal model geometry, offering a path toward scalable alignment without external supervision.

arXiv:2505.07409v2 Announce Type: replace Abstract: Democratic societies need reliable information. Misinformation in popular media, such as news articles or videos, threatens to impair civic discourse. Citizens are, unfortunately, not equipped to verify the flood of content consumed daily at increasing rates. This work aims to quantify the scientific accuracy of online media semi-automatically. We investigate the state of the art of climate-related ground truth knowledge representation. By semantifying media content of unknown veracity, their statements can be compared against these ground truth knowledge graphs. We implemented a workflow using LLM-based statement extraction and knowledge graph analysis. Our implementation can streamline content processing towards state-of-the-art knowledge representation and veracity quantification. Developed and evaluated with the help of 27 experts and detailed interviews with 10, the tool evidently provides a beneficial veracity indication. These findings are supported by 43 anonymous participants from a parallel user survey. This initial step, however, is unable to annotate public media at the required granularity and scale. Additionally, the identified state of climate change knowledge graphs is vastly insufficient to support this neurosymbolic fact-checking approach. Further work towards a FAIR (Findable, Accessible, Interoperable, Reusable) ground truth and complementary metrics is required to support civic discourse scientifically.

arXiv:2512.02555v1 Announce Type: new Abstract: Relevance modeling in e-commerce search remains challenged by semantic gaps in term-matching methods (e.g., BM25) and neural models’ reliance on the scarcity of domain-specific hard samples. We propose ADORE, a self-sustaining framework that synergizes three innovations: (1) A Rule-aware Relevance Discrimination module, where a Chain-of-Thought LLM generates intent-aligned training data, refined via Kahneman-Tversky Optimization (KTO) to align with user behavior; (2) An Error-type-aware Data Synthesis module that auto-generates adversarial examples to harden robustness; and (3) A Key-attribute-enhanced Knowledge Distillation module that injects domain-specific attribute hierarchies into a deployable student model. ADORE automates annotation, adversarial generation, and distillation, overcoming data scarcity while enhancing reasoning. Large-scale experiments and online A/B testing verify the effectiveness of ADORE. The framework establishes a new paradigm for resource-efficient, cognitively aligned relevance modeling in industrial applications.

arXiv:2512.02472v1 Announce Type: cross Abstract: AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.

arXiv:2512.01255v1 Announce Type: cross Abstract: Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models (LLMs). However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive benchmarks. Unfortunately, existing benchmarks suffer from three critical limitations: (1) incomplete coverage, such as covering a limited subset of CWE types; (2) underestimation of LLM capabilities caused by unreasonable ground truth labeling; and (3) overestimation due to unrealistic cases such as using isolated vulnerable files rather than complete projects. In this paper, we introduce, for the first time, three principles for constructing a benchmark for JavaScript vulnerability detection that directly address these limitations: (1) comprehensiveness, (2) no underestimation, and (3) no overestimation. Guided by these principles, we propose FORGEJS, the first automatic benchmark generation framework for evaluating LLMs’ capability in JavaScript vulnerability detection. Then, we use FORGEJS to construct ARENAJS-the first systematic benchmark for LLM-based JavaScript vulnerability detection-and further propose JUDGEJS, an automatic evaluation framework. We conduct the first systematic evaluation of LLMs for JavaScript vulnerability detection, leveraging JUDGEJS to assess seven popular commercial LLMs on ARENAJS. The results show that LLMs not only exhibit limited reasoning capabilities, but also suffer from severe robustness defects, indicating that reliable JavaScript vulnerability detection with LLMs remains an open challenge.

arXiv:2510.27378v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) outputs let us read a model’s step-by-step reasoning. Since any long, serial reasoning process must pass through this textual trace, the quality of the CoT is a direct window into what the model is thinking. This visibility could help us spot unsafe or misaligned behavior (monitorability), but only if the CoT is transparent about its internal reasoning (faithfulness). Fully measuring faithfulness is difficult, so researchers often focus on examining the CoT in cases where the model changes its answer after adding a cue to the input. This proxy finds some instances of unfaithfulness but loses information when the model maintains its answer, and does not investigate aspects of reasoning not tied to the cue. We extend these results to a more holistic sense of monitorability by introducing verbosity: whether the CoT lists every factor needed to solve the task. We combine faithfulness and verbosity into a single monitorability score that shows how well the CoT serves as the model’s external `working memory’, a property that many safety schemes based on CoT monitoring depend on. We evaluate instruction-tuned and reasoning models on BBH, GPQA, and MMLU. Our results show that models can appear faithful yet remain hard to monitor when they leave out key factors, and that monitorability differs sharply across model families. We release our evaluation code using the Inspect library to support reproducible future work.

arXiv:2511.10643v2 Announce Type: replace Abstract: Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model’s text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.

arXiv:2512.01439v1 Announce Type: new Abstract: India’s linguistic diversity presents both opportunities and challenges for fintech platforms. While the country has 31 major languages and over 100 minor ones, only 10% of the population understands English, creating barriers to financial inclusion. We present a multilingual conversational AI system for a financial assistance use case that supports code-mixed languages like Hinglish, enabling natural interactions for India’s diverse user base. Our system employs a multi-agent architecture with language classification, function management, and multilingual response generation. Through comparative analysis of multiple language models and real-world deployment, we demonstrate significant improvements in user engagement while maintaining low latency overhead (4-8%). This work contributes to bridging the language gap in digital financial services for emerging markets.