Noticias

arXiv:2602.08321v1 Announce Type: new Abstract: Solving open-ended science questions remains challenging for large language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the data construction and reward design for scientific post-training. We develop a large-scale, systematic data processing pipeline that transforms heterogeneous open-source science data into Dr. SCI dataset, which comprises of 1M questions across eight STEM subjects, with explicit verifiable/open-ended splits, scalable difficulty annotation, and fine-grained rubrics that operationalize evaluation for open-ended answers. Building on this dataset, we propose the Dr. SCI post-training pipeline, which redesigns the standard SFT -> RL workflow through three components: (i) Exploration-Expanding SFT, which broadens the model’s reasoning pattern coverage prior to RL; (ii) Dynamic Difficulty Curriculum, which adapts training data to the model’s evolving scientific capability; and (iii) SciRubric-Guided RL, which enables stable reinforcement learning on open-ended scientific questions via rubric-based evaluation with explicit answer correctness. Qwen3-4B-Base trained using Dr.SCI pipeline achieves 63.2 on GPQA-diamond and 32.4 on GPQA-general, consistently improves over strong post-trained baselines such as o1-mini and GPT-4o, demonstrating substantial gains in scientific reasoning, especially in open-ended settings.

arXiv:2512.00884v2 Announce Type: replace-cross Abstract: A common and effective means for improving language model capabilities involves finetuning a “student” language model’s parameters on generations from a more proficient “teacher” model. Termed “synthetic data”, these generations are often produced before any student finetuning, but some work has considered generating new synthetic samples as training progresses. This paper studies and advocates for the latter case, where data are generated in an iterative, closed-loop fashion that is guided by the current state of the student model. For a fixed budget of generated samples, or a budget in terms of compute spent querying a teacher, we show that this curation of finetuning data affords improved student performance over static generation. Further, while there have been several LLM-specific methods proposed that operate in this regime, we find that simple, inexpensive selection criteria from the active learning literature tend to be most performant. We validate these claims across four mathematical and logical reasoning datasets using four different small language models.

arXiv:2602.06384v1 Announce Type: new Abstract: Producing outputs that satisfy both semantic intent and format constraints is essential for deploying large language models in user-facing and system-integrated workflows. In this work, we focus on Markdown formatting, which is ubiquitous in assistants, documentation, and tool-augmented pipelines but still prone to subtle, hard-to-detect errors (e.g., broken lists, malformed tables, inconsistent headings, and invalid code blocks) that can significantly degrade downstream usability. We present FMBench, a benchmark for adaptive Markdown output formatting that evaluates models under a wide range of instruction-following scenarios with diverse structural requirements. FMBench emphasizes real-world formatting behaviors such as multi-level organization, mixed content (natural language interleaved with lists/tables/code), and strict adherence to user-specified layout constraints. To improve Markdown compliance without relying on hard decoding constraints, we propose a lightweight alignment pipeline that combines supervised fine-tuning (SFT) with reinforcement learning fine-tuning. Starting from a base model, we first perform SFT on instruction-response pairs, and then optimize a composite objective that balances semantic fidelity with structural correctness. Experiments on two model families (OpenPangu and Qwen) show that SFT consistently improves semantic alignment, while reinforcement learning provides additional gains in robustness to challenging Markdown instructions when initialized from a strong SFT policy. Our results also reveal an inherent trade-off between semantic and structural objectives, highlighting the importance of carefully designed rewards for reliable formatted generation. Code is available at: https://github.com/FudanCVL/FMBench.

arXiv:2511.19279v3 Announce Type: replace-cross Abstract: A cognitive map is an internal model which encodes the abstract relationships among entities in the world, giving humans and animals the flexibility to adapt to new situations, with a strong out-of-distribution (OOD) generalization that current AI systems still do not possess. To bridge this gap, we introduce MapFormers, new architectures based on Transformer models, which can learn cognitive maps from observational data and perform path integration in parallel, in a self-supervised manner. Cognitive maps are learned in the model by disentangling structural relationships in the inputs from their specific content, a property that can be achieved naturally by updating the positional encoding in Transformers with input-dependent matrices. We developed two variants of MapFormers that unify absolute and relative positional encoding to model episodic (EM) and working memory (WM), respectively. We tested MapFormers on several tasks, including a classic 2D navigation task, showing that our models can learn a cognitive map of the underlying space and generalize OOD (e.g., to longer sequences) with near-perfect performance, unlike current architectures. Together, these results demonstrate the superiority of models designed to learn a cognitive map, and the importance of introducing a structural bias for structure-content disentanglement, which can be achieved in Transformers with input-dependent positional encoding. MapFormers have broad applications in both neuroscience and AI, by explaining the neural mechanisms giving rise to cognitive maps, while allowing these relation models to be learned at scale.

arXiv:2602.06412v1 Announce Type: new Abstract: Masked Diffusion Language Models generate sequences via iterative sampling that progressively unmasks tokens. However, they still recompute the attention and feed-forward blocks for every token position at every step — even when many unmasked tokens are essentially fixed, resulting in substantial waste in compute. We propose SureLock: when the posterior at an unmasked position has stabilized across steps (our sure condition), we lock that position — thereafter skipping its query projection and feed-forward sublayers — while caching its attention keys and values so other positions can continue to attend to it. This reduces the dominant per-iteration computational cost from $O(N^2d)$ to $O(MNd)$ where $N$ is the sequence length, $M$ is the number of unlocked token positions, and $d$ is the model dimension. In practice, $M$ decreases as the iteration progresses, yielding substantial savings. On LLaDA-8B, SureLock reduces algorithmic FLOPs by 30–50% relative to the same sampler without locking, while maintaining comparable generation quality. We also provide a theoretical analysis to justify the design rationale of SureLock: monitoring only the local KL at the lock step suffices to bound the deviation in final token probabilities. Our code will be available at https://daioba.github.io/surelock .

In this tutorial, we show how we treat prompts as first-class, versioned artifacts and apply rigorous regression testing to large language model behavior using MLflow. We design an evaluation pipeline that logs prompt versions, prompt diffs, model outputs, and multiple quality metrics in a fully reproducible manner. By combining classical text metrics with semantic similarity and automated regression flags, we demonstrate how we can systematically detect performance drift caused by seemingly small prompt changes. Along the tutorial, we focus on building a workflow that mirrors real software engineering practices, but applied to prompt engineering and LLM evaluation. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser !pip -q install -U “openai>=1.0.0” mlflow rouge-score nltk sentence-transformers scikit-learn pandas import os, json, time, difflib, re from typing import List, Dict, Any, Tuple import mlflow import pandas as pd import numpy as np from openai import OpenAI from rouge_score import rouge_scorer import nltk from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction from sentence_transformers import SentenceTransformer from sklearn.metrics.pairwise import cosine_similarity nltk.download(“punkt”, quiet=True) nltk.download(“punkt_tab”, quiet=True) if not os.getenv(“OPENAI_API_KEY”): try: from google.colab import userdata # type: ignore k = userdata.get(“OPENAI_API_KEY”) if k: os.environ[“OPENAI_API_KEY”] = k except Exception: pass if not os.getenv(“OPENAI_API_KEY”): import getpass os.environ[“OPENAI_API_KEY”] = getpass.getpass(“Enter OPENAI_API_KEY (input hidden): “).strip() assert os.getenv(“OPENAI_API_KEY”), “OPENAI_API_KEY is required.” We set up the execution environment by installing all required dependencies and importing the core libraries used throughout the tutorial. We securely load the OpenAI API key at runtime, ensuring credentials are never hard-coded in the notebook. We also initialize essential NLP resources to ensure the evaluation pipeline runs reliably across different environments. Copy CodeCopiedUse a different Browser MODEL = “gpt-4o-mini” TEMPERATURE = 0.2 MAX_OUTPUT_TOKENS = 250 ABS_SEM_SIM_MIN = 0.78 DELTA_SEM_SIM_MAX_DROP = 0.05 DELTA_ROUGE_L_MAX_DROP = 0.08 DELTA_BLEU_MAX_DROP = 0.10 mlflow.set_tracking_uri(“file:/content/mlruns”) mlflow.set_experiment(“prompt_versioning_llm_regression”) client = OpenAI() embedder = SentenceTransformer(“all-MiniLM-L6-v2”) EVAL_SET = [ { “id”: “q1”, “input”: “Summarize in one sentence: MLflow tracks experiments, runs, parameters, metrics, and artifacts.”, “reference”: “MLflow helps track machine learning experiments by logging runs with parameters, metrics, and artifacts.” }, { “id”: “q2”, “input”: “Rewrite professionally: ‘this model is kinda slow but it works ok.'”, “reference”: “The model is somewhat slow, but it performs reliably.” }, { “id”: “q3”, “input”: “Extract key fields as JSON: ‘Order 5531 by Alice costs $42.50 and ships to Toronto.'”, “reference”: ‘{“order_id”:”5531″,”customer”:”Alice”,”amount_usd”:42.50,”city”:”Toronto”}’ }, { “id”: “q4”, “input”: “Answer briefly: What is prompt regression testing?”, “reference”: “Prompt regression testing checks whether prompt changes degrade model outputs compared to a baseline.” }, ] PROMPTS = [ { “version”: “v1_baseline”, “prompt”: ( “You are a precise assistant.n” “Follow the user request carefully.n” “If asked for JSON, output valid JSON only.n” “User: {user_input}” ) }, { “version”: “v2_formatting”, “prompt”: ( “You are a helpful, structured assistant.n” “Respond clearly and concisely.n” “Prefer clean formatting.n” “User request: {user_input}” ) }, { “version”: “v3_guardrailed”, “prompt”: ( “You are a rigorous assistant.n” “Rules:n” “1) If user asks for JSON, output ONLY valid minified JSON.n” “2) Otherwise, keep the answer short and factual.n” “User: {user_input}” ) }, ] We define all experimental configurations, including model parameters, regression thresholds, and MLflow tracking settings. We construct the evaluation dataset and explicitly declare multiple prompt versions to compare and test for regressions. By centralizing these definitions, we ensure that prompt changes and evaluation logic remain controlled and reproducible. Copy CodeCopiedUse a different Browser def call_llm(formatted_prompt: str) -> str: resp = client.responses.create( model=MODEL, input=formatted_prompt, temperature=TEMPERATURE, max_output_tokens=MAX_OUTPUT_TOKENS, ) out = getattr(resp, “output_text”, None) if out: return out.strip() try: texts = [] for item in resp.output: if getattr(item, “type”, “”) == “message”: for c in item.content: if getattr(c, “type”, “”) in (“output_text”, “text”): texts.append(getattr(c, “text”, “”)) return “n”.join(texts).strip() except Exception: return “” smooth = SmoothingFunction().method3 rouge = rouge_scorer.RougeScorer([“rougeL”], use_stemmer=True) def safe_tokenize(s: str) -> List[str]: s = (s or “”).strip().lower() if not s: return [] try: return nltk.word_tokenize(s) except LookupError: return re.findall(r”bw+b”, s) def bleu_score(ref: str, hyp: str) -> float: r = safe_tokenize(ref) h = safe_tokenize(hyp) if len(h) == 0 or len(r) == 0: return 0.0 return float(sentence_bleu([r], h, smoothing_function=smooth)) def rougeL_f1(ref: str, hyp: str) -> float: scores = rouge.score(ref or “”, hyp or “”) return float(scores[“rougeL”].fmeasure) def semantic_sim(ref: str, hyp: str) -> float: embs = embedder.encode([ref or “”, hyp or “”], normalize_embeddings=True) return float(cosine_similarity([embs[0]], [embs[1]])[0][0]) We implement the core LLM invocation and evaluation metrics used to assess prompt quality. We compute BLEU, ROUGE-L, and semantic similarity scores to capture both surface-level and semantic differences in model outputs. It allows us to evaluate prompt changes from multiple complementary perspectives rather than relying on a single metric. Copy CodeCopiedUse a different Browser def evaluate_prompt(prompt_template: str) -> Tuple[pd.DataFrame, Dict[str, float], str]: rows = [] for ex in EVAL_SET: p = prompt_template.format(user_input=ex[“input”]) y = call_llm(p) ref = ex[“reference”] rows.append({ “id”: ex[“id”], “input”: ex[“input”], “reference”: ref, “output”: y, “bleu”: bleu_score(ref, y), “rougeL_f1”: rougeL_f1(ref, y), “semantic_sim”: semantic_sim(ref, y), }) df = pd.DataFrame(rows) agg = { “bleu_mean”: float(df[“bleu”].mean()), “rougeL_f1_mean”: float(df[“rougeL_f1”].mean()), “semantic_sim_mean”: float(df[“semantic_sim”].mean()), } outputs_jsonl = “n”.join(json.dumps(r, ensure_ascii=False) for r in rows) return df, agg, outputs_jsonl def log_text_artifact(text: str, artifact_path: str): mlflow.log_text(text, artifact_path) def prompt_diff(old: str, new: str) -> str: a = old.splitlines(keepends=True) b = new.splitlines(keepends=True) return “”.join(difflib.unified_diff(a, b, fromfile=”previous_prompt”, tofile=”current_prompt”)) def compute_regression_flags(baseline: Dict[str, float], current: Dict[str, float]) -> Dict[str, Any]: d_sem = baseline[“semantic_sim_mean”] – current[“semantic_sim_mean”] d_rouge = baseline[“rougeL_f1_mean”] – current[“rougeL_f1_mean”] d_bleu = baseline[“bleu_mean”] – current[“bleu_mean”] flags = { “abs_semantic_fail”: current[“semantic_sim_mean”] < ABS_SEM_SIM_MIN, “drop_semantic_fail”: d_sem > DELTA_SEM_SIM_MAX_DROP, “drop_rouge_fail”: d_rouge > DELTA_ROUGE_L_MAX_DROP, “drop_bleu_fail”: d_bleu > DELTA_BLEU_MAX_DROP, “delta_semantic”: float(d_sem), “delta_rougeL”: float(d_rouge), “delta_bleu”: float(d_bleu), } flags[“regression”] = any([flags[“abs_semantic_fail”], flags[“drop_semantic_fail”], flags[“drop_rouge_fail”], flags[“drop_bleu_fail”]]) return flags We build the evaluation and regression logic that runs each prompt against the evaluation set and aggregates results. We log prompt artifacts, prompt diffs, and evaluation outputs to MLflow, ensuring every experiment remains auditable. We also compute regression flags that automatically identify whether a prompt version degrades performance relative to the baseline. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“Running prompt versioning + regression testing with MLflow…”) print(f”Tracking URI: {mlflow.get_tracking_uri()}”) print(f”Experiment: {mlflow.get_experiment_by_name(‘prompt_versioning_llm_regression’).name}”) run_summary = [] baseline_metrics = None baseline_prompt = None baseline_df = None baseline_metrics_name = None with mlflow.start_run(run_name=f”prompt_regression_suite_{int(time.time())}”) as parent_run: mlflow.set_tag(“task”, “prompt_versioning_regression_testing”) mlflow.log_param(“model”, MODEL)

In this tutorial, we walk through an advanced, end-to-end exploration of Polyfactory, focusing on how we can generate rich, realistic mock data directly from Python type hints. We start by setting up the environment and progressively build factories for data classes, Pydantic models, and attrs-based classes, while demonstrating customization, overrides, calculated fields, and the generation of nested objects. As we move through each snippet, we show how we can control randomness, enforce constraints, and model real-world structures, making this tutorial directly applicable to testing, prototyping, and data-driven development workflows. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser import subprocess import sys def install_package(package): subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, package]) packages = [ “polyfactory”, “pydantic”, “email-validator”, “faker”, “msgspec”, “attrs” ] for package in packages: try: install_package(package) print(f”✓ Installed {package}”) except Exception as e: print(f”✗ Failed to install {package}: {e}”) print(“n”) print(“=” * 80) print(“SECTION 2: Basic Dataclass Factories”) print(“=” * 80) from dataclasses import dataclass from typing import List, Optional from datetime import datetime, date from uuid import UUID from polyfactory.factories import DataclassFactory @dataclass class Address: street: str city: str country: str zip_code: str @dataclass class Person: id: UUID name: str email: str age: int birth_date: date is_active: bool address: Address phone_numbers: List[str] bio: Optional[str] = None class PersonFactory(DataclassFactory[Person]): pass person = PersonFactory.build() print(f”Generated Person:”) print(f” ID: {person.id}”) print(f” Name: {person.name}”) print(f” Email: {person.email}”) print(f” Age: {person.age}”) print(f” Address: {person.address.city}, {person.address.country}”) print(f” Phone Numbers: {person.phone_numbers[:2]}”) print() people = PersonFactory.batch(5) print(f”Generated {len(people)} people:”) for i, p in enumerate(people, 1): print(f” {i}. {p.name} – {p.email}”) print(“n”) We set up the environment and ensure all required dependencies are installed. We also introduce the core idea of using Polyfactory to generate mock data from type hints. By initializing the basic dataclass factories, we establish the foundation for all subsequent examples. Copy CodeCopiedUse a different Browser print(“=” * 80) print(“SECTION 3: Customizing Factory Behavior”) print(“=” * 80) from faker import Faker from polyfactory.fields import Use, Ignore @dataclass class Employee: employee_id: str full_name: str department: str salary: float hire_date: date is_manager: bool email: str internal_notes: Optional[str] = None class EmployeeFactory(DataclassFactory[Employee]): __faker__ = Faker(locale=”en_US”) __random_seed__ = 42 @classmethod def employee_id(cls) -> str: return f”EMP-{cls.__random__.randint(10000, 99999)}” @classmethod def full_name(cls) -> str: return cls.__faker__.name() @classmethod def department(cls) -> str: departments = [“Engineering”, “Marketing”, “Sales”, “HR”, “Finance”] return cls.__random__.choice(departments) @classmethod def salary(cls) -> float: return round(cls.__random__.uniform(50000, 150000), 2) @classmethod def email(cls) -> str: return cls.__faker__.company_email() employees = EmployeeFactory.batch(3) print(“Generated Employees:”) for emp in employees: print(f” {emp.employee_id}: {emp.full_name}”) print(f” Department: {emp.department}”) print(f” Salary: ${emp.salary:,.2f}”) print(f” Email: {emp.email}”) print() print() print(“=” * 80) print(“SECTION 4: Field Constraints and Calculated Fields”) print(“=” * 80) @dataclass class Product: product_id: str name: str description: str price: float discount_percentage: float stock_quantity: int final_price: Optional[float] = None sku: Optional[str] = None class ProductFactory(DataclassFactory[Product]): @classmethod def product_id(cls) -> str: return f”PROD-{cls.__random__.randint(1000, 9999)}” @classmethod def name(cls) -> str: adjectives = [“Premium”, “Deluxe”, “Classic”, “Modern”, “Eco”] nouns = [“Widget”, “Gadget”, “Device”, “Tool”, “Appliance”] return f”{cls.__random__.choice(adjectives)} {cls.__random__.choice(nouns)}” @classmethod def price(cls) -> float: return round(cls.__random__.uniform(10.0, 1000.0), 2) @classmethod def discount_percentage(cls) -> float: return round(cls.__random__.uniform(0, 30), 2) @classmethod def stock_quantity(cls) -> int: return cls.__random__.randint(0, 500) @classmethod def build(cls, **kwargs): instance = super().build(**kwargs) if instance.final_price is None: instance.final_price = round( instance.price * (1 – instance.discount_percentage / 100), 2 ) if instance.sku is None: name_part = instance.name.replace(” “, “-“).upper()[:10] instance.sku = f”{instance.product_id}-{name_part}” return instance products = ProductFactory.batch(3) print(“Generated Products:”) for prod in products: print(f” {prod.sku}”) print(f” Name: {prod.name}”) print(f” Price: ${prod.price:.2f}”) print(f” Discount: {prod.discount_percentage}%”) print(f” Final Price: ${prod.final_price:.2f}”) print(f” Stock: {prod.stock_quantity} units”) print() print() We focus on generating simple but realistic mock data using dataclasses and default Polyfactory behavior. We show how to quickly create single instances and batches without writing any custom logic. It helps us validate how Polyfactory automatically interprets type hints to populate nested structures. Copy CodeCopiedUse a different Browser print(“=” * 80) print(“SECTION 6: Complex Nested Structures”) print(“=” * 80) from enum import Enum class OrderStatus(str, Enum): PENDING = “pending” PROCESSING = “processing” SHIPPED = “shipped” DELIVERED = “delivered” CANCELLED = “cancelled” @dataclass class OrderItem: product_name: str quantity: int unit_price: float total_price: Optional[float] = None @dataclass class ShippingInfo: carrier: str tracking_number: str estimated_delivery: date @dataclass class Order: order_id: str customer_name: str customer_email: str status: OrderStatus items: List[OrderItem] order_date: datetime shipping_info: Optional[ShippingInfo] = None total_amount: Optional[float] = None notes: Optional[str] = None class OrderItemFactory(DataclassFactory[OrderItem]): @classmethod def product_name(cls) -> str: products = [“Laptop”, “Mouse”, “Keyboard”, “Monitor”, “Headphones”, “Webcam”, “USB Cable”, “Phone Case”, “Charger”, “Tablet”] return cls.__random__.choice(products) @classmethod def quantity(cls) -> int: return cls.__random__.randint(1, 5) @classmethod def unit_price(cls) -> float: return round(cls.__random__.uniform(5.0, 500.0), 2) @classmethod def build(cls, **kwargs): instance = super().build(**kwargs) if instance.total_price is None: instance.total_price = round(instance.quantity * instance.unit_price, 2) return instance class ShippingInfoFactory(DataclassFactory[ShippingInfo]): @classmethod def carrier(cls) -> str: carriers = [“FedEx”, “UPS”, “DHL”, “USPS”] return cls.__random__.choice(carriers) @classmethod def tracking_number(cls) -> str: return ”.join(cls.__random__.choices(‘0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ’, k=12)) class OrderFactory(DataclassFactory[Order]): @classmethod def order_id(cls) -> str: return f”ORD-{datetime.now().year}-{cls.__random__.randint(100000, 999999)}” @classmethod def items(cls) -> List[OrderItem]: return OrderItemFactory.batch(cls.__random__.randint(1, 5)) @classmethod def build(cls, **kwargs): instance = super().build(**kwargs) if instance.total_amount is None: instance.total_amount = round(sum(item.total_price for item in instance.items), 2) if instance.shipping_info is None and instance.status in [OrderStatus.SHIPPED, OrderStatus.DELIVERED]: instance.shipping_info = ShippingInfoFactory.build() return instance orders = OrderFactory.batch(2) print(“Generated Orders:”) for order in orders: print(f”n Order {order.order_id}”) print(f” Customer: {order.customer_name} ({order.customer_email})”) print(f” Status: {order.status.value}”) print(f” Items ({len(order.items)}):”) for item in order.items: print(f” – {item.quantity}x {item.product_name} @ ${item.unit_price:.2f} = ${item.total_price:.2f}”) print(f” Total: ${order.total_amount:.2f}”) if order.shipping_info: print(f” Shipping: {order.shipping_info.carrier} – {order.shipping_info.tracking_number}”) print(“n”) We build more complex domain logic by introducing calculated and dependent fields within factories. We show how we can derive values such as final prices, totals, and shipping details after object creation. This allows us to model realistic business rules directly inside our test data generators. Copy CodeCopiedUse a different Browser print(“=” * 80) print(“SECTION 7: Attrs Integration”) print(“=” * 80) import attrs from polyfactory.factories.attrs_factory import AttrsFactory @attrs.define class BlogPost: title: str author: str content: str views: int = 0 likes: int = 0 published: bool = False published_at: Optional[datetime] = None tags: List[str] = attrs.field(factory=list) class BlogPostFactory(AttrsFactory[BlogPost]): @classmethod def title(cls) -> str: templates = [ “10 Tips for {}”,