Committee

arXiv:2406.13069v4 Announce Type: replace Abstract: How novel are texts generated by language models (LMs) relative to their training corpora? In this work, we investigate the extent to which modern LMs generate $n$-grams from their training data, evaluating both (i) the probability LMs assign to complete training $n$-grams and (ii) $n$-novelty, the proportion of $n$-grams generated by an LM that did not appear in the training data (for arbitrarily large $n$). To enable arbitrary-length $n$-gram search over a corpus in constant time w.r.t. corpus size, we develop Rusty-DAWG, a novel search tool inspired by indexing of genomic data. We compare the novelty of LM-generated text to human-written text and explore factors that affect generation novelty, focusing on the Pythia models. We find that, for $n > 4$, LM-generated text is less novel than human-written text, though it is more novel for smaller $n$. Larger LMs and more constrained decoding strategies both decrease novelty. Finally, we show that LMs complete $n$-grams with lower loss if they are more frequent in the training data. Overall, our results reveal factors influencing the novelty of LM-generated text, and we release Rusty-DAWG to facilitate further pretraining data research.

arXiv:2504.09071v2 Announce Type: replace Abstract: Plan-guided summarization attempts to reduce hallucinations in small language models (SLMs) by grounding generated summaries to the source text, typically by targeting fine-grained details such as dates or named entities. In this work, we investigate whether plan-based approaches in SLMs improve summarization in long document, narrative tasks. Narrative texts’ length and complexity often mean they are difficult to summarize faithfully. We analyze existing plan-guided solutions targeting fine-grained details, and also propose our own higher-level, narrative-based plan formulation. Our results show that neither approach significantly improves on a baseline without planning in either summary quality or faithfulness. Human evaluation reveals that while plan-guided approaches are often well grounded to their plan, plans are equally likely to contain hallucinations compared to summaries. As a result, the plan-guided summaries are just as unfaithful as those from models without planning. Our work serves as a cautionary tale to plan-guided approaches to summarization, especially for long, complex domains such as narrative texts. Code available at https://github.com/amazon-science/plan-guided-summarization

arXiv:2502.19954v2 Announce Type: replace Abstract: Stance detection on social media aims to identify attitudes expressed in tweets towards specific targets. Current studies prioritize Large Language Models (LLMs) over Small Language Models (SLMs) due to the overwhelming performance improving provided by LLMs. However, heavily relying on LLMs for stance detection, regardless of the cost, is impractical for real-world social media monitoring systems that require vast data analysis. To this end, we propose textbf{underline{Co}}llaborative Stance Detection via Small-Large Language Model Consistency textbf{underline{Ver}}ification (textbf{CoVer}) framework, which enhances LLM utilization via context-shared batch reasoning and logical verification between LLM and SLM. Specifically, instead of processing each text individually, CoVer processes texts batch-by-batch, obtaining stance predictions and corresponding explanations via LLM reasoning in a shared context. Then, to exclude the bias caused by context noises, CoVer introduces the SLM for logical consistency verification. Finally, texts that repeatedly exhibit low logical consistency are classified using consistency-weighted aggregation of prior LLM stance predictions. Our experiments show that CoVer outperforms state-of-the-art methods across multiple benchmarks in the zero-shot setting, achieving 0.54 LLM queries per tweet while significantly enhancing performance. Our CoVer offers a more practical solution for LLM deploying for social media stance detection.

arXiv:2508.15810v1 Announce Type: new Abstract: The rise of social media and online communication platforms has led to the spread of Arabic textual posts and memes as a key form of digital expression. While these contents can be humorous and informative, they are also increasingly being used to spread offensive language and hate speech. Consequently, there is a growing demand for precise analysis of content in Arabic text and memes. This paper explores the potential of large language models to effectively identify hope, hate speech, offensive language, and emotional expressions within such content. We evaluate the performance of base LLMs, fine-tuned LLMs, and pre-trained embedding models. The evaluation is conducted using a dataset of Arabic textual speech and memes proposed in the ArabicNLP MAHED 2025 challenge. The results underscore the capacity of LLMs such as GPT-4o-mini, fine-tuned with Arabic textual speech, and Gemini Flash 2.5, fine-tuned with Arabic memes, to deliver the superior performance. They achieve up to 72.1%, 57.8%, and 79.6% macro F1 scores for tasks 1, 2, and 3, respectively, and secure first place overall in the Mahed 2025 challenge. The proposed solutions offer a more nuanced understanding of both text and memes for accurate and efficient Arabic content moderation systems.

arXiv:2508.14913v2 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated significant capabilities in solving mathematical problems expressed in natural language. However, multilingual and culturally-grounded mathematical reasoning in low-resource languages lags behind English due to the scarcity of socio-cultural task datasets that reflect accurate native entities such as person names, organization names, and currencies. Existing multilingual benchmarks are predominantly produced via translation and typically retain English-centric entities, owing to the high cost associated with human annotater-based localization. Moreover, automated localization tools are limited, and hence, truly localized datasets remain scarce. To bridge this gap, we introduce a framework for LLM-driven cultural localization of math word problems that automatically constructs datasets with native names, organizations, and currencies from existing sources. We find that translated benchmarks can obscure true multilingual math ability under appropriate socio-cultural contexts. Through extensive experiments, we also show that our framework can help mitigate English-centric entity bias and improves robustness when native entities are introduced across various languages.

arXiv:2508.15807v1 Announce Type: new Abstract: Large pre-trained language models often struggle to incorporate new domain-specific terminology when fine-tuned on small, specialized corpora. In this work, we address the challenge of vocabulary expansion in frozen LLMs by introducing a mathematically grounded method for knowledge distillation via KL divergence, even when the original and extended models use different tokenizations. This allows the student model to inherit distributional knowledge from the teacher despite differing vocabularies. We compare our KL-based distillation approach to conventional cross-entropy training, evaluating both methods across multiple strategies for initializing new token embeddings. After embedding initialization, models are further fine-tuned to integrate the new vocabulary. Each trained model is benchmarked on approximately 2000 code-generation tasks, where our approach achieves the best performance across the board. Finally, through mechanistic interpretability, we analyze how models learn representations for the new tokens, providing an explanation for the observed gains and offering insight into the structure of embedding space during vocabulary expansion.

Table of contents 1. Regulatory and Risk Posture 2. Capability vs. Cost, Latency, and Footprint 3. Security and Compliance Trade-offs 4. Deployment Patterns 5. Decision Matrix (Quick Reference) 6. Concrete Use-Cases 7. Performance/Cost Levers Before “Going Bigger” EXAMPLES No single solution universally wins between Large Language Models (LLMs, ≥30B parameters, often via APIs) and Small Language Models (SLMs, ~1–15B, typically open-weights or proprietary specialist models). For banks, insurers, and asset managers in 2025, your selection should be governed by regulatory risk, data sensitivity, latency and cost requirements, and the complexity of the use case. SLM-first is recommended for structured information extraction, customer service, coding assistance, and internal knowledge tasks, especially with retrieval-augmented generation (RAG) and strong guardrails. Escalate to LLMs for heavy synthesis, multi-step reasoning, or when SLMs cannot meet your performance bar within latency/cost envelope. Governance is mandatory for both: treat LLMs and SLMs under your model risk management framework (MRM), align to NIST AI RMF, and map high-risk applications (such as credit scoring) to obligations under the EU AI Act. 1. Regulatory and Risk Posture Financial services are subject to mature model governance standards. In the US, Federal Reserve/OCC/FDIC SR 11-7 covers any model used for business decisioning, including LLMs and SLMs. This means required validation, monitoring, and documentation—irrespective of model size. The NIST AI Risk Management Framework (AI RMF 1.0) is the gold standard for AI risk controls, now widely adopted by financial institutions for both traditional and generative AI risks. In the EU, the AI Act is in force, with staged compliance dates (Aug 2025 for general purpose models, Aug 2026 for high-risk systems such as credit scoring per Annex III). High-risk means pre-market conformity, risk management, documentation, logging, and human oversight. Institutions targeting the EU must align remediation timelines accordingly. Core sectoral data rules apply: GLBA Safeguards Rule: Security controls and vendor oversight for consumer financial data. PCI DSS v4.0: New cardholder data controls—mandatory from March 31, 2025, with upgraded authentication, retention, and encryption. Supervisors (FSB/BIS/ECB) and standard setters highlight systemic risk from concentration, vendor lock-in, and model risk—neutral to model size. Key point: High-risk uses (credit, underwriting) require tight controls regardless of parameters. Both SLMs and LLMs demand traceable validation, privacy assurance, and sector compliance. 2. Capability vs. Cost, Latency, and Footprint SLMs (3–15B) now deliver strong accuracy on domain workloads, especially after fine-tuning and with retrieval augmentation. Recent SLMs (e.g., Phi-3, FinBERT, COiN) excel at targeted extraction, classification, and workflow augmentation, cut latency (<50ms), and allow self-hosting for strict data residency—and are feasible for edge deployment. LLMs unlock cross-document synthesis, heterogeneous data reasoning, and long-context operations (>100K tokens). Domain-specialized LLMs (e.g., BloombergGPT, 50B) outperform general models on financial benchmarks and multi-step reasoning tasks. Compute economics: Transformer self-attention scales quadratically with sequence length. FlashAttention/SlimAttention optimizations reduce compute costs, but don’t defeat the quadratic lower bound; long-context LLMs can be exponentially costlier at inference than short-context SLMs. Key point: Short, structured, latency-sensitive tasks (contact center, claims, KYC extraction, knowledge search) fit SLMs. If you need 100K+ token contexts or deep synthesis, budget for LLMs and mitigate cost via caching and selective “escalation.” 3. Security and Compliance Trade-offs Common risks: Both model types are exposed to prompt injection, insecure output handling, data leakage, and supply chain risks. SLMs: Preferred for self-hosting—satisfying GLBA/PCI/data sovereignty concerns and minimizing legal risks from cross-border transfers. LLMs: APIs introduce concentration and lock-in risks; supervisors require documented exit, fallback, and multi-vendor strategies. Explainability: High-risk uses require transparent features, challenger models, full decision logs, and human oversight; LLM reasoning traces cannot substitute for formal validation required by SR 11-7 / EU AI Act. 4. Deployment Patterns Three proven modes in finance: SLM-first, LLM fallback: Route 80%+ queries to a tuned SLM with RAG; escalate low-confidence/long-context cases to an LLM. Predictable cost/latency; good for call centers, operations, and form parsing. LLM-primary with tool-use: LLM as orchestrator for synthesis, with deterministic tools for data access, calculations, and protected by DLP. Suited for complex research, policy/regulatory work. Domain-specialized LLM: Large models adapted to financial corpora; higher MRM burden but measurable gains for niche tasks. Regardless, always implement content filters, PII redaction, least-privilege connectors, output verification, red-teaming, and continuous monitoring under NIST AI RMF and OWASP guidance. 5. Decision Matrix (Quick Reference) Criterion Prefer SLM Prefer LLM Regulatory exposure Internal assist, non-decisioning High-risk use (credit scoring) w/ full validation Data sensitivity On-prem/VPC, PCI/GLBA constraints External API with DLP, encryption, DPAs Latency & cost Sub-second, high QPS, cost-sensitive Seconds-latency, batch, low QPS Complexity Extraction, routing, RAG-aided draft Synthesis, ambiguous input, long-form context Engineering ops Self-hosted, CUDA, integration Managed API, vendor risk, rapid deployment 6. Concrete Use-Cases Customer Service: SLM-first with RAG/tools for common issues, LLM escalation for complex multi-policy queries. KYC/AML & Adverse Media: SLMs suffice for extraction/normalization; escalate to LLMs for fraud or multilingual synthesis. Credit Underwriting: High-risk (EU AI Act Annex III); use SLM/classical ML for decisioning, LLMs for explanatory narratives, always with human review. Research/Portfolio Notes: LLMs enable draft synthesis and cross-source collation; read-only access, citation logging, tool verification recommended. Developer Productivity: On-prem SLM code assistants for speed/IP safety; LLM escalation for refactoring or complex synthesis. 7. Performance/Cost Levers Before “Going Bigger” RAG optimization: Most failures are retrieval, not “model IQ.” Improve chunking, recency, relevance ranking before increasing size. Prompt/IO controls: Guardrails for input/output schema, anti-prompt-injection per OWASP. Serve-time: Quantize SLMs, page KV cache, batch/stream, cache frequent answers; quadratic attention inflates indiscriminate long contexts. Selective escalation: Route by confidence; >70% cost saving possible. Domain adaptation: Lightweight tuning/LoRA on SLMs closes most gaps; use large models only for clear, measurable lift in performance. EXAMPLES Example 1: Contract Intelligence at JPMorgan (COiN) JPMorgan Chase deployed a specialized Small Language Model (SLM), called COiN, to automate the review of commercial loan agreements—a process traditionally handled manually by legal staff. By training COiN on thousands of legal documents and regulatory filings, the bank slashed contract review times from several weeks to mere hours, achieving high accuracy and compliance traceability while drastically reducing operational cost. This targeted SLM solution