Uncategorized

arXiv:2510.08956v2 Announce Type: replace Abstract: We study how open source communities describe participation and control through version controlled governance documents. Using a corpus of 710 projects with paired snapshots, we parse text into actors, rules, actions, and objects, then group them and measure change with entropy for evenness, richness for diversity, and Jensen Shannon divergence for drift. Projects define more roles and more actions over time, and these are distributed more evenly, while the composition of rules remains stable. These findings indicate that governance grows by expanding and balancing categories of participation without major shifts in prescriptive force. The analysis provides a reproducible baseline for evaluating whether future AI mediated workflows concentrate or redistribute authority.

arXiv:2508.19843v3 Announce Type: replace-cross Abstract: The broad capabilities and substantial resources required to train Large Language Models (LLMs) make them valuable intellectual property, yet they remain vulnerable to copyright infringement, such as unauthorized use and model theft. LLM fingerprinting, a non-intrusive technique that compares the distinctive features (i.e., fingerprint) of LLMs to identify whether an LLM is derived from another, offers a promising solution to copyright auditing. However, its reliability remains uncertain due to the prevalence of diverse model modifications and the lack of standardized evaluation. In this SoK, we present the first comprehensive study of the emerging LLM fingerprinting. We introduce a unified framework and taxonomy that structures the field: white-box methods are classified based on their feature source as static, forward-pass, or backward-pass fingerprinting, while black-box methods are distinguished by their query strategy as either untargeted or targeted. Furthermore, we propose LeaFBench, the first systematic benchmark for evaluating LLM fingerprinting under realistic deployment scenarios. Built upon 7 mainstream foundation models and comprising 149 distinct model instances, LeaFBench integrates 13 representative post-development techniques, spanning both parameter-altering methods (e.g., fine-tuning, quantization) and parameter-independent techniques (e.g., system prompts, RAG). Extensive experiments on LeaFBench reveal the strengths and weaknesses of existing methods, thereby outlining future research directions and critical open problems in this emerging field. The code is available at https://github.com/shaoshuo-ss/LeaFBench.

arXiv:2511.12784v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently emerged as powerful tools for autoformalization. Despite their impressive performance, these models can still struggle to produce grounded and verifiable formalizations. Recent work in text-to-SQL, has revealed that LLMs can be sensitive to paraphrased natural language (NL) inputs, even when high degrees of semantic fidelity are preserved (Safarzadeh, Oroojlooyjadid, and Roth 2025). In this paper, we investigate this claim in the autoformalization domain. Specifically, we evaluate the robustness of LLMs generating formal proofs with semantically similar paraphrased NL statements by measuring semantic and compilation validity. Using the formal benchmarks MiniF2F (Zheng, Han, and Polu 2021) and Lean 4 version of ProofNet (Xin et al. 2024), and two modern LLMs, we generate paraphrased natural language statements and cross-evaluate these statements across both models. The results of this paper reveal performance variability across paraphrased inputs, demonstrating that minor shifts in NL statements can significantly impact model outputs.

arXiv:2511.11324v1 Announce Type: new Abstract: Digitized histopathology analysis involves complex, time-intensive workflows and specialized expertise, limiting its accessibility. We introduce NOVA, an agentic framework that translates scientific queries into executable analysis pipelines by iteratively generating and running Python code. NOVA integrates 49 domain-specific tools (e.g., nuclei segmentation, whole-slide encoding) built on open-source software, and can also create new tools ad hoc. To evaluate such systems, we present SlideQuest, a 90-question benchmark — verified by pathologists and biomedical scientists — spanning data processing, quantitative analysis, and hypothesis testing. Unlike prior biomedical benchmarks focused on knowledge recall or diagnostic QA, SlideQuest demands multi-step reasoning, iterative coding, and computational problem solving. Quantitative evaluation shows NOVA outperforms coding-agent baselines, and a pathologist-verified case study links morphology to prognostically relevant PAM50 subtypes, demonstrating its scalable discovery potential.

arXiv:2411.03895v2 Announce Type: replace Abstract: In theatre, playwrights use the portrayal of characters to explore culturally based gender norms. In this paper, we develop quantitative methods to study gender depiction in the non-religious works (comedias) of Pedro Calder’on de la Barca, a prolific Spanish 17th century author. We gather insights from a corpus of more than 100 plays by using a gender classifier and applying model explainability (attribution) methods to determine which text features are most influential in the model’s decision to classify speech as ‘male’ or ‘female’, indicating the most gendered elements of dialogue in Calder’on’s comedias in a human accessible manner. We find that female and male characters are portrayed differently and can be identified by the gender prediction model at practically useful accuracies (up to f=0.83). Analysis reveals semantic aspects of gender portrayal, and demonstrates that the model is even useful in providing a relatively accurate scene-by-scene prediction of cross-dressing characters.

arXiv:2511.10846v1 Announce Type: new Abstract: Automated emotion detection is widely used in applications ranging from well-being monitoring to high-stakes domains like mental health and hiring. However, models often rely on annotations that reflect dominant cultural norms, limiting model ability to recognize emotional expression in dialects often excluded from training data distributions, such as African American Vernacular English (AAVE). This study examines emotion recognition model performance on AAVE compared to General American English (GAE). We analyze 2.7 million tweets geo-tagged within Los Angeles. Texts are scored for strength of AAVE using computational approximations of dialect features. Annotations of emotion presence and intensity are collected on a dataset of 875 tweets with both high and low AAVE densities. To assess model accuracy on a task as subjective as emotion perception, we calculate community-informed “silver” labels where AAVE-dense tweets are labeled by African American, AAVE-fluent (ingroup) annotators. On our labeled sample, GPT and BERT-based models exhibit false positive prediction rates of anger on AAVE more than double than on GAE. SpanEmo, a popular text-based emotion model, increases false positive rates of anger from 25 percent on GAE to 60 percent on AAVE. Additionally, a series of linear regressions reveals that models and non-ingroup annotations are significantly more correlated with profanity-based AAVE features than ingroup annotations. Linking Census tract demographics, we observe that neighborhoods with higher proportions of African American residents are associated with higher predictions of anger (Pearson’s correlation r = 0.27) and lower joy (r = -0.10). These results find an emergent safety issue of emotion AI reinforcing racial stereotypes through biased emotion classification. We emphasize the need for culturally and dialect-informed affective computing systems.

arXiv:2511.10686v1 Announce Type: new Abstract: This work aims to investigate how different Large Language Models (LLMs) alignment methods affect the models’ responses to prompt attacks. We selected open source models based on the most common alignment methods, namely, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Human Feedback (RLHF). We conducted a systematic analysis using statistical methods to verify how sensitive the Attack Success Rate (ASR) is when we apply variations to prompts designed to elicit inappropriate content from LLMs. Our results show that even small prompt modifications can significantly change the Attack Success Rate (ASR) according to the statistical tests we run, making the models more or less susceptible to types of attack. Critically, our results demonstrate that running existing ‘attack benchmarks’ alone may not be sufficient to elicit all possible vulnerabilities of both models and alignment methods. This paper thus contributes to ongoing efforts on model attack evaluation by means of systematic and statistically-based analyses of the different alignment methods and how sensitive their ASR is to prompt variation.

arXiv:2511.10688v1 Announce Type: new Abstract: As large language models (LLMs) are adopted in an increasingly wide range of applications, user-model interactions have grown in both frequency and scale. Consequently, research has focused on evaluating the robustness of LLMs, an essential quality for real-world tasks. In this paper, we employ simple multi-turn follow-up prompts to evaluate models’ answer changes, model accuracy dynamics across turns with Markov chains, and examine whether linear probes can predict these changes. Our results show significant vulnerabilities in LLM robustness: a simple “Think again” prompt led to an approximate 10% accuracy drop for Gemini 1.5 Flash over nine turns, while combining this prompt with a semantically equivalent reworded question caused a 7.5% drop for Claude 3.5 Haiku. Additionally, we find that model accuracy across turns can be effectively modeled using Markov chains, enabling the prediction of accuracy probabilities over time. This allows for estimation of the model’s stationary (long-run) accuracy, which we find to be on average approximately 8% lower than its first-turn accuracy for Gemini 1.5 Flash. Our results from a model’s hidden states also reveal evidence that linear probes can help predict future answer changes. Together, these results establish stationary accuracy as a principled robustness metric for interactive settings and expose the fragility of models under repeated questioning. Addressing this instability will be essential for deploying LLMs in high-stakes and interactive settings where consistent reasoning is as important as initial accuracy.

In this tutorial, we build an advanced interactive dashboard using Textual, and we explore how terminal-first UI frameworks can feel as expressive and dynamic as modern web dashboards. As we write and run each snippet, we actively construct the interface piece by piece, widgets, layouts, reactive state, and event flows, so we can see how Textual behaves like a live UI engine right inside Google Colab. By the end, we notice how naturally we can blend tables, trees, forms, and progress indicators into a cohesive application that feels fast, clean, and responsive. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser !pip install textual textual-web nest-asyncio from textual.app import App, ComposeResult from textual.containers import Container, Horizontal, Vertical from textual.widgets import ( Header, Footer, Button, DataTable, Static, Input, Label, ProgressBar, Tree, Select ) from textual.reactive import reactive from textual import on from datetime import datetime import random class StatsCard(Static): value = reactive(0) def __init__(self, title: str, *args, **kwargs): super().__init__(*args, **kwargs) self.title = title def compose(self) -> ComposeResult: yield Label(self.title) yield Label(str(self.value), id=”stat-value”) def watch_value(self, new_value: int) -> None: if self.is_mounted: try: self.query_one(“#stat-value”, Label).update(str(new_value)) except Exception: pass We set up the environment and import all the necessary components to build our Textual application. As we define the StatsCard widget, we establish a reusable component that reacts to changes in value and updates itself automatically. We begin to see how Textual’s reactive system lets us create dynamic UI elements with minimal effort. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser class DataDashboard(App): CSS = “”” Screen { background: $surface; } #main-container { height: 100%; padding: 1; } #stats-row { height: auto; margin-bottom: 1; } StatsCard { border: solid $primary; height: 5; padding: 1; margin-right: 1; width: 1fr; } #stat-value { text-style: bold; color: $accent; content-align: center middle; } #control-panel { height: 12; border: solid $secondary; padding: 1; margin-bottom: 1; } #data-section { height: 1fr; } #left-panel { width: 30; border: solid $secondary; padding: 1; margin-right: 1; } DataTable { height: 100%; border: solid $primary; } Input { margin: 1 0; } Button { margin: 1 1 1 0; } ProgressBar { margin: 1 0; } “”” BINDINGS = [ (“d”, “toggle_dark”, “Toggle Dark Mode”), (“q”, “quit”, “Quit”), (“a”, “add_row”, “Add Row”), (“c”, “clear_table”, “Clear Table”), ] total_rows = reactive(0) total_sales = reactive(0) avg_rating = reactive(0.0) We define the DataDashboard class and configure global styles, key bindings, and reactive attributes. We decide how the app should look and behave right from the top, giving us full control over themes and interactivity. This structure helps us create a polished dashboard without writing any HTML or JS. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser def compose(self) -> ComposeResult: yield Header(show_clock=True) with Container(id=”main-container”): with Horizontal(id=”stats-row”): yield StatsCard(“Total Rows”, id=”card-rows”) yield StatsCard(“Total Sales”, id=”card-sales”) yield StatsCard(“Avg Rating”, id=”card-rating”) with Vertical(id=”control-panel”): yield Input(placeholder=”Product Name”, id=”input-name”) yield Select( [(“Electronics”, “electronics”), (“Books”, “books”), (“Clothing”, “clothing”)], prompt=”Select Category”, id=”select-category” ) with Horizontal(): yield Button(“Add Row”, variant=”primary”, id=”btn-add”) yield Button(“Clear Table”, variant=”warning”, id=”btn-clear”) yield Button(“Generate Data”, variant=”success”, id=”btn-generate”) yield ProgressBar(total=100, id=”progress”) with Horizontal(id=”data-section”): with Container(id=”left-panel”): yield Label(“Navigation”) tree = Tree(“Dashboard”) tree.root.expand() products = tree.root.add(“Products”, expand=True) products.add_leaf(“Electronics”) products.add_leaf(“Books”) products.add_leaf(“Clothing”) tree.root.add_leaf(“Reports”) tree.root.add_leaf(“Settings”) yield tree yield DataTable(id=”data-table”) yield Footer() We compose the entire UI layout, arranging containers, cards, form inputs, buttons, a navigation tree, and a data table. As we structure these components, we watch the interface take shape exactly the way we envision it. This snippet lets us design the visual skeleton of the dashboard in a clean, declarative manner. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser def on_mount(self) -> None: table = self.query_one(DataTable) table.add_columns(“ID”, “Product”, “Category”, “Price”, “Sales”, “Rating”) table.cursor_type = “row” self.generate_sample_data(5) self.set_interval(0.1, self.update_progress) def generate_sample_data(self, count: int = 5) -> None: table = self.query_one(DataTable) categories = [“Electronics”, “Books”, “Clothing”] products = { “Electronics”: [“Laptop”, “Phone”, “Tablet”, “Headphones”], “Books”: [“Novel”, “Textbook”, “Magazine”, “Comic”], “Clothing”: [“Shirt”, “Pants”, “Jacket”, “Shoes”] } for _ in range(count): category = random.choice(categories) product = random.choice(products[category]) row_id = self.total_rows + 1 price = round(random.uniform(10, 500), 2) sales = random.randint(1, 100) rating = round(random.uniform(1, 5), 1) table.add_row( str(row_id), product, category, f”${price}”, str(sales), str(rating) ) self.total_rows += 1 self.total_sales += sales self.update_stats() def update_stats(self) -> None: self.query_one(“#card-rows”, StatsCard).value = self.total_rows self.query_one(“#card-sales”, StatsCard).value = self.total_sales if self.total_rows > 0: table = self.query_one(DataTable) total_rating = sum(float(row[5]) for row in table.rows) self.avg_rating = round(total_rating / self.total_rows, 2) self.query_one(“#card-rating”, StatsCard).value = self.avg_rating def update_progress(self) -> None: progress = self.query_one(ProgressBar) progress.advance(1) if progress.progress >= 100: progress.progress = 0 We implement all the logic for generating data, computing statistics, animating progress, and updating cards. We see how quickly we can bind backend logic to frontend components using Textual’s reactive model. This step makes the dashboard feel alive as numbers update instantly and progress bars animate smoothly. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser @on(Button.Pressed, “#btn-add”) def handle_add_button(self) -> None: name_input = self.query_one(“#input-name”, Input) category = self.query_one(“#select-category”, Select).value if name_input.value and category: table = self.query_one(DataTable) row_id = self.total_rows + 1 price = round(random.uniform(10, 500), 2) sales = random.randint(1, 100) rating = round(random.uniform(1, 5), 1) table.add_row( str(row_id), name_input.value, str(category), f”${price}”, str(sales), str(rating) ) self.total_rows += 1 self.total_sales += sales self.update_stats() name_input.value = “” @on(Button.Pressed, “#btn-clear”) def handle_clear_button(self) -> None: table = self.query_one(DataTable) table.clear() self.total_rows = 0 self.total_sales = 0 self.avg_rating = 0 self.update_stats() @on(Button.Pressed, “#btn-generate”) def handle_generate_button(self) -> None: self.generate_sample_data(10) def action_toggle_dark(self) -> None: self.dark = not self.dark def action_add_row(self) -> None: self.handle_add_button() def action_clear_table(self) -> None: self.handle_clear_button() if __name__ == “__main__”: import nest_asyncio nest_asyncio.apply() app = DataDashboard() app.run() We connect UI events to backend actions using button handlers, keyboard shortcuts, and app-level functions. As we run the app, we interact with a fully functional dashboard that responds instantly to every click and command. This snippet completes the application and demonstrates how easily Textual enables us to build dynamic, state-driven UIs. In conclusion, we see the whole dashboard come together in a fully functional, interactive form that runs directly from a notebook environment. We experience firsthand how Textual lets us design terminal UIs with the structure and