Uncategorized

arXiv:2511.20459v1 Announce Type: new Abstract: Recent advances in large language models have created new opportunities for stylometry, the study of writing styles and authorship. Two challenges, however, remain central: training generative models when no paired data exist, and evaluating stylistic text without relying only on human judgment. In this work, we present a framework for both generating and evaluating sentences in the style of 19th-century novelists. Large language models are fine-tuned with minimal, single-token prompts to produce text in the voices of authors such as Dickens, Austen, Twain, Alcott, and Melville. To assess these generative models, we employ a transformer-based detector trained on authentic sentences, using it both as a classifier and as a tool for stylistic explanation. We complement this with syntactic comparisons and explainable AI methods, including attention-based and gradient-based analyses, to identify the linguistic cues that drive stylistic imitation. Our findings show that the generated text reflects the authors’ distinctive patterns and that AI-based evaluation offers a reliable alternative to human assessment. All artifacts of this work are published online.

arXiv:2511.20056v1 Announce Type: new Abstract: Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user’s bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.

arXiv:2511.19935v1 Announce Type: cross Abstract: The rapid advancement of large language models (LLMs) has increased the demand for domain-specialized variants in areas such as law, healthcare, and finance. However, their large size remains a barrier to deployment in resource-constrained environments, and existing compression methods either generalize poorly across domains or incur high overhead. In this work, we propose textbf{EfficientXpert}, a lightweight domain-pruning framework that combines a propagation-aware pruning criterion (Foresight Mask) with an efficient adapter-update algorithm (Partial Brain Surgeon). Integrated into the LoRA fine-tuning process, EfficientXpert enables a one-step transformation of general pretrained models into sparse, domain-adapted experts. Across health and legal tasks, it retains up to 98% of dense-model performance at 40% sparsity, outperforming state-of-the-art methods. Further analysis reveals substantial domain-dependent structural shifts that degrade the effectiveness of general pruning masks, underscoring the need for adaptive, domain-aware pruning strategies tailored to each domain.

In this tutorial, we explore how to build neural networks from scratch using Tinygrad while remaining fully hands-on with tensors, autograd, attention mechanisms, and transformer architectures. We progressively build every component ourselves, from basic tensor operations to multi-head attention, transformer blocks, and, finally, a working mini-GPT model. Through each stage, we observe how Tinygrad’s simplicity helps us understand what happens under the hood when models train, optimize, and fuse kernels for performance. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser import subprocess, sys, os print(“Installing dependencies…”) subprocess.check_call([“apt-get”, “install”, “-qq”, “clang”], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, “git+https://github.com/tinygrad/tinygrad.git”]) import numpy as np from tinygrad import Tensor, nn, Device from tinygrad.nn import optim import time print(f” Using device: {Device.DEFAULT}”) print(“=” * 60) print(“n PART 1: Tensor Operations & Autograd”) print(“-” * 60) x = Tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True) y = Tensor([[2.0, 0.0], [1.0, 2.0]], requires_grad=True) z = (x @ y).sum() + (x ** 2).mean() z.backward() print(f”x:n{x.numpy()}”) print(f”y:n{y.numpy()}”) print(f”z (scalar): {z.numpy()}”) print(f”∂z/∂x:n{x.grad.numpy()}”) print(f”∂z/∂y:n{y.grad.numpy()}”) We set up Tinygrad in our Colab environment and immediately begin experimenting with tensors and automatic differentiation. We create a small computation graph and observe how gradients flow through matrix operations. As we print the outputs, we gain an intuitive understanding of how Tinygrad handles backpropagation under the hood. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“nn PART 2: Building Custom Layers”) print(“-” * 60) class MultiHeadAttention: def __init__(self, dim, num_heads): self.num_heads = num_heads self.dim = dim self.head_dim = dim // num_heads self.qkv = Tensor.glorot_uniform(dim, 3 * dim) self.out = Tensor.glorot_uniform(dim, dim) def __call__(self, x): B, T, C = x.shape[0], x.shape[1], x.shape[2] qkv = x.reshape(B * T, C).dot(self.qkv).reshape(B, T, 3, self.num_heads, self.head_dim) q, k, v = qkv[:, :, 0], qkv[:, :, 1], qkv[:, :, 2] scale = (self.head_dim ** -0.5) attn = (q @ k.transpose(-2, -1)) * scale attn = attn.softmax(axis=-1) out = (attn @ v).transpose(1, 2).reshape(B, T, C) return out.reshape(B * T, C).dot(self.out).reshape(B, T, C) class TransformerBlock: def __init__(self, dim, num_heads): self.attn = MultiHeadAttention(dim, num_heads) self.ff1 = Tensor.glorot_uniform(dim, 4 * dim) self.ff2 = Tensor.glorot_uniform(4 * dim, dim) self.ln1_w = Tensor.ones(dim) self.ln2_w = Tensor.ones(dim) def __call__(self, x): x = x + self.attn(self._layernorm(x, self.ln1_w)) ff = x.reshape(-1, x.shape[-1]) ff = ff.dot(self.ff1).gelu().dot(self.ff2) x = x + ff.reshape(x.shape) return self._layernorm(x, self.ln2_w) def _layernorm(self, x, w): mean = x.mean(axis=-1, keepdim=True) var = ((x – mean) ** 2).mean(axis=-1, keepdim=True) return w * (x – mean) / (var + 1e-5).sqrt() We design our own multi-head attention module and a transformer block entirely from scratch. We implement the projections, attention scores, softmax, feedforward layers, and layer normalization manually. As we run this code, we see how each component contributes to a transformer layer’s overall behavior. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n PART 3: Mini-GPT Architecture”) print(“-” * 60) class MiniGPT: def __init__(self, vocab_size=256, dim=128, num_heads=4, num_layers=2, max_len=32): self.vocab_size = vocab_size self.dim = dim self.tok_emb = Tensor.glorot_uniform(vocab_size, dim) self.pos_emb = Tensor.glorot_uniform(max_len, dim) self.blocks = [TransformerBlock(dim, num_heads) for _ in range(num_layers)] self.ln_f = Tensor.ones(dim) self.head = Tensor.glorot_uniform(dim, vocab_size) def __call__(self, idx): B, T = idx.shape[0], idx.shape[1] tok_emb = self.tok_emb[idx.flatten()].reshape(B, T, self.dim) pos_emb = self.pos_emb[:T].reshape(1, T, self.dim) x = tok_emb + pos_emb for block in self.blocks: x = block(x) mean = x.mean(axis=-1, keepdim=True) var = ((x – mean) ** 2).mean(axis=-1, keepdim=True) x = self.ln_f * (x – mean) / (var + 1e-5).sqrt() return x.reshape(B * T, self.dim).dot(self.head).reshape(B, T, self.vocab_size) def get_params(self): params = [self.tok_emb, self.pos_emb, self.ln_f, self.head] for block in self.blocks: params.extend([block.attn.qkv, block.attn.out, block.ff1, block.ff2, block.ln1_w, block.ln2_w]) return params model = MiniGPT(vocab_size=256, dim=64, num_heads=4, num_layers=2, max_len=16) params = model.get_params() total_params = sum(p.numel() for p in params) print(f”Model initialized with {total_params:,} parameters”) We assemble the full MiniGPT architecture using the components built earlier. We embed tokens, add positional information, stack multiple transformer blocks, and project the final outputs back to vocab logits. As we initialize the model, we begin to appreciate how a compact transformer can be built with surprisingly few moving parts. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“nn PART 4: Training Loop”) print(“-” * 60) def gen_data(batch_size, seq_len): x = np.random.randint(0, 256, (batch_size, seq_len)) y = np.roll(x, 1, axis=1) y[:, 0] = x[:, 0] return Tensor(x, dtype=’int32′), Tensor(y, dtype=’int32′) optimizer = optim.Adam(params, lr=0.001) losses = [] print(“Training to predict previous token in sequence…”) with Tensor.train(): for step in range(20): start = time.time() x_batch, y_batch = gen_data(batch_size=16, seq_len=16) logits = model(x_batch) B, T, V = logits.shape[0], logits.shape[1], logits.shape[2] loss = logits.reshape(B * T, V).sparse_categorical_crossentropy(y_batch.reshape(B * T)) optimizer.zero_grad() loss.backward() optimizer.step() losses.append(loss.numpy()) elapsed = time.time() – start if step % 5 == 0: print(f”Step {step:3d} | Loss: {loss.numpy():.4f} | Time: {elapsed*1000:.1f}ms”) print(“nn PART 5: Lazy Evaluation & Kernel Fusion”) print(“-” * 60) N = 512 a = Tensor.randn(N, N) b = Tensor.randn(N, N) print(“Creating computation: (A @ B.T + A).sum()”) lazy_result = (a @ b.T + a).sum() print(“→ No computation done yet (lazy evaluation)”) print(“nCalling .realize() to execute…”) start = time.time() realized = lazy_result.realize() elapsed = time.time() – start print(f”✓ Computed in {elapsed*1000:.2f}ms”) print(f”Result: {realized.numpy():.4f}”) print(“nNote: Operations were fused into optimized kernels!”) We train the MiniGPT model on simple synthetic data and observe the loss decreasing across steps. We also explore Tinygrad’s lazy execution model by creating a fused kernel that executes only when it is realized. As we monitor timings, we understand how kernel fusion improves performance. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“nn PART 6: Custom Operations”) print(“-” * 60) def custom_activation(x): return x * x.sigmoid() x = Tensor([[-2.0, -1.0, 0.0, 1.0, 2.0]], requires_grad=True) y = custom_activation(x) loss = y.sum() loss.backward() print(f”Input: {x.numpy()}”) print(f”Swish(x): {y.numpy()}”) print(f”Gradient: {x.grad.numpy()}”) print(“nn” + “=” * 60) print(” Tutorial Complete!”) print(“=” * 60) print(“”” Key Concepts Covered: 1. Tensor operations with automatic differentiation 2. Custom neural network layers (Attention, Transformer) 3. Building a mini-GPT language model from scratch 4. Training loop with Adam optimizer 5. Lazy evaluation and kernel fusion 6. Custom activation functions “””) We implement a custom activation function and verify that gradients propagate correctly through it. We then print a summary

arXiv:2511.18619v1 Announce Type: new Abstract: Language Models are extremely susceptible to performance collapse with even small changes to input prompt strings. Libraries such as DSpy (from Stanford NLP) avoid this problem through demonstration-based prompt optimisation. Inspired by this, I propose an alternative approach that treats prompt optimisation as a classical state-space search problem. I model the prompt space as a graph where nodes represent prompt states and edges correspond to deliberate transformations such as shortening, adding examples, or re- ordering content. Using beam search and random walk algorithms, I systematically explore this space, evaluating candidates on development sets and pruning unpromising branches. Across five NLP tasks (sentiment classification, question answering, summarisation, reason- ing, and natural language inference), I find that even shallow search configurations (beam width=2, depth=2) improve upon seed prompts on development sets. For instance, beam search achieves development accuracy gains from 0.40 to 0.80 on reasoning tasks, though test set improvements are more modest (0.20 to 0.50), indicating overfitting to the develop- ment heuristic. Analysis of successful optimisation paths reveals that transformations that make prompts concise appear most frequently, while verbosity operators are never selected. My results validate prompt optimization as a search problem and suggest that with greater computational resources and improved evaluation metrics, deeper exploration could yield more robust prompts that generalize beyond development sets. Code and implementation are available at [https://github.com/MaanasTaneja/PromptOptimiser].

arXiv:2505.12842v4 Announce Type: replace-cross Abstract: Graphical user interface (GUI) agents have recently emerged as an intriguing paradigm for human-computer interaction, capable of automatically executing user instructions to operate intelligent terminal devices. However, when encountering out-of-distribution (OOD) instructions that violate environmental constraints or exceed the current capabilities of agents, GUI agents may suffer task breakdowns or even pose security threats. Therefore, effective OOD detection for GUI agents is essential. Traditional OOD detection methods perform suboptimally in this domain due to the complex embedding space and evolving GUI environments. In this work, we observe that the in-distribution input semantic space of GUI agents exhibits a clustering pattern with respect to the distance from the centroid. Based on the finding, we propose GEM, a novel method based on fitting a Gaussian mixture model over input embedding distances extracted from the GUI agent that reflect its capability boundary. Evaluated on eight datasets spanning smartphones, computers, and web browsers, our method achieves an average accuracy improvement of 23.70% over the best-performing baseline while only increasing training time by 4.9% and testing time by 6.5%. We also experimentally demonstrate that GEM can improve the step-wise success rate by 9.40% by requesting assistance from the cloud model when encountering OOD samples. Analysis verifies the generalization ability of our method through experiments on nine different backbones. The codes are available at https://github.com/Wuzheng02/GEM-OODforGUIagents.

arXiv:2511.18622v1 Announce Type: new Abstract: We present OpenGloss, a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. OpenGloss contains 537K senses across 150K lexemes, on par with WordNet 3.1 and Open English WordNet, while providing more than four times as many sense definitions. These lexemes include 9.1M semantic edges, 1M usage examples, 3M collocations, and 60M words of encyclopedic content. Generated through a multi-agent procedural generation pipeline with schema-validated LLM outputs and automated quality assurance, the entire resource was produced in under one week for under $1,000. This demonstrates that structured generation can create comprehensive lexical resources at cost and time scales impractical for manual curation, enabling rapid iteration as foundation models improve. The resource addresses gaps in pedagogical applications by providing integrated content — definitions, examples, collocations, encyclopedias, etymology — that supports both vocabulary learning and natural language processing tasks. As a synthetically generated resource, OpenGloss reflects both the capabilities and limitations of current foundation models. The dataset is publicly available on Hugging Face under CC-BY 4.0, enabling researchers and educators to build upon and adapt this resource.