Uncategorized

Every few centuries, changes in how information moves reshape how societies govern themselves. The printing press spread vernacular literacy, helping give rise to the Reformation and, eventually, representative government. The telegraph made it possible to administer vast nations like the US, accelerating the growth of the modern bureaucratic state. Broadcast media created shared national audiences, which in turn fueled mass democracy. We are now in the early stages of another such shift. Faster than many realize, AI is becoming the primary interface through which we form beliefs and participate in democratic self-governance. If left unchecked, this shift could further strain America’s already fragile institutions. But it could also help address long-standing problems, like lagging civic engagement and deepening polarization. What happens next depends on design choices that are already being made, whether we know it or not. Start with what might be called the epistemic layer—how we come to know things. People are increasingly relying on AI to know what is true, what is happening, and whom to trust. Search is already substantially AI-mediated. The next generation of AI assistants will synthesize information, frame it, and present it with authority. For a growing number of people, asking an AI will become the default way to form views on a candidate, a policy, or a public figure. Whoever controls what these models say therefore has increasing influence over what people believe.  Technology has always shaped the way citizens interact with information. But a new problem will soon arise in the form of personal AI agents, which can change not only how people receive information but how they act on it. These systems will conduct research, draft communications, highlight causes, and lobby on a user’s behalf. They will inform decisions such as how to vote on a ballot measure, which organizations are worth supporting, or how to respond to a government notice. They will, in a meaningful sense, begin to mediate the relationship between individuals and the institutions that govern them. We’ve already seen with social media what happens when algorithms optimize for engagement over understanding. Platforms do not need to have an explicit political agenda to produce polarization and radicalization. An agent that knows your preferences and your anxieties—one shaped to keep you engaged—poses the same risks. And in this case the risks may be even more difficult to detect, because an agent presents itself as your advocate. It speaks for you, acts on your behalf, and may earn trust precisely through that intimacy. Now zoom out to the collective. AI agents and humans could soon participate in the same forums, where it may be impossible to tell them apart. Even if every individual AI agent were well-designed and aligned with its user’s interests, the interactions of millions of agents could produce outcomes that no individual wanted or chose. For example, research shows that agents displaying no individual bias can still generate collective biases at scale. And setting aside what agents do to each other, there is what they do for their users. A public sphere in which everyone has a personalized agent attuned to their existing views is not, in aggregate, a public sphere at all. It is a collection of private worlds, each internally coherent but collectively inhospitable to the kind of shared deliberation that democracy requires. Taken together, these three transformations—in how we know, how we act, and how we engage in collective governance—amount to a fundamental change in the texture of citizenship. In the near future, people will form their political views through AI filters, exercise their civic agency through AI agents, and participate in institutions and public discussions that are themselves shaped by the interactions of millions of such agents. Today’s democracy is not ready for this. Our institutions were designed for a world in which power was exercised visibly, information traveled slowly enough to be contested, and reality felt more shared, if imperfectly. All of this was already fraying long before generative AI arrived. And yet this need not be a story of decline. Avoiding that outcome requires us to design for something better. On the informational layer, AI companies must ramp up existing efforts to ensure that models’ outputs are truthful. They should also explore some promising early findings that AI models can help reduce polarization. A recent field evaluation of AI-generated fact checks on X found that people with a variety of political viewpoints deemed AI-written notes more helpful than human-written ones. The paper is yet to be peer-reviewed, but that is a potentially revolutionary finding: AI-assisted fact-checking may be able to achieve the kind of cross-partisan credibility that has eluded most manual human efforts. Greater understanding of and transparency about how models make these assertions and prioritize sources in the process could help build further public trust. On the agentic layer, we need ways to evaluate whether AI agents faithfully represent their users. An agent must never have an agenda of its own or misrepresent its user’s views—a technically daunting requirement in domains where users may have not explicitly stated any preferences. But faithful representation also cannot become an accessory to motivated reasoning. An agent that refuses to present uncomfortable information, that shields its user from ever questioning prior beliefs or fails to adjust to a change of heart, is not acting in the person’s best interest. Finally, on the institutional level, policymakers should hurry to harness AI’s potential to make governance more responsive and legitimate. Several states and localities are already using AI-mediated platforms to conduct democratic deliberation at scale, building on research showing that AI mediators can help citizens find common ground. As agents become increasingly common participants in public input processes—and there is already evidence that bots are skewing those processes—identity verification for both humans and their agentic proxies must be built in from the start. What is needed is a new generation of democratic infrastructure, technological and institutional, built for the world that is actually here. Failing to design for democratic outcomes, in a domain this consequential,

If you’ve ever built a production AI pipeline that runs long jobs — processing thousands of prompts overnight, kicking off a Deep Research agent, or generating a long video — you’ve almost certainly dealt with the polling problem. Your code sits in a loop, firing GET requests every few seconds asking, “Is the job done yet?” It’s wasteful, it adds latency, and at scale it becomes a reliability headache. Google just shipped the fix. Google introduced event-driven Webhooks for the Gemini API — a push-based notification system that eliminates the need for inefficient polling. The feature is available now for all developers using the Gemini API and targets a core pain point in agentic and high-volume AI workflows. Why Polling Breaks Down at Scale To understand the problem, it helps to know what Long-Running Operation (LRO) is. Webhooks allow the Gemini API to push real-time notifications to your server when asynchronous or Long-Running Operations complete, replacing the need to poll the API for status updates and reducing latency and overhead. Before webhooks, the only option was continuous polling — repeatedly calling GET /operations to check if a job had finished. As Gemini shifts toward agentic workflows and high-volume processing — like Deep Research, long video generation, or processing thousands of prompts via the Batch API — operations can take minutes or even hours. Polling for hours is expensive in both compute and API quota, and it introduces unnecessary delays between when a job completes and when your application learns about it. The fix is conceptually simple: instead of your code asking “are you done?” repeatedly, the Gemini API calls your server the moment a task finishes, by pushing a real-time HTTP POST payload to your endpoint the instant a task completes. Two Configuration Modes: Static and Dynamic The Gemini API supports two ways to configure webhooks. Static webhooks are project-level endpoints configured with the WebhookService API and are suited for global integrations like notifying Slack or syncing a database — they are registered once per project and trigger for any matching event. Dynamic webhooks are request-level overrides that pass a webhook URL in the webhook_config payload of a specific job call, making them ideal for routing specific jobs to dedicated endpoints, for example in agent-orchestration queues. You can think of static webhooks like a standing instruction to your mail carrier: “Always deliver packages to the front desk.” Dynamic webhooks are more like saying: “For this one shipment, send it to my home address.” An additional feature of dynamic webhooks is the user_metadata field, which lets you attach arbitrary key-value metadata to a job at dispatch time — for example, {“job_group”: “nightly-eval”, “priority”: “high”}. This metadata travels with the job notification and is particularly useful when you need to fan out different job types to different downstream processors without building a separate tracking layer. Security Architecture: Standard Webhooks, HMAC, and JWKS Security is where this implementation gets technically interesting. Google’s implementation strictly adheres to the Standard Webhooks specification. Every request is signed using webhook-signature, webhook-id, and webhook-timestamp headers, ensuring idempotency and preventing replay attacks. For static webhooks, the signing is done with HMAC (Hash-based Message Authentication Code) using a symmetric shared secret, which is provided once at creation time and must be stored securely in your environment variables — the API returns this signing secret only once and it cannot be retrieved again. If you lose it, you have to rotate it. The rotation endpoint supports a revocation_behavior parameter — specifically REVOKE_PREVIOUS_SECRETS_AFTER_H24, which keeps the old secret valid for a 24-hour grace period so you can safely transition production systems, or an immediate revocation option for incident response. For dynamic webhooks, Google uses asymmetric public-key JWKS (JSON Web Key Set) signatures instead of symmetric secrets. Dynamic webhook requests emit a JSON Web Token (JWT) signature, and your listener must extract and verify it using Google’s public certificate endpoints at https://generativelanguage.googleapis.com/.well-known/jwks.json. The RS256 algorithm is used for this verification. This means your server never blindly trusts incoming requests — every webhook hit can be cryptographically verified before you act on it. The webhook-timestamp header is particularly important: best practices call for always validating this timestamp and rejecting payloads older than five minutes to mitigate replay attacks. Thin Payloads and the Event Catalog One architectural decision worth noting is the thin payload model. To avoid bandwidth congestion, Gemini webhooks deliver a snapshot containing status details and pointers to results, rather than the raw output file itself. The exact fields in that snapshot depend on the event type. For batch jobs, a completed notification carries the job id and an output_file_uri pointing to your results — for example, a Cloud Storage path like gs://my-bucket/results.jsonl. For video generation, the video.generated event delivers a different set of fields: file_id and video_uri. Your server-side handler needs to branch on event type before reading the payload data fields. The full event catalog covers three categories: batch jobs (batch.succeeded, batch.cancelled, batch.expired, batch.failed), Interactions API operations (interaction.requires_action, interaction.completed, interaction.failed, interaction.cancelled), and video generation (video.generated). For developers writing code: the official code samples in Google’s documentation subscribe to and handle batch.completed rather than batch.succeeded — both appear across the documentation, so match whichever your implementation uses. The Interactions API, for readers unfamiliar with it, is Gemini’s API for async multi-turn agent conversations. The interaction.requires_action event is particularly useful — it fires when a function call is pending and your application needs to step in and take an action before the agent can continue. Delivery Guarantees and Best Practices Google guarantees “at-least-once” delivery with automatic retries for up to 24 hours using exponential backoff. The “at-least-once” guarantee means your endpoint could occasionally receive the same event more than once under high-congestion conditions. The consistent webhook-id header should be used to deduplicate these. Your server should also respond with a 2xx status code immediately upon valid signature detection and queue any heavier parsing internally — prolonged listener hold times trigger the retry cycle, which is the opposite of what you want. Key

In this tutorial, we walk through a complete, end-to-end workflow for correcting bias in survey data using the balance library. We simulate a realistic population, deliberately introduce sampling bias, and then apply multiple re-weighting techniques to recover unbiased estimates. We focus on four widely used methods: Inverse Probability Weighting (IPW), Covariate Balancing Propensity Scores (CBPS), ranking, and post-stratification, and evaluate how effectively each method restores balance between the sample and the target population. Throughout the process, we analyze diagnostics such as ASMD, outcome estimates, and design effects to build a strong intuitive and practical understanding of survey weighting. Copy CodeCopiedUse a different Browser import subprocess, sys subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, “balance”]) import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import warnings warnings.filterwarnings(“ignore”) from balance import Sample np.random.seed(2024) sns.set_theme(style=”whitegrid”, context=”notebook”) We begin by installing the balance package and importing all the required libraries for data manipulation and visualization. We set a random seed to ensure reproducibility and configure plotting aesthetics for clearer diagnostics. This setup prepares a clean, consistent environment for running the full reweighting workflow. Copy CodeCopiedUse a different Browser def simulate_population(n=50_000): age = np.clip(np.random.normal(45, 17, n), 18, 90).astype(int) gender = np.random.choice([“M”, “F”], size=n, p=[0.49, 0.51]) education = np.random.choice( [“HS”, “SomeCollege”, “Bachelor”, “Graduate”], size=n, p=[0.35, 0.25, 0.25, 0.15], ) income = np.exp(np.random.normal(10.5, 0.5, n)) region = np.random.choice( [“Urban”, “Suburban”, “Rural”], size=n, p=[0.40, 0.35, 0.25] ) happiness = ( 50 + 0.20 * (age – 45) + (education == “Graduate”) * 8 + (education == “Bachelor”) * 4 + (region == “Urban”) * 3 + np.log(income) * 2 + np.random.normal(0, 5, n) ) return pd.DataFrame({ “id”: np.arange(n).astype(str), “age”: age, “gender”: gender, “education”: education, “income”: income.round(2), “region”: region, “happiness”: happiness.round(2), }) def biased_sample(pop, n=2_000): score = ( -0.04 * (pop[“age”] – 30) + (pop[“education”] == “Graduate”) * 1.0 + (pop[“education”] == “Bachelor”) * 0.6 + (pop[“region”] == “Urban”) * 0.7 – (pop[“region”] == “Rural”) * 0.5 ) p = 1 / (1 + np.exp(-score)) p = p / p.sum() idx = np.random.choice(pop.index, size=n, replace=False, p=p) return pop.loc[idx].reset_index(drop=True) target_df = simulate_population(50_000) sample_df = biased_sample(target_df, 2_000) target_for_balance = target_df.drop(columns=[“happiness”]) print(f”Sample size : {len(sample_df):,}”) print(f”Target size : {len(target_for_balance):,}”) print(f”nTRUE population mean happiness : {target_df[‘happiness’].mean():.2f}”) print(f”Naive sample mean happiness : {sample_df[‘happiness’].mean():.2f} <– biased!”) We simulate a realistic population dataset with demographic and socioeconomic features along with an outcome variable. We then introduce sampling bias by preferentially selecting younger, more educated, and urban individuals to mimic real-world survey bias. Finally, we compare the naive sample mean to the true population mean to highlight bias. Copy CodeCopiedUse a different Browser sample = Sample.from_frame( sample_df, id_column=”id”, outcome_columns=[“happiness”] ) target = Sample.from_frame(target_for_balance, id_column=”id”) sample_with_target = sample.set_target(target) print(“n— Sample object —“) print(sample_with_target) print(“n” + “=” * 60) print(” PRE-ADJUSTMENT DIAGNOSTICS”) print(“=” * 60) asmd_before = sample_with_target.covars().asmd() print(“nASMD (Absolute Standardized Mean Difference) — lower = better balance”) print(“Rule of thumb: |ASMD| > 0.10 indicates meaningful imbalance.”) print(asmd_before.T.round(3)) print(“nMean of covariates (sample vs target):”) print(sample_with_target.covars().mean().T.round(3)) We convert both the biased sample and the target population into structured Sample objects for processing. We compute pre-adjustment diagnostics, such as ASMD and covariate means, to quantify imbalance between the sample and the target. This step helps us clearly understand how far the sample deviates before applying any correction. Copy CodeCopiedUse a different Browser print(“n” + “=” * 60) print(” FITTING WEIGHTS — 4 METHODS”) print(“=” * 60) print(“n>>> [1/4] IPW with LASSO logistic regression”) adjusted_ipw = sample_with_target.adjust(method=”ipw”) print(adjusted_ipw.summary()) print(“n>>> [2/4] CBPS — Covariate Balancing Propensity Score”) try: adjusted_cbps = sample_with_target.adjust(method=”cbps”) print(adjusted_cbps.summary()) except Exception as e: print(“CBPS failed (skipping):”, e) adjusted_cbps = None print(“n>>> [3/4] Raking (iterative proportional fitting)”) adjusted_rake = sample_with_target.adjust(method=”rake”) print(adjusted_rake.summary()) print(“n>>> [4/4] Post-stratification (categoricals only)”) cat_cols = [“id”, “gender”, “education”, “region”] sample_cat = Sample.from_frame( sample_df[cat_cols + [“happiness”]], id_column=”id”, outcome_columns=[“happiness”], ) target_cat = Sample.from_frame(target_for_balance[cat_cols], id_column=”id”) adjusted_post = sample_cat.set_target(target_cat).adjust(method=”poststratify”) print(adjusted_post.summary()) print(“n” + “=” * 60) print(” METHOD COMPARISON”) print(“=” * 60) methods = { “IPW”: adjusted_ipw, “CBPS”: adjusted_cbps, “Rake”: adjusted_rake, “PostStrat”: adjusted_post, } def safe_mean_asmd(asmd_df, prefer=”self”): “””Mean ASMD across covariates from a balance asmd DataFrame.””” row = prefer if prefer in asmd_df.index else asmd_df.index[0] if “mean(asmd)” in asmd_df.columns: return float(asmd_df.loc[row, “mean(asmd)”]) return float(asmd_df.loc[row].mean()) asmd_means = {“Unadjusted”: safe_mean_asmd(asmd_before)} outcome_means = {“Naive sample”: float(sample_df[“happiness”].mean())} deff_vals = {} for name, m in methods.items(): if m is None: continue asmd_means[name] = safe_mean_asmd(m.covars().asmd(), prefer=”self”) outcome_means[name] = float(m.outcomes().mean()[“happiness”].iloc[0]) w = m.to_df()[“weight”].values deff_vals[name] = (w.sum() ** 2) / (len(w) * np.sum(w ** 2)) outcome_means[“TRUE pop”] = float(target_df[“happiness”].mean()) print(“nMean ASMD across covariates (lower = better balance):”) for k, v in asmd_means.items(): print(f” {k:14s}: {v:.4f}”) print(“nWeighted estimate of mean happiness:”) for k, v in outcome_means.items(): print(f” {k:14s}: {v:.3f}”) print(“nKish’s effective sample-size ratio (1.0 = no info loss):”) for k, v in deff_vals.items(): print(f” {k:14s}: {v:.3f} (n_eff ≈ {int(v * len(sample_df))})”) We apply four different weighting methods, IPW, CBPS, ranking, and post-stratification, to adjust the biased sample. We evaluate each method using balance metrics, outcome estimates, and calculations of effective sample size. This comparison allows us to understand how different techniques trade off bias reduction and variance. Copy CodeCopiedUse a different Browser fig, axes = plt.subplots(2, 2, figsize=(14, 10)) colors_a = [“gray”, “#1f77b4”, “#ff7f0e”, “#2ca02c”, “#d62728″][: len(asmd_means)] axes[0, 0].bar(list(asmd_means.keys()), list(asmd_means.values()), color=colors_a) axes[0, 0].axhline(0.1, ls=”–“, color=”red”, label=”0.10 imbalance threshold”) axes[0, 0].set_title(“Mean ASMD across covariates”) axes[0, 0].set_ylabel(“Mean ASMD”); axes[0, 0].legend() axes[0, 0].tick_params(axis=”x”, rotation=20) truth = target_df[“happiness”].mean() colors_b = [“#888”] + [“#1f77b4”, “#ff7f0e”, “#2ca02c”, “#d62728”][: len(methods)] + [“black”] axes[0, 1].bar(list(outcome_means.keys()), list(outcome_means.values()), color=colors_b[: len(outcome_means)]) axes[0, 1].axhline(truth, ls=”–“, color=”black”, label=f”truth = {truth:.2f}”) axes[0, 1].set_title(“Estimated mean happiness vs ground truth”) axes[0, 1].set_ylabel(“Mean happiness”); axes[0, 1].legend() axes[0, 1].tick_params(axis=”x”, rotation=20) w_ipw = adjusted_ipw.to_df()[“weight”].values axes[1, 0].hist(w_ipw, bins=40, color=”steelblue”, edgecolor=”white”) axes[1, 0].set_title( f”IPW weight distributionn” f”min={w_ipw.min():.2f} median={np.median(w_ipw):.2f} max={w_ipw.max():.2f}” ) axes[1, 0].set_xlabel(“weight”); axes[1, 0].set_ylabel(“count”) ages = sample_df[“age”].values bins = np.linspace(18, 90, 31) axes[1, 1].hist(target_df[“age”], bins=bins, density=True, alpha=0.45, color=”green”, label=”Target (truth)”) axes[1, 1].hist(ages, bins=bins, density=True, alpha=0.45, color=”red”, label=”Sample (biased)”) axes[1, 1].hist(ages, bins=bins, density=True, alpha=0.45, color=”blue”, weights=w_ipw, label=”Sample (IPW-weighted)”) axes[1, 1].set_title(“Age distribution: bias correction by IPW”) axes[1, 1].set_xlabel(“Age”); axes[1, 1].set_ylabel(“density”); axes[1, 1].legend() plt.tight_layout() plt.savefig(“balance_diagnostics.png”, dpi=110, bbox_inches=”tight”) plt.show() print(“n” + “=” * 60) print(” ADVANCED — controlling variance with max_de”)

arXiv:2605.00025v1 Announce Type: cross Abstract: Speech neuroprosthesis systems decode intended speech from neural activity in the absence of audible output, offering a path to restoring communication for individuals with speech-impairing conditions. Current approaches decode predominantly from motor cortical areas, discarding others — such as area 44, part of Broca’s area — that may encode complementary linguistic information. We introduce MoDAl (Modality Decorrelation and Alignment), a framework that discovers complementary neural modalities through the interplay of two objectives in a shared projection space. A contrastive loss aligns each of several parallel brain encoders with the text embeddings of a pretrained large language model (LLM), while a decorrelation loss prevents the encoders from coalescing to duplicative representations. We prove that these objectives are in productive tension: Contrastive alignment induces transitive modality coalescence, which decorrelation must counteract for the framework to discover diverse neurolinguistic modalities. On the Brain-to-Text Benchmark ’24, MoDAl reduces word error rate (WER) from 26.3% to 21.6% compared to the previous best end-to-end method, with the gain from incorporating previously discarded area 44 signals arising entirely from the decorrelation mechanism. Analysis of the discovered modalities reveals functional specialization: Encoders receiving area 44 input capture structural and syntactic properties (sentence length, grammatical voice, wh-words), consistent with the neurolinguistic understanding of Broca’s area.

arXiv:2605.00318v1 Announce Type: new Abstract: Tabular documents such as CSV and Excel files are widely used in enterprise data pipelines, yet existing chunking strategies for retrieval-augmented generation (RAG) are primarily designed for unstructured text and do not account for tabular structure. We propose a structure-aware tabular chunking (STC) framework that operates on row-level units by constructing a hierarchical Row Tree representation, where each row is encoded as a key-value block. STC performs token-constrained splitting aligned with structural boundaries and applies overlap-free greedy merging to produce dense, non-overlapping chunks. This design preserves semantic relationships between fields within a row while improving token utilization and reducing fragmentation. Across evaluations on the MAUD dataset, STC reduces chunk count by up to 40% and 56% compared to standard recursive and key-value based baselines, respectively, while improving token utilization and processing efficiency. In retrieval benchmarks, STC improves MRR from 0.3576 to 0.5945 in a hybrid setting and increases Recall@1 from 0.366 to 0.754 in BM25-only retrieval. These results demonstrate that preserving structure during chunking improves retrieval performance, highlighting the importance of structure-aware chunking for RAG over tabular data.

arXiv:2605.00294v1 Announce Type: new Abstract: This study presents a systematic approach to identifying and characterizing student misconceptions in online learning environments through a novel combination of quantitative performance analysis and large language model (LLM) assessment. We analyzed data from 9 course periods across 5 online biomedical science courses, encompassing 3,802 medical student enrollments. Using data from 40-50 topic-focused quizzes per course, we developed a two-stage methodology. First, we identified challenging central topics using quiz-level performance metrics. Second, we employed LLMs to characterize the underlying misconceptions in these high-priority areas. By examining student performance on first attempts across primarily multiple-choice questions (MCQs), we identified consistently challenging topics that were also central to course objectives. We then leveraged recent advances in generative AI to analyze three distinct data sources in combination: quiz question content, student response patterns, and lecture transcripts. This approach revealed actionable insights about student misconceptions that were not apparent from performance data alone. The quality of the LLM-identified misconceptions was rated as excellent by subject matter experts. We also conducted teacher interviews to assess the perceived utility of our topic identification method. Faculty found that data-driven identification of challenging topics was valuable and corroborated their own classroom observations. This methodology provides a scalable approach to characterizing student difficulties in learning environments where quizzes are used. Our findings demonstrate the potential for targeted and potentially personalized interventions in future course iterations, with clear pathways for measuring intervention effectiveness through follow-up quiz performance.