Uncategorized

arXiv:2511.04583v3 Announce Type: replace-cross Abstract: Understanding the current capabilities and risks of AI Scientist systems is essential for ensuring trustworthy and sustainable AI-driven scientific progress while preserving the integrity of the academic ecosystem. To this end, we develop Jr. AI Scientist, a state-of-the-art autonomous AI scientist system that mimics the core research workflow of a novice student researcher: Given the baseline paper from the human mentor, it analyzes its limitations, formulates novel hypotheses for improvement, validates them through rigorous experimentation, and writes a paper with the results. Unlike previous approaches that assume full automation or operate on small-scale code, Jr. AI Scientist follows a well-defined research workflow and leverages modern coding agents to handle complex, multi-file implementations, leading to scientifically valuable contributions. Through our experiments, the Jr. AI Scientist successfully generated new research papers that build upon real NeurIPS, IJCV, and ICLR works by proposing and implementing novel algorithms. For evaluation, we conducted automated assessments using AI Reviewers, author-led evaluations, and submissions to Agents4Science, a venue dedicated to AI-driven scientific contributions. The findings demonstrate that Jr. AI Scientist generates papers receiving higher review scores by DeepReviewer than existing fully automated systems. Nevertheless, we identify important limitations from both the author evaluation and the Agents4Science reviews, indicating the potential risks of directly applying current AI Scientist systems and key challenges for future research. Finally, we comprehensively report various risks identified during development. We believe this study clarifies the current role and limitations of AI Scientist systems, offering insights into the areas that still require human expertise and the risks that may emerge as these systems evolve.

Large Language Models (LLMs) are the world’s best mimics, but when it comes to the cold, hard logic of updating beliefs based on new evidence, they are surprisingly stubborn. A team of researchers from Google argue that the current crop of AI agents falls far short of ‘probabilistic reasoning’—the ability to maintain and update a ‘world model’ as new information trickles in. The solution? Stop trying to give them the right answers and start teaching them how to guess like a mathematician. The Problem: The ‘One-and-Done’ Plateau While LLMs like Gemini-1.5 Pro and GPT-4.1 Mini can write code or summarize emails, they struggle as interactive agents. Imagine a flight booking assistant: it needs to infer your preferences (price vs. duration) by watching which flights you pick over several rounds. The research team found that off-the-shelf LLMs—including heavyweights like Llama-3-70B and Qwen-2.5-32B—showed ‘little or no improvement’ after the first round of interaction. While a ‘Bayesian Assistant’ (a symbolic model using Bayes’ rule) gets more accurate with every data point, standard LLMs plateaued almost immediately, failing to adapt their internal ‘beliefs’ to the user’s specific reward function. Meet Bayesian Teaching The research team introduced a technique called Bayesian Teaching. Instead of fine-tuning a model on ‘correct’ data (what they call an Oracle Teacher), they fine-tuned it to mimic a Bayesian Assistant—a model that explicitly uses Bayes’ rule to update a probability distribution over possible user preferences. Here is the technical breakdown: The Task: A five-round flight recommendation interaction. Flights are defined by features like price, duration, and stops. The Reward Function: A vector representing user preferences (e.g., a strong preference for low prices). The Posterior Update: After each round, the Bayesian Assistant updates its posterior distribution based on the prior (initial assumptions) and the likelihood (the probability the user would pick a certain flight given a specific reward function). By using Supervised Fine-Tuning (SFT) on these Bayesian interactions, the research team forced the LLMs to adopt the process of reasoning under uncertainty, not just the final result. Why ‘Educated Guesses’ Beat Correct Answers The most counter-intuitive finding of the research is that Bayesian Teaching consistently outperformed Oracle Teaching. In ‘Oracle Teaching,’ the model is trained on a teacher that already knows exactly what the user wants. In ‘Bayesian Teaching,’ the teacher is often wrong in early rounds because it is still learning. However, those ‘educated guesses’ provide a much stronger learning signal. By watching the Bayesian Assistant struggle with uncertainty and then update its beliefs after receiving feedback, the LLM learns the ‘skill’ of belief updating. The results were stark: Bayesian-tuned models (like Gemma-2-9B or Llama-3-8B) were not only more accurate but agreed with the ‘gold standard’ Bayesian strategy roughly 80% of the time—significantly higher than their original versions. Generalization: Beyond Flights to Web Shopping For devs, the ‘holy grail’ is generalization. A model trained on flight data shouldn’t just be good at flights; it should understand the concept of learning from a user. The research team tested their fine-tuned models on: Increased Complexity: Moving from four flight features to eight. New Domains: Hotel recommendations. Real-World Scenarios: A web shopping task using real products (titles and descriptions) from a simulated environment. Even though the models were only fine-tuned on synthetic flight data, they successfully transferred those probabilistic reasoning skills to hotel booking and web shopping. In fact, the Bayesian LLMs even outperformed human participants in some rounds, as humans often deviate from normative reasoning standards due to biases or inattention. The Neuro-Symbolic Bridge This research highlights a unique strength of deep learning: the ability to distill a classic, symbolic model (the Bayesian Assistant) into a neural network (the LLM). While symbolic models are great for simple, codified tasks, they are notoriously difficult to build for ‘messy’ real-world domains like web shopping. By teaching the LLM to mimic the symbolic model’s strategy, it is possible to get the best of both worlds: the rigorous reasoning of a Bayesian and the flexible, natural-language understanding of a transformer. Key Takeaways LLMs Struggle with Belief Updating: Off-the-shelf LLMs, including state-of-the-art models like Gemini-1.5 Pro and GPT-4.1 Mini, fail to effectively update their beliefs as they receive new information, with performance often plateauing after a single interaction. Bayesian Teaching Outperforms Direct Training: Teaching an LLM to mimic the ‘educated guesses’ and uncertainty of a normative Bayesian model is more effective than training it directly on correct answers (oracle teaching). Probabilistic Skills Generalize Across Domains: LLMs fine-tuned on simple synthetic tasks (e.g., flight recommendations) can successfully transfer their belief-updating skills to more complex, real-world scenarios like web shopping and hotel recommendations. Neural Models Are More Robust to Human Noise: While a purely symbolic Bayesian model is optimal for consistent simulated users, fine-tuned LLMs demonstrate greater robustness when interacting with humans, whose choices often deviate from their stated preferences due to noise or bias. Effective Distillation of Symbolic Strategies: The research proves that LLMs can learn to approximate complex symbolic reasoning strategies through supervised fine-tuning, allowing them to apply these strategies in domains too messy or complex to be codified explicitly in a classic symbolic model. Check out Paper and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post The ‘Bayesian’ Upgrade: Why Google AI’s New Teaching Method is the Key to LLM Reasoning appeared first on MarkTechPost.

In this tutorial, we build a complete pipeline for single-cell RNA sequencing analysis using Scanpy. We start by installing the required libraries and loading the PBMC 3k dataset, then perform quality control, filtering, and normalization to prepare the data for downstream analysis. We then identify highly variable genes, perform PCA for dimensionality reduction, and construct a neighborhood graph to generate UMAP embeddings and Leiden clusters. Through marker gene discovery and visualization, we explore how clusters correspond to biological cell populations and implement a simple rule-based annotation strategy to infer cell types. Copy CodeCopiedUse a different Browser import sys import subprocess import importlib def pip_install(*packages): subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, *packages]) required = [ “scanpy”, “anndata”, “leidenalg”, “igraph”, “harmonypy”, “seaborn” ] pip_install(*required) import os import warnings warnings.filterwarnings(“ignore”) import numpy as np import pandas as pd import matplotlib.pyplot as plt import scanpy as sc import anndata as ad sc.settings.verbosity = 2 sc.settings.set_figure_params(dpi=110, facecolor=”white”, frameon=False) np.random.seed(42) print(“Scanpy version:”, sc.__version__) adata = sc.datasets.pbmc3k() adata.var_names_make_unique() print(“nInitial AnnData:”) print(adata) adata.layers[“counts”] = adata.X.copy() adata.var[“mt”] = adata.var_names.str.upper().str.startswith(“MT-“) sc.pp.calculate_qc_metrics(adata, qc_vars=[“mt”], percent_top=None, log1p=False, inplace=True) print(“nQC summary:”) display( adata.obs[[“n_genes_by_counts”, “total_counts”, “pct_counts_mt”]].describe().T ) We install all required dependencies and import the core scientific computing libraries needed for the analysis. We configure Scanpy settings, initialize the environment, and load the PBMC 3k single-cell RNA-seq dataset. We then compute quality-control metrics, including mitochondrial gene percentage, total counts, and the number of detected genes, for each cell. Copy CodeCopiedUse a different Browser fig, axs = plt.subplots(1, 3, figsize=(15, 4)) sc.pl.violin(adata, [“n_genes_by_counts”], jitter=0.4, ax=axs[0], show=False) sc.pl.violin(adata, [“total_counts”], jitter=0.4, ax=axs[1], show=False) sc.pl.violin(adata, [“pct_counts_mt”], jitter=0.4, ax=axs[2], show=False) plt.tight_layout() plt.show() sc.pl.scatter(adata, x=”total_counts”, y=”n_genes_by_counts”, color=”pct_counts_mt”) adata = adata[adata.obs[“n_genes_by_counts”] >= 200].copy() adata = adata[adata.obs[“n_genes_by_counts”] <= 5000].copy() adata = adata[adata.obs[“pct_counts_mt”] < 10].copy() sc.pp.filter_genes(adata, min_cells=3) print(“nAfter filtering:”) print(adata) sc.pp.normalize_total(adata, target_sum=1e4) sc.pp.log1p(adata) adata.raw = adata.copy() sc.pp.highly_variable_genes( adata, flavor=”seurat”, min_mean=0.0125, max_mean=3, min_disp=0.5 ) print(“nHighly variable genes selected:”, int(adata.var[“highly_variable”].sum())) sc.pl.highly_variable_genes(adata) adata = adata[:, adata.var[“highly_variable”]].copy() We visualize quality control metrics using plots to check the distribution of gene counts and mitochondrial content. We apply filtering steps to remove low-quality cells and genes that do not meet basic expression thresholds. We then normalize the data, apply a log transformation, and identify highly variable genes for downstream analysis. Copy CodeCopiedUse a different Browser sc.pp.regress_out(adata, [“total_counts”, “pct_counts_mt”]) sc.pp.scale(adata, max_value=10) sc.tl.pca(adata, svd_solver=”arpack”) sc.pl.pca_variance_ratio(adata, log=True) sc.pl.pca(adata, color=None) sc.pp.neighbors(adata, n_neighbors=12, n_pcs=30, metric=”euclidean”) sc.tl.umap(adata, min_dist=0.35, spread=1.0) sc.tl.leiden(adata, resolution=0.6, key_added=”leiden”) print(“nCluster counts:”) display(adata.obs[“leiden”].value_counts().sort_index().rename(“cells_per_cluster”).to_frame()) sc.pl.umap(adata, color=[“leiden”], legend_loc=”on data”, title=”PBMC 3k – Leiden clusters”) sc.tl.rank_genes_groups(adata, groupby=”leiden”, method=”wilcoxon”) sc.pl.rank_genes_groups(adata, n_genes=20, sharey=False) marker_table = sc.get.rank_genes_groups_df(adata, group=None) print(“nTop marker rows:”) display(marker_table.head(20)) We regress out technical confounders and scale the dataset to prepare it for dimensionality reduction. We perform principal component analysis to capture the dataset’s major variance structure. We then construct the neighborhood graph, compute UMAP embeddings, perform Leiden clustering, and identify marker genes for each cluster. Copy CodeCopiedUse a different Browser top_markers_per_cluster = ( marker_table.groupby(“group”) .head(10) .loc[:, [“group”, “names”, “logfoldchanges”, “pvals_adj”]] .reset_index(drop=True) ) print(“nTop 10 markers per cluster:”) display(top_markers_per_cluster) candidate_markers = [ “IL7R”, “LTB”, “MALAT1”, “CCR7”, “NKG7”, “GNLY”, “PRF1”, “MS4A1”, “CD79A”, “CD79B”, “LYZ”, “S100A8”, “FCER1A”, “CST3”, “PPBP”, “FCGR3A”, “LGALS3”, “CTSS”, “CD3D”, “TRBC1”, “TRAC” ] candidate_markers = [g for g in candidate_markers if g in adata.var_names] if candidate_markers: sc.pl.dotplot( adata, var_names=candidate_markers, groupby=”leiden”, standard_scale=”var”, dendrogram=True ) sc.pl.matrixplot( adata, var_names=candidate_markers, groupby=”leiden”, standard_scale=”var”, dendrogram=True ) cluster_marker_reference = { “T_cells”: [“IL7R”, “LTB”, “CCR7”, “CD3D”, “TRBC1”, “TRAC”], “NK_cells”: [“NKG7”, “GNLY”, “PRF1”], “B_cells”: [“MS4A1”, “CD79A”, “CD79B”], “Monocytes”: [“LYZ”, “FCGR3A”, “LGALS3”, “CTSS”, “S100A8”, “CST3”], “Dendritic_cells”: [“FCER1A”, “CST3”], “Platelets”: [“PPBP”] } We examine the most significant marker genes detected for each cluster and summarize the top markers. We visualize gene expression patterns across clusters using dot plots and matrix plots for known immune cell markers. We also define a reference mapping of marker genes associated with major immune cell types for later annotation. Copy CodeCopiedUse a different Browser available_reference = { celltype: [g for g in genes if g in adata.var_names] for celltype, genes in cluster_marker_reference.items() } available_reference = {k: v for k, v in available_reference.items() if len(v) > 0} for celltype, genes in available_reference.items(): sc.tl.score_genes(adata, gene_list=genes, score_name=f”{celltype}_score”, use_raw=False) score_cols = [f”{ct}_score” for ct in available_reference.keys()] cluster_scores = adata.obs.groupby(“leiden”)[score_cols].mean() display(cluster_scores) cluster_to_celltype = {} for cluster in cluster_scores.index: best = cluster_scores.loc[cluster].idxmax().replace(“_score”, “”) cluster_to_celltype[cluster] = best adata.obs[“cell_type”] = adata.obs[“leiden”].map(cluster_to_celltype).astype(“category”) print(“nCluster to cell-type mapping:”) display(pd.DataFrame.from_dict(cluster_to_celltype, orient=”index”, columns=[“assigned_cell_type”])) sc.pl.umap( adata, color=[“leiden”, “cell_type”], legend_loc=”on data”, wspace=0.45 ) sc.tl.rank_genes_groups(adata, groupby=”cell_type”, method=”wilcoxon”) sc.pl.rank_genes_groups(adata, n_genes=15, sharey=False) celltype_markers = sc.get.rank_genes_groups_df(adata, group=None) print(“nTop markers by annotated cell type:”) display( celltype_markers.groupby(“group”).head(8)[[“group”, “names”, “logfoldchanges”, “pvals_adj”]] ) cluster_prop = ( adata.obs[“cell_type”] .value_counts(normalize=True) .mul(100) .round(2) .rename(“percent”) .to_frame() ) print(“nCell-type proportions (%):”) display(cluster_prop) plt.figure(figsize=(7, 4)) cluster_prop[“percent”].sort_values().plot(kind=”barh”) plt.xlabel(“Percent of cells”) plt.ylabel(“Cell type”) plt.title(“Estimated cell-type composition”) plt.tight_layout() plt.show() output_dir = “scanpy_pbmc3k_outputs” os.makedirs(output_dir, exist_ok=True) adata.write(os.path.join(output_dir, “pbmc3k_scanpy_advanced.h5ad”)) marker_table.to_csv(os.path.join(output_dir, “cluster_markers.csv”), index=False) celltype_markers.to_csv(os.path.join(output_dir, “celltype_markers.csv”), index=False) cluster_scores.to_csv(os.path.join(output_dir, “cluster_score_matrix.csv”)) print(f”nSaved outputs to: {output_dir}”) print(“Files:”) for f in sorted(os.listdir(output_dir)): print(” -“, f) summary = { “n_cells_final”: int(adata.n_obs), “n_genes_final”: int(adata.n_vars), “n_clusters”: int(adata.obs[“leiden”].nunique()), “clusters”: sorted(adata.obs[“leiden”].unique().tolist()), “cell_types”: sorted(adata.obs[“cell_type”].unique().tolist()), } print(“nAnalysis summary:”) for k, v in summary.items(): print(f”{k}: {v}”) We score each cell using known marker gene sets and assign probable cell types to clusters based on expression patterns. We visualize the annotated cell types on the UMAP embedding and perform differential gene expression analysis across the predicted cell populations. Also, we compute cell-type proportions, generate summary visualizations, and save the processed dataset and analysis outputs for further research. In conclusion, we developed a full end-to-end workflow for analyzing single-cell transcriptomic data using Scanpy. We performed preprocessing, clustering, marker-gene analysis, and cell-type annotation, and visualized the data structure using UMAP and gene expression plots. By saving the processed AnnData object and analysis outputs, we created a reusable dataset for further biological interpretation and advanced modeling. This workflow demonstrates how Scanpy enables scalable, reproducible single-cell analysis through a structured, modular Python pipeline. Check out the Full Codes here. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post A Coding Guide to Build a Complete Single Cell RNA Sequencing Analysis Pipeline Using Scanpy for Clustering Visualization and Cell Type Annotation appeared first on MarkTechPost.

arXiv:2603.06492v1 Announce Type: cross Abstract: We introduce NOBLE (Nonlinear lOw-rank Branch for Linear Enhancement), an architectural augmentation that adds nonlinear low-rank branches to transformer linear layers. Unlike LoRA and other parameter-efficient fine-tuning (PEFT) methods, NOBLE is designed for pretraining from scratch. The branch is a permanent part of the architecture as opposed to an adapter for finetuning on top of frozen weights. The branch computes {sigma}(xWdown)Wup where {sigma} is a learnable nonlinearity. We evaluate several activation functions and find that CosNet, a two-layer cosine nonlinearity with learnable frequency and phase with a linear projection in between them in the bottleneck space, performs best. NOBLE achieves substantial improvements with minimal overhead: up to 1.47x step speedup to reach baseline eval loss (up to 32% fewer training steps), with as low as 4% additional parameters and 7% step time overhead, resulting in up to 1.22x net wallclock speedup. Experiments on LLMs (250M and 1.5B parameters), BERT, VQGAN, and ViT consistently show improved training efficiency. We identify one caveat: Mixup/CutMix augmentation interferes with NOBLE’s benefits in Imagenet classification along with other stochastic augmentations, but when disabled, ViT also improves. This discrepancy is possibly explained by regularization techniques that encourage smoother fits to the target function while NOBLE may specialize more in sharper aspects of the target function.

arXiv:2602.23136v2 Announce Type: replace Abstract: Numerous studies have shown that multimodal LLMs process speech and images well but fail in non-intuitive ways rendering trivial tasks such as object counting unreliable. We investigate this behavior from an information-theoretic perspective by framing multimodal LLM inference as a mismatched decoder problem: a decoder trained primarily on text can only extract information along text-aligned directions (removing up to 98% of the variation in modality-specific (non-text) directions improves decoder loss) and the amount of accessible information is bounded by the Generalized Mutual Information (GMI). We show that information loss is bounded as the distributional mismatch between the source data and the text data increases, and as the sensitivity of the decoder increases. This bound is a function of the model’s scoring rule not its architecture. We validate the predictions across five models spanning speech and vision. A controlled study (two Prismatic VLMs differing only in encoder text-alignment) shows that the bottleneck lies in the scoring rule of the decoder rather than the text-alignment of the encoder or the learned projection. A LoRA intervention demonstrates that simply training with an emotion-related objective improves emotion detection from 17.3% to 61.8% task accuracy without affecting other attributes, confirming that the training objective determines what becomes accessible.

What if the AI industry is optimizing for a goal that cannot be clearly defined or reliably measured? That is the central argument of a new paper by Yann LeCun, and his team, which claims that Artificial General Intelligence has become an overloaded term used in inconsistent ways across academia and industry. The research team argued that because AGI lacks a stable operational definition, it has become a weak scientific target for evaluating progress or guiding research. Why Human Intelligence Is Not Truly ‘General‘ The research team in the paper starts by challenging a common assumption behind many AGI discussions: that human intelligence is a meaningful template for’ ‘general’ intelligence. The research team argue that humans only appear general because we evaluate intelligence from inside the task distribution shaped by human biology and survival. We are good at the kinds of tasks that mattered for our existence, such as perception, motor control, planning, and social reasoning. But outside that range, human ability is limited, and in many cases machines already outperform us. The research paper’s point is not that humans are narrow in every sense, but that human intelligence is better understood as specialized and adaptable rather than general in any universal sense. The Problem With Human-Centered AGI Definitions That distinction matters because many AGI definitions quietly inherit a human-centered benchmark. The research team argues there is no real consensus on what AGI means across academia or industry. Some definitions focus on doing everything a human can do. Others focus on economic usefulness, broad task competence, open-ended reasoning, or the ability to learn. These are not equivalent definitions, and they do not produce one clean evaluation target. The research team therefore argue that existing AGI definitions are insufficient because they are often ambiguous, difficult to assess, or not truly general once examined closely. The Shift From AGI to SAI The research paper’s alternative is Superhuman Adaptable Intelligence, or SAI. It defines SAI as intelligence that can adapt to exceed humans at any task humans can do, while also adapting to useful tasks outside the human domain. That is a subtle but important shift. Instead of asking whether a system already matches humans across a fixed checklist of tasks, the research team asks how quickly the system can learn something new and how broadly it can continue adapting. In this framework, the key metric is adaptation speed: the speed with which an agent acquires new skills and learns new tasks. Why Adaptation Speed Matters More Than Static Benchmarks This reframes the problem in a more engineering-friendly way. A benchmark based on a growing catalog of tasks becomes messy fast; the space of possible skills is effectively unbounded. The research team argued that evaluating intelligence as a static inventory of competencies is the wrong abstraction. What matters more is whether a system can specialize rapidly when it encounters a new domain, new objective, or new environment. That is why the research paper treats adaptability, rather than generality, as the better North Star. Specialization as a Feature, Not a Failure A second major claim in the research paper is that AI progress should not be framed as a march toward one universal model that does everything equally well. The research team argued that specialization is not a weakness of intelligence but a practical route to high performance. Humans themselves are not a counterexample; they are part of the evidence. The research paper suggests that future AI systems will likely need internal specialization, hierarchy, and diversity across models and modalities rather than a single monolithic system. In plain terms, the research paper argues that one model should not be expected to master all domains with equal efficiency just because current marketing language likes the word ‘general.’ Why the Research Paper Points to Self-Supervised Learning? From there, the research paper connects SAI to self-supervised learning. The logic is straightforward. If the goal is fast adaptation across a very large task space, then relying only on supervised learning becomes limiting because supervised methods assume access to large, reliable labeled datasets. In real settings, that assumption often fails. The research team argues that self-supervised learning is a promising pathway because it can exploit structure in raw data and has already driven strong results across domains. Importantly, they do not claim that SAI requires one specific architecture. They present self-supervised learning as a promising route, not a final architectural answer. World Models and the Limits of Surface-Level Prediction The research paper also argues that strong adaptation likely benefits from world models. Here the research team move away from the idea that token-level or pixel-level prediction alone is enough for robust intelligence in the physical world. They argue that what matters is learning compact representations that capture system dynamics. In that view, a world model supports simulation and planning, which in turn support zero-shot and few-shot adaptation. The research paper points to latent prediction architectures such as JEPA, Dreamer 4, and Genie 2 as examples of the kind of direction the field should explore, while again stating that SAI does not dictate a single architecture. A Warning Against Architectural Monoculture The research team also criticize the current level of architectural homogeneity in advanced AI. They note that autoregressive LLMs and LMMs dominate the ‘general’ AI landscape in part because shared tooling and benchmarks create momentum. But the research paper argues that this concentration narrows the search space and can slow progress. It further claims that autoregressive systems have well-known weaknesses, including error accumulation over long horizons, which makes long-horizon interaction brittle. Their broader point is not that current large models are useless. It is that the field should avoid treating one successful paradigm as the final template for intelligence. Key Takeaways The research paper argues AGI is not a precise scientific target: According to the research team, AGI is used inconsistently across academia and industry, making it difficult to define, measure, or use as a stable research goal. Human intelligence should not be treated as the definition of

In this tutorial, we explore tqdm in depth and demonstrate how we build powerful, real-time progress tracking into modern Python workflows. We begin with nested progress bars and manual progress control, then move into practical scenarios such as streaming downloads, pandas data processing, parallel execution, structured logging, and asynchronous tasks. Throughout this tutorial, we focus on writing clean, production-ready code that runs in Colab while showcasing the advanced capabilities of tqdm beyond simple loops. Copy CodeCopiedUse a different Browser !pip -q install -U tqdm import time, math, random, asyncio, hashlib, logging import pandas as pd import requests from tqdm.auto import tqdm, trange from tqdm.contrib.concurrent import thread_map, process_map from tqdm.contrib.logging import logging_redirect_tqdm import tqdm as tqdm_pkg print(“tqdm version:”, tqdm_pkg.__version__) print(“pandas version:”, pd.__version__) print(“requests version:”, requests.__version__) We install and configure tqdm in a Colab-safe manner while preserving the existing environment dependencies. We import all required libraries, including concurrency and logging helpers from tqdm.contrib. We also print version information to verify that our runtime setup is stable before proceeding. Copy CodeCopiedUse a different Browser print(“1) Nested progress bars (position/leave) + tqdm.write()”) outer = trange(5, desc=”Outer loop”, leave=True) for i in outer: inner = trange(20, desc=f”Inner loop {i}”, leave=False, position=1) for j in inner: time.sleep(0.01) if j in (0, 10, 19): tqdm.write(f” note: i={i}, j={j}”) print() print(“2) Manual progress (unknown -> known total, update(), set_postfix())”) items = list(range(1, 101)) pbar = tqdm(total=None, desc=”Processing (discovering total)”, unit=”item”) seen = 0 for x in items: time.sleep(0.005) seen += 1 if seen == 25: pbar.total = len(items) pbar.refresh() pbar.update(1) if x % 20 == 0: pbar.set_postfix(last=x, sqrt=round(math.sqrt(x), 3)) pbar.close() print() We demonstrate nested progress bars and show how we manage multiple levels of iteration cleanly using position and leave. We also explore manual progress control by dynamically setting totals and updating progress explicitly. By using set_postfix, we enrich the progress bar with live metadata as it runs. Copy CodeCopiedUse a different Browser print(“3) Download with streaming progress”) url = “https://raw.githubusercontent.com/tqdm/tqdm/master/README.rst” out_path = “/content/tqdm_README.rst” with requests.get(url, stream=True, timeout=30) as r: r.raise_for_status() total = int(r.headers.get(“Content-Length”, 0)) or None chunk = 1024 * 32 with open(out_path, “wb”) as f, tqdm( total=total, unit=”B”, unit_scale=True, unit_divisor=1024, desc=”Downloading README”, miniters=1, ) as bar: for part in r.iter_content(chunk_size=chunk): if not part: continue f.write(part) bar.update(len(part)) print(“Saved:”, out_path) print() We implement a real-world streaming download scenario using requests with chunk-based updates. We track byte-level progress with automatic unit scaling and accurate handling of content length. This shows how we efficiently and transparently monitor external I/O operations. Copy CodeCopiedUse a different Browser print(“4) pandas progress_apply (Series) + DataFrame row-wise progress (safe)”) tqdm.pandas() df = pd.DataFrame({ “user_id”: range(1, 2001), “value”: [random.random() for _ in range(2000)], }) def heavy_fn(v: float) -> str: time.sleep(0.0005) s = f”{v:.10f}”.encode(“utf-8”) return hashlib.sha256(s).hexdigest()[:10] df[“hash”] = df[“value”].progress_apply(heavy_fn) df2 = df[[“value”]].copy() df2[“hash2”] = [ heavy_fn(float(v)) for v in tqdm(df2[“value”].to_list(), desc=”Row-wise hash2″, total=len(df2)) ] df[“hash2”] = df2[“hash2”] print(df.head(3)) print() We integrate tqdm with pandas to monitor vectorized operations using progress_apply. We implement a hashing workload to simulate realistic, computationally heavy transformations. We also demonstrate a safe row-wise progress pattern to ensure compatibility with Colab’s pinned pandas version. Copy CodeCopiedUse a different Browser print(“5) Concurrency progress: thread_map / process_map”) def cpuish(n: int) -> int: x = 0 for i in range(50_000): x = (x + (n * i)) % 1_000_003 return x nums = list(range(80)) thread_results = thread_map(cpuish, nums, max_workers=8, desc=”thread_map”) print(“thread_map done:”, len(thread_results)) proc_results = process_map(cpuish, nums[:20], max_workers=2, chunksize=2, desc=”process_map”) print(“process_map done:”, len(proc_results)) print() print(“6) logging_redirect_tqdm (logs won’t break bars)”) logger = logging.getLogger(“demo”) logger.setLevel(logging.INFO) handler = logging.StreamHandler() handler.setFormatter(logging.Formatter(“%(levelname)s: %(message)s”)) logger.handlers = [handler] with logging_redirect_tqdm(): for k in tqdm(range(60), desc=”Work with logs”): time.sleep(0.01) if k in (5, 25, 45): logger.info(f”checkpoint k={k}”) print() print(“7) asyncio progress (as_completed) — Colab/Jupyter-safe”) async def io_task(i: int): await asyncio.sleep(random.uniform(0.02, 0.12)) return i, random.random() async def run_async(): tasks = [asyncio.create_task(io_task(i)) for i in range(80)] results = [] for fut in tqdm(asyncio.as_completed(tasks), total=len(tasks), desc=”async tasks”): results.append(await fut) return results results = await run_async() print(“async done:”, len(results), “results”) We explore advanced execution patterns including multithreading, multiprocessing, structured logging, and asynchronous task tracking. We use thread_map and process_map to parallelize CPU-bound workloads with visible progress. Also, we handle asyncio safely in a notebook environment using top-level await, ensuring smooth progress tracking without event loop conflicts. In conclusion, we integrated tqdm across synchronous, parallel, logging-aware, and asynchronous environments. We saw how progress bars enhance observability, improve debugging clarity, and make long-running workloads more transparent and user-friendly. With these advanced patterns, we now have a solid foundation to incorporate robust progress monitoring into data pipelines, machine learning workflows, distributed systems, and real-world production applications. Check out the Full Codes here. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post How to Build Progress Monitoring Using Advanced tqdm for Async, Parallel, Pandas, Logging, and High-Performance Workflows appeared first on MarkTechPost.

The ongoing public feud between the Department of Defense and the AI company Anthropic has raised a deep and still unanswered question: Does the law actually allow the US government to conduct mass surveillance on Americans? Surprisingly, the answer is not straightforward. More than a decade after Edward Snowden exposed the NSA’s collection of bulk metadata from the phones of Americans, the US is still navigating a gap between what ordinary people think and what the law allows.  The flashpoint in the standoff between Anthropic and the government was the Pentagon’s desire to use Anthropic’s AI Claude to analyze bulk commercial data on Americans. Anthropic demanded that its AI not be used for mass domestic surveillance (or for autonomous weapons, which are machines that can kill targets without human oversight). A week after negotiations broke down, the Pentagon designated Anthropic a supply chain risk, a label typically reserved for foreign companies that pose a threat to national security.  Meanwhile, OpenAI, the rival AI company behind ChatGPT, sealed a deal that allowed the Pentagon to use its AI for “all lawful purposes”—language that critics say left the door open to domestic surveillance. Over the following weekend, users uninstalled ChatGPT in droves. Protesters chalked messages around OpenAI’s headquarters in San Francisco: “What are your redlines?”  OpenAI announced on Monday that it had reworked its deal to make sure that its AI will not be used for domestic surveillance. The company added that its services will not be used by intelligence agencies, such as the NSA.  CEO Sam Altman suggested that existing law prohibits domestic surveillance by the Department of Defense (now sometimes called the Department of War) and that OpenAI’s contract simply needed to reference this law. “The DoW agrees with these principles, reflects them in law and policy, and we put them into our agreement,” he wrote on X. Anthropic CEO Dario Amodei argued the opposite. “To the extent that such surveillance is currently legal, this is only because the law has not yet caught up with the rapidly growing capabilities of AI,” he wrote in a policy statement.  So, who is right? Does the law allow the Pentagon to surveil Americans using AI? Supercharged surveillance The answer depends on what we think counts as surveillance. “A lot of stuff that normal people would consider a search or surveillance … is not actually considered a search or surveillance by the law,” says Alan Rozenshtein, a law professor at the University of Minnesota Law School. That means public information—such as social media posts, surveillance camera footage, and voter registration records—is fair game. So is information on Americans picked up incidentally from surveillance of foreign nationals.  Most notably, the government can purchase commercial data from companies, which can include sensitive personal information like mobile location and web browsing records. In recent years, agencies from ICE and IRS to the FBI and NSA have increasingly tapped into this data marketplace, fueled by an internet economy that harvests user data for advertising. These data sets can let the government access information that might not be available without a warrant or subpoena, which are normally required to obtain sensitive personal data. “There’s a huge amount of information that the government can collect on Americans that is not itself regulated either by the Constitution, which is the Fourth Amendment, or statute,” says Rozenshtein. And there aren’t meaningful limits on what the government can do with all this data.  That’s because until the last several decades, people weren’t generating massive clouds of data that opened up new possibilities for surveillance. The Fourth Amendment, which protects against unreasonable search and seizure, was written when collecting information meant entering people’s homes.  Subsequent laws, like the Foreign Intelligence Surveillance Act of 1978 or the Electronic Communications Privacy Act of 1986, were passed when surveillance involved wiretapping phone calls and intercepting emails. The bulk of laws governing surveillance were on the books before the internet took off. We weren’t generating vast trails of online data, and the government didn’t have sophisticated tools to analyze the data.  Now we do, and AI supercharges what kind of surveillance can be carried out. “What AI can do is it can take a lot of information, none of which is by itself sensitive, and therefore none of which by itself is regulated, and it can give the government a lot of powers that the government didn’t have before,” says Rozenshtein.  AI can aggregate individual pieces of information to spot patterns, draw inferences, and build detailed profiles of people—at massive scale. And as long as the government collects the information lawfully, it can do whatever it wants with that information, including feeding it to AI systems. “The law has not caught up with technological reality,” says Rozenshtein. While surveillance can raise serious privacy concerns, the Pentagon can have legitimate national security interests in collecting and analyzing data on Americans. “In order to collect information on Americans, it has to be for a very specific subset of missions,” says Loren Voss, a former military intelligence officer at the Pentagon.  For example, a counterintelligence mission might require information about an American who is working for a foreign country, or plotting to engage in international terrorist activities. But targeted intelligence can sometimes stretch into collecting more data. “This kind of collection does make people nervous,” says Voss.  Lawful use OpenAI has amended its contract to say that the company’s AI system “shall not be intentionally used for domestic surveillance of U.S. persons and nationals,” in line with relevant laws. The amendment clarifies that this prohibits “deliberate tracking, surveillance or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information.” But the added language might not do much to override the clause that the Pentagon may use the company’s AI system for all lawful purposes, which could include collecting and analyzing sensitive personal information. “OpenAI can say whatever it wants in its agreement … but the Pentagon’s gonna use the tech