Committee

arXiv:2509.15958v1 Announce Type: new Abstract: We introduce a new discrete-time attention model, termed the localmax dynamics, which interpolates between the classic softmax dynamics and the hardmax dynamics, where only the tokens that maximize the influence toward a given token have a positive weight. As in hardmax, uniform weights are determined by a parameter controlling neighbor influence, but the key extension lies in relaxing neighborhood interactions through an alignment-sensitivity parameter, which allows controlled deviations from pure hardmax behavior. As we prove, while the convex hull of the token states still converges to a convex polytope, its structure can no longer be fully described by a maximal alignment set, prompting the introduction of quiescent sets to capture the invariant behavior of tokens near vertices. We show that these sets play a key role in understanding the asymptotic behavior of the system, even under time-varying alignment sensitivity parameters. We further show that localmax dynamics does not exhibit finite-time convergence and provide results for vanishing, nonzero, time-varying alignment-sensitivity parameters, recovering the limiting behavior of hardmax as a by-product. Finally, we adapt Lyapunov-based methods from classical opinion dynamics, highlighting their limitations in the asymmetric setting of localmax interactions and outlining directions for future research.

arXiv:2501.08454v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have become essential tools for digital task assistance. Their training relies heavily on the collection of vast amounts of data, which may include copyright-protected or sensitive information. Recent studies on detecting pretraining data in LLMs have primarily focused on sentence- or paragraph-level membership inference attacks (MIAs), usually involving probability analysis of the target model’s predicted tokens. However, these methods often exhibit poor accuracy, failing to account for the semantic importance of textual content and word significance. To address these shortcomings, we propose Tag&Tab, a novel approach for detecting data used in LLM pretraining. Our method leverages established natural language processing (NLP) techniques to tag keywords in the input text, a process we term Tagging. Then, the LLM is used to obtain probabilities for these keywords and calculate their average log-likelihood to determine input text membership, a process we refer to as Tabbing. Our experiments on four benchmark datasets (BookMIA, MIMIR, PatentMIA, and the Pile) and several open-source LLMs of varying sizes demonstrate an average increase in AUC scores ranging from 5.3% to 17.6% over state-of-the-art methods. Tag&Tab not only sets a new standard for data leakage detection in LLMs, but its outstanding performance is a testament to the importance of words in MIAs on LLMs.

What exactly is being measured when a judge LLM assigns a 1–5 (or pairwise) score? Most “correctness/faithfulness/completeness” rubrics are project-specific. Without task-grounded definitions, a scalar score can drift from business outcomes (e.g., “useful marketing post” vs. “high completeness”). Surveys of LLM-as-a-judge (LAJ) note that rubric ambiguity and prompt template choices materially shift scores and human correlations. How stable are judge decisions to prompt position and formatting? Large controlled studies find position bias: identical candidates receive different preferences depending on order; list-wise and pairwise setups both show measurable drift (e.g., repetition stability, position consistency, preference fairness). Work cataloging verbosity bias shows longer responses are often favored independent of quality; several reports also describe self-preference (judges prefer text closer to their own style/policy). Do judge scores consistently match human judgments of factuality? Empirical results are mixed. For summary factuality, one study reported low or inconsistent correlations with humans for strong models (GPT-4, PaLM-2), with only partial signal from GPT-3.5 on certain error types. Conversely, domain-bounded setups (e.g., explanation quality for recommenders) have reported usable agreement with careful prompt design and ensembling across heterogeneous judges. Taken together, correlation seems task- and setup-dependent, not a general guarantee. How robust are judge LLMs to strategic manipulation? LLM-as-a-Judge (LAJ) pipelines are attackable. Studies show universal and transferable prompt attacks can inflate assessment scores; defenses (template hardening, sanitization, re-tokenization filters) mitigate but do not eliminate susceptibility. Newer evaluations differentiate content-author vs. system-prompt attacks and document degradation across several families (Gemma, Llama, GPT-4, Claude) under controlled perturbations. Is pairwise preference safer than absolute scoring? Preference learning often favors pairwise ranking, yet recent research finds protocol choice itself introduces artifacts: pairwise judges can be more vulnerable to distractors that generator models learn to exploit; absolute (pointwise) scores avoid order bias but suffer scale drift. Reliability therefore hinges on protocol, randomization, and controls rather than a single universally superior scheme. Could “judging” encourage overconfident model behavior? Recent reporting on evaluation incentives argues that test-centric scoring can reward guessing and penalize abstention, shaping models toward confident hallucinations; proposals suggest scoring schemes that explicitly value calibrated uncertainty. While this is a training-time concern, it feeds back into how evaluations are designed and interpreted. Where do generic “judge” scores fall short for production systems? When an application has deterministic sub-steps (retrieval, routing, ranking), component metrics offer crisp targets and regression tests. Common retrieval metrics include Precision@k, Recall@k, MRR, and nDCG; these are well-defined, auditable, and comparable across runs. Industry guides emphasize separating retrieval and generation and aligning subsystem metrics with end goals, independent of any judge LLM. If judge LLMs are fragile, what does “evaluation” look like in the wild? Public engineering playbooks increasingly describe trace-first, outcome-linked evaluation: capture end-to-end traces (inputs, retrieved chunks, tool calls, prompts, responses) using OpenTelemetry GenAI semantic conventions and attach explicit outcome labels (resolved/unresolved, complaint/no-complaint). This supports longitudinal analysis, controlled experiments, and error clustering—regardless of whether any judge model is used for triage. Tooling ecosystems (e.g., LangSmith and others) document trace/eval wiring and OTel interoperability; these are descriptions of current practice rather than endorsements of a particular vendor. Are there domains where LLM-as-a-Judge (LAJ) seems comparatively reliable? Some constrained tasks with tight rubrics and short outputs report better reproducibility, especially when ensembles of judges and human-anchored calibration sets are used. But cross-domain generalization remains limited, and bias/attack vectors persist. Does LLM-as-a-Judge (LAJ) performance drift with content style, domain, or “polish”? Beyond length and order, studies and news coverage indicate LLMs sometimes over-simplify or over-generalize scientific claims compared to domain experts—useful context when using LAJ to score technical material or safety-critical text. Key Technical Observations Biases are measurable (position, verbosity, self-preference) and can materially change rankings without content changes. Controls (randomization, de-biasing templates) reduce but do not eliminate effects. Adversarial pressure matters: prompt-level attacks can systematically inflate scores; current defenses are partial. Human agreement varies by task: factuality and long-form quality show mixed correlations; narrow domains with careful design and ensembling fare better. Component metrics remain well-posed for deterministic steps (retrieval/routing), enabling precise regression tracking independent of judge LLMs. Trace-based online evaluation described in industry literature (OTel GenAI) supports outcome-linked monitoring and experimentation. Summary In conclusion, this article does not argue against the existence of LLM-as-a-Judge but highlights the nuances, limitations, and ongoing debates around its reliability and robustness. The intention is not to dismiss its use but to frame open questions that need further exploration. Companies and research groups actively developing or deploying LLM-as-a-Judge (LAJ) pipelines are invited to share their perspectives, empirical findings, and mitigation strategies—adding valuable depth and balance to the broader conversation on evaluation in the GenAI era. The post LLM-as-a-Judge: Where Do Its Signals Break, When Do They Hold, and What Should “Evaluation” Mean? appeared first on MarkTechPost.

Production-grade agents live or die on data plumbing, controls, and observability—not on model choice. The doc-to-chat pipeline below maps the concrete layers and why they matter. What is a “doc-to-chat” pipeline? A doc-to-chat pipeline ingests enterprise documents, standardizes them, enforces governance, indexes embeddings alongside relational features, and serves retrieval + generation behind authenticated APIs with human-in-the-loop (HITL) checkpoints. It’s the reference architecture for agentic Q&A, copilots, and workflow automation where answers must respect permissions and be audit-ready. Production implementations are variations of RAG (retrieval-augmented generation) hardened with LLM guardrails, governance, and OpenTelemetry-backed tracing. How do you integrate cleanly with the existing stack? Use standard service boundaries (REST/JSON, gRPC) over a storage layer your org already trusts. For tables, Iceberg gives ACID, schema evolution, partition evolution, and snapshots—critical for reproducible retrieval and backfills. For vectors, use a system that coexists with SQL filters: pgvector collocates embeddings with business keys and ACL tags in PostgreSQL; dedicated engines like Milvus handle high-QPS ANN with disaggregated storage/compute. In practice, many teams run both: SQL+pgvector for transactional joins and Milvus for heavy retrieval. Key properties Iceberg tables: ACID, hidden partitioning, snapshot isolation; vendor support across warehouses. pgvector: SQL + vector similarity in one query plan for precise joins and policy enforcement. Milvus: layered, horizontally scalable architecture for large-scale similarity search. How do agents, humans, and workflows coordinate on one “knowledge fabric”? Production agents require explicit coordination points where humans approve, correct, or escalate. AWS A2I provides managed HITL loops (private workforces, flow definitions) and is a concrete blueprint for gating low-confidence outputs. Frameworks like LangGraph model these human checkpoints inside agent graphs so approvals are first-class steps in the DAG, not ad hoc callbacks. Use them to gate actions like publishing summaries, filing tickets, or committing code. Pattern: LLM → confidence/guardrail checks → HITL gate → side-effects. Persist every artifact (prompt, retrieval set, decision) for auditability and future re-runs. How is reliability enforced before anything reaches the model? Treat reliability as layered defenses: Language + content guardrails: Pre-validate inputs/outputs for safety and policy. Options span managed (Bedrock Guardrails) and OSS (NeMo Guardrails, Guardrails AI; Llama Guard). Independent comparisons and a position paper catalog the trade-offs. PII detection/redaction: Run analyzers on both source docs and model I/O. Microsoft Presidio offers recognizers and masking, with explicit caveats to combine with additional controls. Access control and lineage: Enforce row-/column-level ACLs and audit across catalogs (Unity Catalog) so retrieval respects permissions; unify lineage and access policies across workspaces. Retrieval quality gates: Evaluate RAG with reference-free metrics (faithfulness, context precision/recall) using Ragas/related tooling; block or down-rank poor contexts. How do you scale indexing and retrieval under real traffic? Two axes matter: ingest throughput and query concurrency. Ingest: Normalize at the lakehouse edge; write to Iceberg for versioned snapshots, then embed asynchronously. This enables deterministic rebuilds and point-in-time re-indexing. Vector serving: Milvus’s shared-storage, disaggregated compute architecture supports horizontal scaling with independent failure domains; use HNSW/IVF/Flat hybrids and replica sets to balance recall/latency. SQL + vector: Keep business joins server-side (pgvector), e.g., WHERE tenant_id = ? AND acl_tag @> … ORDER BY embedding <-> :q LIMIT k. This avoids N+1 trips and respects policies. Chunking/embedding strategy: Tune chunk size/overlap and semantic boundaries; bad chunking is the silent killer of recall. For structured+unstructured fusion, prefer hybrid retrieval (BM25 + ANN + reranker) and store structured features next to vectors to support filters and re-ranking features at query time. How do you monitor beyond logs? You need traces, metrics, and evaluations stitched together: Distributed tracing: Emit OpenTelemetry spans across ingestion, retrieval, model calls, and tools; LangSmith natively ingests OTEL traces and interoperates with external APMs (Jaeger, Datadog, Elastic). This gives end-to-end timing, prompts, contexts, and costs per request. LLM observability platforms: Compare options (LangSmith, Arize Phoenix, LangFuse, Datadog) by tracing, evals, cost tracking, and enterprise readiness. Independent roundups and matrixes are available. Continuous evaluation: Schedule RAG evals (Ragas/DeepEval/MLflow) on canary sets and live traffic replays; track faithfulness and grounding drift over time. Add schema profiling/mapping on ingestion to keep observability attached to data shape changes (e.g., new templates, table evolution) and to explain retrieval regressions when upstream sources shift. Example: doc-to-chat reference flow (signals and gates) Ingest: connectors → text extraction → normalization → Iceberg write (ACID, snapshots). Govern: PII scan (Presidio) → redact/mask → catalog registration with ACL policies. Index: embedding jobs → pgvector (policy-aware joins) and Milvus (high-QPS ANN). Serve: REST/gRPC → hybrid retrieval → guardrails → LLM → tool use. HITL: low-confidence paths route to A2I/LangGraph approval steps. Observe: OTEL traces to LangSmith/APM + scheduled RAG evaluations. Why “5% AI, 100% software engineering” is accurate in practice? Most outages and trust failures in agent systems are not model regressions; they’re data quality, permissioning, retrieval decay, or missing telemetry. The controls above—ACID tables, ACL catalogs, PII guardrails, hybrid retrieval, OTEL traces, and human gates—determine whether the same base model is safe, fast, and credibly correct for your users. Invest in these first; swap models later if needed. References: https://iceberg.apache.org/docs/1.9.0/evolution/ https://iceberg.apache.org/docs/1.5.2/ https://docs.snowflake.com/en/user-guide/tables-iceberg https://docs.dremio.com/current/developer/data-formats/apache-iceberg/ https://github.com/pgvector/pgvector https://www.postgresql.org/about/news/pgvector-070-released-2852/ https://github.com/pgvector/pgvector-go https://github.com/pgvector/pgvector-rust https://github.com/pgvector/pgvector-java https://milvus.io/docs/four_layers.md https://milvus.io/docs/v2.3.x/architecture_overview.md https://milvus.io/docs/v2.2.x/architecture.md https://www.linkedin.com/posts/armand-ruiz_ https://docs.vespa.ai/en/tutorials/hybrid-search.html https://www.elastic.co/what-is/hybrid-search https://www.elastic.co/search-labs/blog/hybrid-search-elasticsearch https://docs.cohere.com/reference/rerank https://docs.cohere.com/docs/rerank https://cohere.com/rerank https://opentelemetry.io/docs/concepts/signals/traces/ https://opentelemetry.io/docs/specs/otel/logs/ https://docs.smith.langchain.com/evaluation https://docs.smith.langchain.com/evaluation/concepts https://docs.smith.langchain.com/reference/python/evaluation https://docs.smith.langchain.com/observability https://www.langchain.com/langsmith https://arize.com/docs/phoenix https://github.com/Arize-ai/phoenix https://langfuse.com/docs/observability/get-started https://langfuse.com/docs/observability/overview https://docs.datadoghq.com/opentelemetry/ https://langchain-ai.github.io/langgraph/concepts/human_in_the_loop/ https://langchain-ai.github.io/langgraph/tutorials/get-started/4-human-in-the-loop/ https://docs.langchain.com/oss/python/langgraph/add-human-in-the-loop https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-use-augmented-ai-a2i-human-review-loops.html https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-start-human-loop.html https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-a2i-runtime.html https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-monitor-humanloop-results.html https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html https://aws.amazon.com/bedrock/guardrails/ https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateGuardrail.html https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrail.html https://docs.nvidia.com/nemo-guardrails/index.html https://developer.nvidia.com/nemo-guardrails https://github.com/NVIDIA/NeMo-Guardrails https://docs.nvidia.com/nemo/guardrails/latest/user-guides/guardrails-library.html https://guardrailsai.com/docs/ https://github.com/guardrails-ai/guardrails https://guardrailsai.com/docs/getting_started/quickstart https://guardrailsai.com/docs/getting_started/guardrails_server https://pypi.org/project/guardrails-ai/ https://github.com/guardrails-ai/guardrails_pii https://huggingface.co/meta-llama/Llama-Guard-4-12B https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/ https://microsoft.github.io/presidio/ https://github.com/microsoft/presidio https://github.com/microsoft/presidio-research https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/ https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/ https://docs.ragas.io/ https://docs.ragas.io/en/stable/references/evaluate/ https://docs.ragas.io/en/latest/tutorials/rag/ https://python.langchain.com/docs/concepts/text_splitters/ https://python.langchain.com/api_reference/text_splitters/index.html https://pypi.org/project/langchain-text-splitters/ https://milvus.io/docs/evaluation_with_deepeval.md https://mlflow.org/docs/latest/genai/eval-monitor/ https://mlflow.org/docs/2.10.1/llms/rag/notebooks/mlflow-e2e-evaluation.html The post Building AI agents is 5% AI and 100% software engineering appeared first on MarkTechPost.

Computer vision moved fast in 2025: new multimodal backbones, larger open datasets, and tighter model–systems integration. Practitioners need sources that publish rigorously, link code and benchmarks, and track deployment patterns—not marketing posts. This list prioritizes primary research hubs, lab blogs, and production-oriented engineering outlets with consistent update cadence. Use it to monitor SOTA shifts, grab reproducible code paths, and translate papers into deployable pipelines. Google Research (AI Blog) Primary source for advances from Google/DeepMind teams, including vision architectures (e.g., V-MoE) and periodic research year-in-review posts across CV and multimodal. Posts typically include method summaries, figures, and links to papers/code. Marktechpost Consistent reporting on new computer-vision models, datasets, and benchmarks with links to papers, code, and demos. Dedicated CV category plus frequent deep-dives (e.g., DINOv3 releases and analysis). Useful for staying on top of weekly research drops without wading through raw feeds. AI at Meta High-signal posts with preprints and open-source drops. Recent examples include DINOv3—scaled self-supervised backbones with SOTA across dense prediction tasks—which provide technical detail and artifacts. NVIDIA Technical Blog Production-oriented content on VLM-powered analytics, optimized inference, and GPU pipelines. Category feed for Computer Vision includes blueprints, SDK usage, and performance guidance relevant to enterprise deployments. arXiv cs.CV — raw research firehose The canonical preprint feed for CV. Use the recent or new views for daily updates; taxonomy confirms scope (image processing, pattern recognition, scene understanding). Best paired with RSS + custom filters. CVF Open Access (CVPR/ICCV/ECCV) Final versions of main-conference papers and workshops, searchable and citable. CVPR 2025 proceedings and workshop menus are already live, making this the authoritative archive post-acceptance. BAIR Blog (UC Berkeley) Occasional but deep posts on frontier topics (e.g., extremely large image modeling, robotics-vision crossovers). Good for conceptual clarity directly from authors. Stanford Blog Technical explainers and lab roundups (e.g., SAIL at CVPR 2025) with links to papers/talks. Useful to scan emerging directions across perception, generative models, and embodied vision. Roboflow Blog High-frequency, implementation-focused posts (labeling, training, deployment, apps, and trend reports). Strong for practitioners who need working pipelines and edge deployments. Hugging Face Blog Hands-on guides (VLMs, FiftyOne integrations) and ecosystem notes across Transformers, Diffusers, and timm; good for rapid prototyping and fine-tuning CV/VLM stacks. PyTorch Blog Change logs, APIs, and recipes affecting CV training/inference (Transforms V2, multi-weight support, FX feature extraction). Read when upgrading training stacks. The post Top Computer Vision CV Blogs & News Websites (2025) appeared first on MarkTechPost.

Qwen has released Qwen3-ASR-Toolkit, an MIT-licensed Python CLI that programmatically bypasses the Qwen3-ASR-Flash API’s 3-minute/10 MB per-request limit by performing VAD-aware chunking, parallel API calls, and automatic resampling/format normalization via FFmpeg. The result is stable, hour-scale transcription pipelines with configurable concurrency, context injection, and clean text post-processing. Python ≥3.8 prerequisite, Install with: Copy CodeCopiedUse a different Browser pip install qwen3-asr-toolkit What the toolkit adds on top of the API Long-audio handling. The toolkit slices input using voice activity detection (VAD) at natural pauses, keeping each chunk under the API’s hard duration/size caps, then merges outputs in order. Parallel throughput. A thread pool dispatches multiple chunks concurrently to DashScope endpoints, improving wall-clock latency for hour-long inputs. You control concurrency via -j/–num-threads. Format & rate normalization. Any common audio/video container (MP4/MOV/MKV/MP3/WAV/M4A, etc.) is converted to the API’s required mono 16 kHz before submission. Requires FFmpeg installed on PATH. Text cleanup & context. The tool includes post-processing to reduce repetitions/hallucinations and supports context injection to bias recognition toward domain terms; the underlying API also exposes language detection and inverse text normalization (ITN) toggles. The official Qwen3-ASR-Flash API is single-turn and enforces ≤3 min duration and ≤10 MB payloads per call. That is reasonable for interactive requests but awkward for long media. The toolkit operationalizes best practices—VAD-aware segmentation + concurrent calls—so teams can batch large archives or live capture dumps without writing orchestration from scratch. Quick start Install prerequisites Copy CodeCopiedUse a different Browser # System: FFmpeg must be available # macOS brew install ffmpeg # Ubuntu/Debian sudo apt update && sudo apt install -y ffmpeg Install the CLI Copy CodeCopiedUse a different Browser pip install qwen3-asr-toolkit Configure credentials Copy CodeCopiedUse a different Browser # International endpoint key export DASHSCOPE_API_KEY=”sk-…” Run Copy CodeCopiedUse a different Browser # Basic: local video, default 4 threads qwen3-asr -i “/path/to/lecture.mp4” # Faster: raise parallelism and pass key explicitly (optional if env var set) qwen3-asr -i “/path/to/podcast.wav” -j 8 -key “sk-…” # Improve domain accuracy with context qwen3-asr -i “/path/to/earnings_call.m4a” -c “tickers, CFO name, product names, Q3 revenue guidance” Arguments you’ll actually use:-i/–input-file (file path or http/https URL), -j/–num-threads, -c/–context, -key/–dashscope-api-key, -t/–tmp-dir, -s/–silence. Output is printed and saved as <input_basename>.txt. Minimal pipeline architecture Load local file or URL → 2) VAD to find silence boundaries → 3) Chunk under API caps → 4) Resample to 16 kHz mono → 5) Parallel submit to DashScope → 6) Aggregate segments in order → 7) Post-process text (dedupe, repetitions) → 8) Emit .txt transcript. Summary Qwen3-ASR-Toolkit turns Qwen3-ASR-Flash into a practical long-audio pipeline by combining VAD-based segmentation, FFmpeg normalization (mono/16 kHz), and parallel API dispatch under the 3-minute/10 MB caps. Teams get deterministic chunking, configurable throughput, and optional context/LID/ITN controls without custom orchestration. For production, pin the package version, verify region endpoints/keys, and tune thread count to your network and QPS—then pip install qwen3-asr-toolkit and ship. Check out the GitHub Page for Codes. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. The post Qwen3-ASR-Toolkit: An Advanced Open Source Python Command-Line Toolkit for Using the Qwen-ASR API Beyond the 3 Minutes/10 MB Limit appeared first on MarkTechPost.