Uncategorized

In this tutorial, we build an uncertainty-aware large language model system that not only generates answers but also estimates the confidence in those answers. We implement a three-stage reasoning pipeline in which the model first produces an answer along with a self-reported confidence score and a justification. We then introduce a self-evaluation step that allows the model to critique and refine its own response, simulating a meta-cognitive check. If the model determines that its confidence is low, we automatically trigger a web research phase that retrieves relevant information from live sources and synthesizes a more reliable answer. By combining confidence estimation, self-reflection, and automated research, we create a practical framework for building more trustworthy and transparent AI systems that can recognize uncertainty and actively seek better information. Copy CodeCopiedUse a different Browser import os, json, re, textwrap, getpass, sys, warnings from dataclasses import dataclass, field from typing import Optional from openai import OpenAI from ddgs import DDGS from rich.console import Console from rich.table import Table from rich.panel import Panel from rich import box warnings.filterwarnings(“ignore”, category=DeprecationWarning) def _get_api_key() -> str: key = os.environ.get(“OPENAI_API_KEY”, “”).strip() if key: return key try: from google.colab import userdata key = userdata.get(“OPENAI_API_KEY”) or “” if key.strip(): return key.strip() except Exception: pass console = Console() console.print( “n[bold cyan]OpenAI API Key required[/bold cyan]n” “[dim]Your key will not be echoed and is never stored to disk.n” “To skip this prompt in future runs, set the environment variable:n” ” export OPENAI_API_KEY=sk-…[/dim]n” ) key = getpass.getpass(” Enter your OpenAI API key: “).strip() if not key: Console().print(“[bold red]No API key provided — exiting.[/bold red]”) sys.exit(1) return key OPENAI_API_KEY = _get_api_key() MODEL = “gpt-4o-mini” CONFIDENCE_LOW = 0.55 CONFIDENCE_MED = 0.80 client = OpenAI(api_key=OPENAI_API_KEY) console = Console() @dataclass class LLMResponse: question: str answer: str confidence: float reasoning: str sources: list[str] = field(default_factory=list) researched: bool = False raw_json: dict = field(default_factory=dict) We import all required libraries and configure the runtime environment for the uncertainty-aware LLM pipeline. We securely retrieve the OpenAI API key using environment variables, Colab secrets, or a hidden terminal prompt. We also define the LLMResponse data structure that stores the question, answer, confidence score, reasoning, and research metadata used throughout the system. Copy CodeCopiedUse a different Browser SYSTEM_UNCERTAINTY = “”” You are an expert AI assistant that is HONEST about what it knows and doesn’t know. For every question you MUST respond with valid JSON only (no markdown, no prose outside JSON): { “answer”: “<your best answer — thorough, factual>”, “confidence”: <float 0.0-1.0>, “reasoning”: “<explain WHY you are or aren’t confident; mention specific knowledge gaps>” } Confidence scale: 0.90-1.00 → very high: well-established fact, you are certain 0.75-0.89 → high: strong knowledge, minor uncertainty 0.55-0.74 → medium: plausible but you may be wrong, could be outdated 0.30-0.54 → low: significant uncertainty, answer is a best guess 0.00-0.29 → very low: mostly guessing, minimal reliable knowledge Be CALIBRATED — do not always give high confidence. Genuinely reflect uncertainty about recent events (after your knowledge cutoff), niche topics, numerical claims, and anything that changes over time. “””.strip() SYSTEM_SYNTHESIS = “”” You are a research synthesizer. Given a question, a preliminary answer, and web-search snippets, produce an improved final answer grounded in the evidence. Respond in JSON only: { “answer”: “<improved, evidence-grounded answer>”, “confidence”: <float 0.0-1.0>, “reasoning”: “<explain how the search evidence changed or confirmed the answer>” } “””.strip() def query_llm_with_confidence(question: str) -> LLMResponse: completion = client.chat.completions.create( model=MODEL, temperature=0.2, response_format={“type”: “json_object”}, messages=[ {“role”: “system”, “content”: SYSTEM_UNCERTAINTY}, {“role”: “user”, “content”: question}, ], ) raw = json.loads(completion.choices[0].message.content) return LLMResponse( question=question, answer=raw.get(“answer”, “”), confidence=float(raw.get(“confidence”, 0.5)), reasoning=raw.get(“reasoning”, “”), raw_json=raw, ) We define the system prompts that instruct the model to report answers along with calibrated confidence and reasoning. We then implement the query_llm_with_confidence function that performs the first stage of the pipeline. This stage generates the model’s answer while forcing the output to be structured JSON containing the answer, confidence score, and explanation. Copy CodeCopiedUse a different Browser def self_evaluate(response: LLMResponse) -> LLMResponse: critique_prompt = f””” Review this answer and its stated confidence. Check for: 1. Logical consistency 2. Whether the confidence matches the actual quality of the answer 3. Any factual errors you can spot Question: {response.question} Proposed answer: {response.answer} Stated confidence: {response.confidence} Stated reasoning: {response.reasoning} Respond in JSON: {{ “revised_confidence”: <float — adjust if the self-check changes your view>, “critique”: “<brief critique of the answer quality>”, “revised_answer”: “<improved answer, or repeat original if fine>” }} “””.strip() completion = client.chat.completions.create( model=MODEL, temperature=0.1, response_format={“type”: “json_object”}, messages=[ {“role”: “system”, “content”: “You are a rigorous self-critic. Respond in JSON only.”}, {“role”: “user”, “content”: critique_prompt}, ], ) ev = json.loads(completion.choices[0].message.content) response.confidence = float(ev.get(“revised_confidence”, response.confidence)) response.answer = ev.get(“revised_answer”, response.answer) response.reasoning += f”nn[Self-Eval Critique]: {ev.get(‘critique’, ”)}” return response def web_search(query: str, max_results: int = 5) -> list[dict]: results = DDGS().text(query, max_results=max_results) return list(results) if results else [] def research_and_synthesize(response: LLMResponse) -> LLMResponse: console.print(f” [yellow] Confidence {response.confidence:.0%} is low — triggering auto-research…[/yellow]”) snippets = web_search(response.question) if not snippets: console.print(” [red]No search results found.[/red]”) return response formatted = “nn”.join( f”[{i+1}] {s.get(‘title’,”)}n{s.get(‘body’,”)}nURL: {s.get(‘href’,”)}” for i, s in enumerate(snippets) ) synthesis_prompt = f””” Question: {response.question} Preliminary answer (low confidence): {response.answer} Web search snippets: {formatted} Synthesize an improved answer using the evidence above. “””.strip() completion = client.chat.completions.create( model=MODEL, temperature=0.2, response_format={“type”: “json_object”}, messages=[ {“role”: “system”, “content”: SYSTEM_SYNTHESIS}, {“role”: “user”, “content”: synthesis_prompt}, ], ) syn = json.loads(completion.choices[0].message.content) response.answer = syn.get(“answer”, response.answer) response.confidence = float(syn.get(“confidence”, response.confidence)) response.reasoning += f”nn[Post-Research]: {syn.get(‘reasoning’, ”)}” response.sources = [s.get(“href”, “”) for s in snippets if s.get(“href”)] response.researched = True return response We implement a self-evaluation stage in which the model critiques its own answer and revises its confidence as needed. We also introduce the web search capability that retrieves live information using DuckDuckGo. If the model’s confidence is low, we synthesize the search results with the preliminary answer to produce an improved response grounded in external evidence. Copy CodeCopiedUse a different Browser def self_evaluate(response: LLMResponse) -> LLMResponse: critique_prompt = f””” Review this answer and its stated confidence. Check for: 1. Logical consistency 2. Whether the confidence matches the actual quality of the answer 3. Any factual errors

In this tutorial, we explore the capabilities of the pymatgen library for computational materials science using Python. We begin by constructing crystal structures such as silicon, sodium chloride, and a LiFePO₄-like material, and then investigate their lattice properties, densities, and compositions. Also, we analyze symmetry using space-group detection, examine atomic coordination environments, and apply oxidation-state decorations to better understand the structures’ chemistry. We also generate supercells, perturb atomic positions, and compute distance matrices to study structural relationships at larger scales. Along the way, we simulate X-ray diffraction patterns, construct a simple phase diagram, and demonstrate how disordered alloy structures can be approximated by ordered configurations. Finally, we extend the workflow to include molecule analysis, CIF export, and optional querying of the Materials Project database, thereby illustrating how pymatgen can serve as a powerful toolkit for materials modeling and data analysis. Copy CodeCopiedUse a different Browser !pip -q install pymatgen mp-api spglib import os import json import warnings import sys warnings.filterwarnings(“ignore”) import numpy as np import pandas as pd import matplotlib.pyplot as plt from pymatgen.core import Lattice, Structure, Molecule from pymatgen.core.surface import SlabGenerator from pymatgen.core.composition import Composition from pymatgen.symmetry.analyzer import SpacegroupAnalyzer from pymatgen.analysis.local_env import CrystalNN from pymatgen.analysis.diffraction.xrd import XRDCalculator from pymatgen.analysis.phase_diagram import PDEntry, PhaseDiagram from pymatgen.transformations.standard_transformations import ( SupercellTransformation, OrderDisorderedStructureTransformation, OxidationStateDecorationTransformation, ) from pymatgen.io.cif import CifWriter print(“Python:”, sys.version.split()[0]) print(“NumPy:”, np.__version__) print(“pandas:”, pd.__version__) try: import pymatgen print(“pymatgen:”, pymatgen.__version__) except Exception: import importlib.metadata print(“pymatgen:”, importlib.metadata.version(“pymatgen”)) def line(): print(“=” * 100) def header(title): line() print(title) line() header(“1. BUILD EXAMPLE STRUCTURES”) si = Structure( Lattice.cubic(5.431), [“Si”, “Si”], [[0, 0, 0], [0.25, 0.25, 0.25]], ) nacl = Structure( Lattice.cubic(5.64), [“Na”, “Cl”], [[0, 0, 0], [0.5, 0.5, 0.5]], ) li_fe_po4 = Structure( Lattice.orthorhombic(10.33, 6.01, 4.69), [“Li”, “Fe”, “P”, “O”, “O”, “O”, “O”], [ [0.0, 0.0, 0.0], [0.5, 0.5, 0.5], [0.1, 0.25, 0.2], [0.22, 0.04, 0.28], [0.72, 0.54, 0.78], [0.31, 0.66, 0.12], [0.81, 0.16, 0.62], ], ) for name, s in [(“Si”, si), (“NaCl”, nacl), (“LiFePO4-like”, li_fe_po4)]: print(f”{name}: formula={s.composition.formula}, sites={len(s)}, volume={s.volume:.3f} Å^3″) We begin by installing the required libraries. We initialize the environment, verify package versions, and define helper functions to organize the output. We then construct example crystal structures such as silicon, NaCl, and a LiFePO₄-like structure and print their basic structural properties. Copy CodeCopiedUse a different Browser header(“2. BASIC INTROSPECTION”) for name, s in [(“Si”, si), (“NaCl”, nacl), (“LiFePO4-like”, li_fe_po4)]: print(f”n{name}”) print(“Reduced formula:”, s.composition.reduced_formula) print(“Density:”, round(s.density, 4), “g/cm^3”) print(“Lattice parameters (a, b, c):”, tuple(round(x, 4) for x in s.lattice.abc)) print(“Angles (alpha, beta, gamma):”, tuple(round(x, 4) for x in s.lattice.angles)) print(“First site:”, s[0]) header(“3. SPACE GROUP AND SYMMETRY ANALYSIS”) for name, s in [(“Si”, si), (“NaCl”, nacl), (“LiFePO4-like”, li_fe_po4)]: sga = SpacegroupAnalyzer(s, symprec=0.1) print(f”n{name}”) print(“Space group symbol:”, sga.get_space_group_symbol()) print(“Space group number:”, sga.get_space_group_number()) print(“Crystal system:”, sga.get_crystal_system()) print(“Lattice type:”, sga.get_lattice_type()) print(“Primitive sites:”, len(sga.find_primitive())) print(“Conventional sites:”, len(sga.get_conventional_standard_structure())) We examine the structures in greater detail by inspecting their formulas, densities, lattice parameters, and site information. We then perform a symmetry analysis using SpacegroupAnalyzer to determine space-group symbols, crystal systems, and lattice types. Through this step, we gain insight into the crystallographic symmetry and structural characteristics of the materials. Copy CodeCopiedUse a different Browser header(“4. LOCAL ENVIRONMENT WITH CRYSTALNN”) cnn = CrystalNN() def summarize_neighbors(structure, label): print(f”n{label}”) for i, site in enumerate(structure[:min(4, len(structure))]): try: nn_info = cnn.get_nn_info(structure, i) species = [str(x[“site”].specie) for x in nn_info] weights = [round(float(x[“weight”]), 3) for x in nn_info] print(f”Site {i} {site.species_string}: CN={len(nn_info)}, neighbors={species}, weights={weights}”) except Exception as e: print(f”Site {i} {site.species_string}: neighbor analysis failed -> {e}”) summarize_neighbors(si, “Si”) summarize_neighbors(nacl, “NaCl”) header(“5. OXIDATION STATE DECORATION”) oxi_transform = OxidationStateDecorationTransformation( {“Li”: 1, “Fe”: 2, “P”: 5, “O”: -2, “Na”: 1, “Cl”: -1, “Si”: 0} ) nacl_oxi = oxi_transform.apply_transformation(nacl.copy()) lfp_oxi = oxi_transform.apply_transformation(li_fe_po4.copy()) print(“NaCl species with oxidation states:”, [str(site.specie) for site in nacl_oxi]) print(“LiFePO4-like species with oxidation states:”, [str(site.specie) for site in lfp_oxi]) We analyze the local atomic environments using the CrystalNN coordination analysis algorithm. We identify neighboring atoms for selected sites and evaluate their coordination numbers and weights. We then decorate the structures with oxidation states to better represent the chemical environment. Copy CodeCopiedUse a different Browser header(“6. MAKE SUPERCELLS”) si_super = SupercellTransformation([[2, 0, 0], [0, 2, 0], [0, 0, 2]]).apply_transformation(si.copy()) nacl_super = SupercellTransformation([[2, 0, 0], [0, 2, 0], [0, 0, 2]]).apply_transformation(nacl.copy()) print(“Si supercell sites:”, len(si_super), “formula:”, si_super.composition.formula) print(“NaCl supercell sites:”, len(nacl_super), “formula:”, nacl_super.composition.formula) header(“7. PERTURB STRUCTURE AND COMPUTE DISTANCE MATRIX”) si_perturbed = si_super.copy() si_perturbed.translate_sites([0], [0.01, -0.005, 0.012], frac_coords=False) dm = si_perturbed.distance_matrix print(“Distance matrix shape:”, dm.shape) print(“First 5 distances from site 0:”, np.round(dm[0][:5], 4)) header(“8. GENERATE A SURFACE SLAB”) slabgen = SlabGenerator( initial_structure=si, miller_index=(1, 1, 1), min_slab_size=8.0, min_vacuum_size=12.0, center_slab=True, in_unit_planes=False, ) slabs = slabgen.get_slabs() slab = slabs[0] print(“Number of generated slabs:”, len(slabs)) print(“Chosen slab formula:”, slab.composition.formula) print(“Chosen slab sites:”, len(slab)) print(“Chosen slab lattice:”, tuple(round(x, 3) for x in slab.lattice.abc)) We expand the crystal structures into larger supercells to study periodic structures at a larger scale. We apply a small perturbation to atomic positions and compute the resulting distance matrix to analyze structural changes. We also generate a surface slab from the silicon crystal to demonstrate how surface structures can be modeled. Copy CodeCopiedUse a different Browser header(“9. XRD SIMULATION”) xrd = XRDCalculator(wavelength=”CuKa”) pattern_si = xrd.get_pattern(si, two_theta_range=(10, 90)) pattern_nacl = xrd.get_pattern(nacl, two_theta_range=(10, 90)) plt.figure(figsize=(12, 4)) plt.vlines(pattern_si.x, [0], pattern_si.y, linewidth=1.5) plt.xlabel(r”2$theta$ (degrees)”) plt.ylabel(“Intensity”) plt.title(“Simulated XRD Pattern: Si”) plt.show() plt.figure(figsize=(12, 4)) plt.vlines(pattern_nacl.x, [0], pattern_nacl.y, linewidth=1.5) plt.xlabel(r”2$theta$ (degrees)”) plt.ylabel(“Intensity”) plt.title(“Simulated XRD Pattern: NaCl”) plt.show() header(“10. SIMPLE PHASE DIAGRAM”) entries = [ PDEntry(Composition(“Li”), 0.0), PDEntry(Composition(“Fe”), 0.0), PDEntry(Composition(“P”), 0.0), PDEntry(Composition(“O2”), 0.0), PDEntry(Composition(“Li2O”), -6.0), PDEntry(Composition(“FeO”), -4.2), PDEntry(Composition(“Fe2O3”), -10.5), PDEntry(Composition(“P2O5”), -15.0), PDEntry(Composition(“Li3PO4”), -18.5), PDEntry(Composition(“FePO4”), -12.2), PDEntry(Composition(“LiFePO4”), -16.9), ] pdg = PhaseDiagram(entries) target = [e for e in entries if e.composition.reduced_formula == “LiFePO4”][0] e_above_hull = pdg.get_e_above_hull(target) decomp, e_hull = pdg.get_decomp_and_e_above_hull(target) print(“Target entry:”, target.composition.reduced_formula) print(“Energy above hull:”, round(float(e_above_hull), 6), “eV/atom”) print(“Decomposition products:”) for k, v in decomp.items(): print(” “, k.composition.reduced_formula, “:”, round(float(v), 6)) We simulate X-ray diffraction patterns for silicon and NaCl using pymatgen’s diffraction tools. We visualize the diffraction peaks to understand how the crystal structure influences the XRD pattern. We then construct a simple thermodynamic phase diagram and calculate the stability of LiFePO₄ relative to competing phases. Copy CodeCopiedUse

OpenAI is refocusing its research efforts and throwing its resources into a new grand challenge. The San Francisco firm has set its sights on building what it calls an AI researcher, a fully automated agent-based system that will be able to go off and tackle large, complex problems by itself. ​​OpenAI says that this new research goal will be its “North Star” for the next few years, pulling together multiple research strands, including work on reasoning models, agents, and interpretability. There’s even a timeline. OpenAI plans to build “an autonomous AI research intern”—a system that can take on a small number of specific research problems by itself—by September. The AI intern will be the precursor to a fully automated multi-agent research system that the company plans to debut in 2028. This AI researcher (OpenAI says) will be able to tackle problems that are too large or complex for humans to cope with. Those tasks might be related to math and physics—such as coming up with new proofs or conjectures—or life sciences like biology and chemistry, or even business and policy dilemmas. In theory, you would throw such a tool any kind of problem that can be formulated in text, code, or whiteboard scribbles—which covers a lot. OpenAI has been setting the agenda for the AI industry for years. Its early dominance with large language models shaped the technology that hundreds of millions of people use every day. But it now faces fierce competition from rival model makers like Anthropic and Google DeepMind. What OpenAI decides to build next matters—for itself and for the future of AI.    A big part of that decision falls to Jakub Pachocki, OpenAI’s chief scientist, who sets the company’s long-term research goals. Pachocki played key roles in the development of both GPT-4, a game-changing LLM released in 2023, and so-called reasoning models, a technology that first appeared in 2024 and now underpins all major chatbots and agent-based systems.  In an exclusive interview this week, Pachocki talked me through OpenAI’s latest vision. “I think we are getting close to a point where we’ll have models capable of working indefinitely in a coherent way just like people do,” he says. “Of course, you still want people in charge and setting the goals. But I think we will get to a point where you kind of have a whole research lab in a data center.” Solving hard problems Such big claims aren’t new. Saving the world by solving its hardest problems is the stated mission of all the top AI firms. Demis Hassabis told me back in 2022 that it was why he started DeepMind. Anthropic CEO Dario Amodei says he is building the equivalent of a country of geniuses in a data center. Pachocki’s boss, Sam Altman, wants to cure cancer. But Pachocki says OpenAI now has most of what it needs to get there. In January, OpenAI released Codex, an agent-based app that can spin up code on the fly to carry out tasks on your computer. It can analyze documents, generate charts, make you a daily digest of your inbox and social media, and much more. (Other firms have released similar tools, such as Anthropic’s Claude Code and Claude Cowork.) OpenAI claims that most of its technical staffers now use Codex in their work. You can look at Codex as a very early version of the AI researcher, says Pachocki: “I expect Codex to get fundamentally better.” The key is to make a system that can run for longer periods of time, with less human guidance. “What we’re really looking at for an automated research intern is a system that you can delegate tasks [to] that would take a person a few days,” says Pachocki. “There are a lot of people excited about building systems that can do more long-running scientific research,” says Doug Downey, a research scientist at the Allen Institute for AI, who is not connected to OpenAI. “I think it’s largely driven by the success of these coding agents. The fact that you can delegate quite substantial coding tasks to tools like Codex is incredibly useful and incredibly impressive. And it raises the question: Can we do similar things outside coding, in broader areas of science?” For Pachocki, that’s a clear Yes. In fact, he thinks it’s just a matter of pushing ahead on the path we’re already on. A simple boost in all-round capability also leads to models that can work longer without help, he says. He points to the leap from 2020’s GPT-3 to 2023’s GPT-4, two of OpenAI’s previous models. GPT-4 was able to work on a problem for far longer than its predecessor, even without specialized training, he says.  So-called reasoning models brought another bump. Training LLMs to work through problems step by step, backtracking when they make a mistake or hit a dead end, has also made models better at working for longer periods of time. And Pachocki is convinced that OpenAI’s reasoning models will continue to get better. But OpenAI is also training its systems to work by themselves for longer by feeding them specific samples of complex tasks, such as hard puzzles taken from math and coding contests, which force the models to learn how to do things like keep track of very large chunks of text and split problems up into (and then manage) multiple subtasks. The aim isn’t to build models that just win math competitions. “That lets you prove that the technology works before you connect it to the real world,” says Pachocki. “If we really wanted to, we could build an amazing automated mathematician. We have all the tools, and I think it would be relatively easy. But it’s not something we’re going to prioritize now because, you know, at the point where you believe you can do it, there’s much more urgent things to do.” “We are much more focused now on research that’s relevant in the real world,” he adds. Right now that means taking what Codex can do

arXiv:2507.02768v2 Announce Type: replace-cross Abstract: We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM’s original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM’s native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs.

arXiv:2603.18446v1 Announce Type: new Abstract: Long-context inference remains challenging for large language models due to attention dilution and out-of-distribution degradation. Context selection mitigates this limitation by attending to a subset of key-value cache entries, yet most methods allocate a fixed context budget throughout decoding despite highly non-uniform token-level contextual demands. To address this issue, we propose Uncertainty-Triggered Adaptive Context Allocation (UT-ACA), an inference-time framework that dynamically adjusts the context window based on token-wise uncertainty. UT-ACA learns an uncertainty detector that combines semantic embeddings with logit-based confidence while accounting for uncertainty accumulation across decoding steps. When insufficient evidence is indicated, UT-ACA selectively rolls back, expands the context window, and regenerates the token with additional support. Experiments show that UT-ACA substantially reduces average context usage while preserving generation quality in long-context settings.