Uncategorized

MIT Technology Review Explains: Let our writers untangle the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here. When two OpenAI models hacked into the website Hugging Face in July, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question. According to a postmortem from OpenAI, the models, which had been stripped of their typical security features for testing, decided to solve a cybersecurity exercise by hacking out of the isolated environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored. The Hugging Face incident has attracted intense attention over the past couple of weeks. It’s a dramatic illustration of just how good AI models have gotten at hacking: In order to get into Hugging Face’s databases, the models had to string together several previously undiscovered cybersecurity exploits. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat. And as models get increasingly powerful, the consequences could get far more severe. What is reward hacking? Researchers have known for a while that AIs tend to take creative approaches to achieving the goals that have been set for them. Back in 2016, Anthropic cofounders Dario Amodei and Jack Clark, who were then working at OpenAI, published a blog post about an AI agent that they had been training to play a boat-racing Flash game called Coast Runners. Instead of driving through the race to the finish line, as the researchers had anticipated, the agent found a corner of the course where it could spin around collecting power-ups, thereby maximizing its score. The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies. Historically, researchers have discussed reward hacking almost exclusively in the context of reinforcement learning, a common AI training regime. Like dog training, reinforcement learning involves giving the subject a reward when it achieves an objective; the rewards then reinforce the behaviors that led up to that achievement. In the case of AI training, the rewards themselves are purely mathematical, but in effect they’re the same as a dog treat: After receiving a reward, the agent is more likely to repeat whatever actions produced it. It can be challenging to write good rules for when and when not to give an agent a reward, though. In the Coast Runners case, the agent was rewarded on the basis of its score in the game, and it found a shortcut to achieving the highest possible score by spinning in circles for power-ups. Once it happened on that strategy and received a reward for it, the strategy was reinforced, and the agent completely abandoned the race. The solution was to tweak the rewards by giving the agent fewer points for hitting power-ups and more for finishing the course. How does reward hacking work for LLMs? With today’s sophisticated LLM-based agents, determining when and when not to give a reward can be much trickier. If an AI system is asked to solve a coding problem, it might work hard to find the solution—the kind of behavior that AI companies want to reinforce. But it could also tweak the code that evaluates whether the problem has been solved, look up the solution on the internet, or otherwise cheat. These are behaviors that AI companies want to stamp out in their models, but if the model cheats convincingly enough, it will instead get rewarded and the behavior will be reinforced. Anthropic has said that it has detected some instances of cheating in its models during training, which suggests that other forms of cheating might be going undetected. If so, the models could be being trained to behave badly. (This problem is different from the Anthropic security incidents announced last week, in which agents were accidentally given access to the internet and did not deliberately hack out of their sandboxes, as the OpenAI models did.)  “We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating,” says Jeffrey Ladish, director of the AI research nonprofit Palisade Research. “We don’t have a way to go in there and be like, No, you need to actually care about what we care about. We have no ability to do that.” The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass. What are the risks? Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. “At the end of the day, you’re sort of playing whack-a-mole,” Ladish says. “You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.” For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. “This seems like a nuisance rather than an existential threat,” says Ariana Azarbal, an AI safety research fellow at Anthropic. It doesn’t

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Humanoid robots usually elicit more cringe than awe: They stumble, kick children, and despite advances are still worse at using their hands than my toddler. It’s a nascent industry, and such robots are more commonly seen in viral videos than real workplaces or homes.  It was a surprise, then, when last week the Federal Trade Commission issued a sweeping ban on foreign imports of advanced robots, including humanoids, quadrupeds, and wheeled robots. The decision, made by an increasingly partisan and Trump-aligned FTC, cites two reasons. One is that foreign-made humanoids will collect so much data—in homes but also potentially at sensitive facilities—that they’d pose a threat to national security. The second is that US robotics companies need protection from Chinese competition to create a more robust and secure domestic supply chain. On its face, it’s a strategy to align political and industry interests that is much older than the Trump administration. Whenever China has gotten good at offering cheap versions of strategic technologies like solar panels, electric vehicles, and drones, the US government has tried to stop it from flooding the market by using tariffs or rules on how government agencies purchase the tech. Such moves are always followed by debates about whether the trade-offs—particularly higher prices for consumers—are worth the benefits. But robotics is now best seen as another piece of the AI industry—in many ways its cutting edge. And the Trump administration is taking an increasingly aggressive approach to protecting the US AI industry, reportedly considering a ban on open-source Chinese models that often rival those from OpenAI and Anthropic while costing far less. Such a move would block businesses from realizing an estimated $25 billion in annual savings. The ban on humanoids, then, should be understood not as another chapter in the old China trade playbook, but as evidence that the Trump administration is expanding its protection of the AI industry beyond today’s leading labs. It is now willing to step in on behalf of an emerging robotics sector that is still barely finding its footing. Some US robotics companies unsurprisingly welcome the FTC’s new move. Gavin Kenneally, CEO of a company called Ghost Robotics that makes four-legged robots for inspections, says the cybersecurity risks from foreign-made robots are real (an FTC document released as part of the ruling cited an incident in which a man was able to gain control of 7,000 robot vacuum cleaners). “If today’s announcement encourages stronger cybersecurity and a more level competitive environment, that’s good for customers and good for the robotics industry,” Kenneally said in an email. But if the new rule aims to boost US robotics companies, there’s a big flaw. Those companies, as well as academic robotics labs, are hugely reliant on cheap robots from China to do research. They’re building fleets of robots that constantly learn new tasks—from flipping waffles to doing laundry—and frequently buy Chinese humanoids instead of US-made ones. The new ruling “creates a challenge for US humanoid researchers,” says Aaron Prather, director of market intelligence for the Association for Advancing Automation, a robotics trade group. “Chinese models offer the best price-to-capability ratio available.” Prather adds that a recent internal review his organization conducted found that 90% of recent robotics research papers from US universities relied on robots from Unitree, China’s top humanoid robotics company. That price gap can be huge. A four-legged robot from Unitree can cost around $4,600. A comparable one from Boston Dynamics might run to $278,000. If robotics research is stunted because these cheap robots are no longer available, the FTC ruling could slow down the industry, not boost it. The US and Chinese robotics industries are in starkly different places. Unitree plans to go public this week, targeting a nearly $6 billion evaluation. No robotics companies in the US offer any meaningful comparison, but those that do exist are undeniably moving fewer robots. Figure’s humanoids are not yet selling at scale, and 1X’s robots aren’t yet shipping to homes. That said, work on humanoids is going increasingly mainstream, as a release from Google last week made clear. The company announced a new AI model meant to make humanoids learn new tasks faster; its most impressive ability appears to be tying a trash bag, but given how finicky robot hands are, that’s real progress.  Even though the many carve-outs in the FTC’s order make its practical impact hard to predict, its symbolic impact is easy to see. The administration sees humanoid robotics not as a novelty, but as a strategic frontier of AI worth protecting from foreign competition. For a technology that until recently was mostly known for falling over onstage, that’s a big change.

In this tutorial, we build an advanced end-to-end time-series forecasting workflow with TimesFM 2.5. We begin by configuring the runtime, installing the required dependencies, detecting available hardware, and generating a realistic multi-store retail dataset with trend, seasonality, pricing, promotions, holidays, temperature effects, and random variation. We then load and compile the TimesFM 2.5 model, examine its forecast configuration, and use it for zero-shot point and probabilistic forecasting. As we progress, we evaluate forecast quality with metrics such as MAE, RMSE, sMAPE, MASE, pinball loss, and prediction-interval coverage, while also testing batched inference, rolling-origin backtesting, context-length sensitivity, covariate integration through XReg, anomaly detection, long-horizon forecasting, throughput tuning, and input robustness. By working through these stages, we develop a practical understanding of how we configure, validate, benchmark, and deploy TimesFM for realistic forecasting tasks. Copy CodeCopiedUse a different Browser FAST_MODE = False SEED = 7 import subprocess, sys, os, time, json, math, warnings warnings.filterwarnings(“ignore”) def _pip(*args): subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, *args]) try: import timesfm except ImportError: print(“Installing timesfm[torch] … (~1-2 min)”) _pip(“timesfm[torch]”) import timesfm import numpy as np import pandas as pd import torch import matplotlib.pyplot as plt import matplotlib.dates as mdates np.random.seed(SEED) torch.manual_seed(SEED) torch.set_float32_matmul_precision(“high”) DEVICE = “cuda” if torch.cuda.is_available() else “cpu” print(“=” * 78) print(f”timesfm : {getattr(timesfm, ‘__version__’, ‘n/a’)}”) print(f”torch : {torch.__version__}”) print(f”device : {DEVICE}”) if DEVICE == “cuda”: print(f”gpu : {torch.cuda.get_device_name(0)} ” f”({torch.cuda.get_device_properties(0).total_memory/1e9:.1f} GB)”) print(“=” * 78) try: import jax from sklearn import preprocessing HAS_XREG_DEPS = True except Exception as e: HAS_XREG_DEPS = False print(f”[warn] XReg deps missing ({e}); section 10 will be skipped.”) N_DAYS = 1200 N_STORES = 6 REGIONS = [“north”, “north”, “south”, “south”, “coast”, “coast”] dates = pd.date_range(“2021-01-01″, periods=N_DAYS, freq=”D”) t = np.arange(N_DAYS) dow = dates.dayofweek.values doy = dates.dayofyear.values temp_base = 18 + 12 * np.sin(2 * np.pi * (doy – 105) / 365.25) temp = temp_base + np.cumsum(np.random.normal(0, 0.6, N_DAYS)) * 0.15 temp = temp – np.linspace(0, temp[-1] – temp_base[-1], N_DAYS) holiday_doy = {1, 2, 45, 100, 120, 185, 240, 300, 358, 359, 360, 361, 362, 363, 364, 365} is_holiday = np.isin(doy, list(holiday_doy)).astype(int) rows = [] for s in range(N_STORES): level = 180 + 60 * s slope = np.random.uniform(0.02, 0.09) week_amp = np.random.uniform(15, 35) year_amp = np.random.uniform(20, 45) elasticity = np.random.uniform(18, 32) promo_lift = np.random.uniform(35, 70) temp_beta = np.random.uniform(0.8, 2.2) phase = np.random.uniform(0, 2 * np.pi) base_price = np.random.uniform(9.0, 13.0) price = base_price + np.random.normal(0, 0.25, N_DAYS) promo = (np.random.rand(N_DAYS) < 0.09).astype(int) price = price – promo * np.random.uniform(1.2, 2.2) weekly = week_amp * np.array([0.9, 0.7, 0.7, 0.85, 1.25, 1.8, 1.5])[dow] yearly = year_amp * np.sin(2 * np.pi * doy / 365.25 + phase) sales = (level + slope * t + weekly + yearly – elasticity * (price – base_price) + promo_lift * promo + 55 * is_holiday + temp_beta * (temp – 18) + np.random.normal(0, 14, N_DAYS)) sales = np.clip(sales, 5, None) rows.append(pd.DataFrame({ “date”: dates, “store”: f”store_{s}”, “region”: REGIONS[s], “sales”: sales.astype(np.float32), “price”: price.astype(np.float32), “promo”: promo.astype(np.int32), “holiday”: is_holiday.astype(np.int32), “dow”: dow.astype(np.int32), “temp”: temp.astype(np.float32), })) df = pd.concat(rows, ignore_index=True) STORES = sorted(df[“store”].unique()) print(f”nDataset: {df.shape[0]:,} rows | {len(STORES)} stores | ” f”{dates[0].date()} → {dates[-1].date()}”) print(df.head(3).to_string(index=False)) wide = df.pivot(index=”date”, columns=”store”, values=”sales”) SEASON = 7 HORIZON = 56 print(“nLoading google/timesfm-2.5-200m-pytorch …”) t0 = time.time() model = timesfm.TimesFM_2p5_200M_torch.from_pretrained( “google/timesfm-2.5-200m-pytorch” ) print(f”loaded in {time.time() – t0:.1f}s”) BASE_CFG = dict( max_context=1024, max_horizon=256, normalize_inputs=True, per_core_batch_size=16, use_continuous_quantile_head=True, force_flip_invariance=True, infer_is_positive=True, fix_quantile_crossing=True, return_backcast=False, ) model.compile(timesfm.ForecastConfig(**BASE_CFG)) print(“compiled:”, {k: v for k, v in BASE_CFG.items() if k in (“max_context”, “max_horizon”, “per_core_batch_size”)}) def recompile(**overrides): cfg = {**BASE_CFG, **overrides} model.compile(timesfm.ForecastConfig(**cfg)) return cfg We configure the Google Colab environment, install TimesFM and its supporting libraries, detect the available CPU or GPU, and initialize reproducible random seeds. We generate a realistic multi-store retail dataset containing trends, weekly and yearly seasonality, pricing effects, promotions, holidays, temperature variations, and random demand noise. We then load the TimesFM 2.5 model, define its baseline forecast configuration, compile it, and create a reusable function for changing model settings in later experiments. Copy CodeCopiedUse a different Browser IDX_MEAN, IDX_Q10, IDX_Q50, IDX_Q90 = 0, 1, 5, 9 QUANTILES = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] target_store = STORES[0] series = wide[target_store].values.astype(np.float32) train, actual = series[:-HORIZON], series[-HORIZON:] point, quant = model.forecast(horizon=HORIZON, inputs=[train.copy()]) print(f”npoint {point.shape} # (n_series, horizon)”) print(f”quantile {quant.shape} # (n_series, horizon, 10)”) fig, ax = plt.subplots(figsize=(14, 5)) hist_n = 180 ax.plot(dates[-HORIZON – hist_n:-HORIZON], train[-hist_n:], color=”#334155″, lw=1.2, label=”history”) ax.plot(dates[-HORIZON:], actual, color=”#0f172a”, lw=1.6, label=”actual”) ax.plot(dates[-HORIZON:], point[0], color=”#ea580c”, lw=2, label=”TimesFM median”) for lo, hi, a in [(1, 9, .12), (2, 8, .16), (3, 7, .20), (4, 6, .24)]: ax.fill_between(dates[-HORIZON:], quant[0, :, lo], quant[0, :, hi], color=”#ea580c”, alpha=a, lw=0) ax.axvline(dates[-HORIZON], color=”#94a3b8″, ls=”–“, lw=1) ax.set_title(f”TimesFM 2.5 zero-shot — {target_store}, {HORIZON}-day horizon ” f”(fan = q10…q90)”) ax.legend(loc=”upper left”) ax.xaxis.set_major_formatter(mdates.DateFormatter(“%b %Y”)) plt.tight_layout() plt.show() print(“n— output anatomy —“) print(“index 0 = MEAN (not q0!). indices 1..9 = q10..q90. index 5 = median.”) print(“point_forecast is literally quantile[…, 5]:”, np.allclose(point, quant[…, IDX_Q50])) print(“monotone quantiles (fix_quantile_crossing):”, bool(np.all(np.diff(quant[0, :, 1:], axis=-1) >= -1e-4))) row = pd.DataFrame({ “index”: range(10), “meaning”: [“mean”] + [f”q{int(q*100)}” for q in QUANTILES], “day+1″: quant[0, 0].round(1), f”day+{HORIZON}”: quant[0, -1].round(1), }) print(row.to_string(index=False)) print(“Interval width grows with horizon — day+1 q10..q90 span ” f”{quant[0,0,9]-quant[0,0,1]:.1f}, day+{HORIZON} span ” f”{quant[0,-1,9]-quant[0,-1,1]:.1f}”) def seasonal_naive(history, horizon, season=SEASON): “””Repeat the last full season forward — the baseline you must beat.””” reps = int(np.ceil(horizon / season)) return np.tile(history[-season:], reps)[:horizon] def pinball(actual, q, quantiles=QUANTILES): “””Mean pinball (quantile) loss over q10..q90 — the probabilistic metric.””” out = [] for i, tau in enumerate(quantiles, start=1): e = actual – q[:, i] out.append(np.mean(np.maximum(tau * e, (tau – 1) * e))) return float(np.mean(out)) def evaluate(actual, pred, history, q=None, season=SEASON): actual, pred = np.asarray(actual, float), np.asarray(pred, float) err = actual – pred scale = np.mean(np.abs(history[season:] – history[:-season])) + 1e-9 m = { “MAE”: float(np.mean(np.abs(err))), “RMSE”: float(np.sqrt(np.mean(err ** 2))), “MAPE%”: float( np.mean(np.abs(err / np.maximum(np.abs(actual), 1e-9))) * 100 ), “sMAPE%”: float( np.mean( 2 * np.abs(err) / (np.abs(actual) + np.abs(pred) + 1e-9) ) * 100 ), “MASE”: float(np.mean(np.abs(err)) / scale), } if q is not None: m[“pinball”] = pinball(actual, q) m[“cov80%”] = float( np.mean( (actual >= q[:, IDX_Q10]) & (actual <= q[:, IDX_Q90]) ) * 100 ) return m base_pred =

DeepSeek published DeepSeek-V4-Flash-0731 on Hugging Face and moved the official V4-Flash API into public beta on July 31, 2026. The model card is explicit that this is the official release superseding the preview, and that the architecture and size are unchanged. The gains come from re-post-training, not a new design. The checkpoint ships with the DSpark speculative decoding module attached, matching the structure of DeepSeek-V4-Flash-DSpark. Hugging Face reports 304B parameters for the repo, which includes that draft module on top of the 284B base. On the API side, deepseek-v4-flash now natively supports the Responses API format and is adapted for Codex. The V4-Pro API and the app and web models were not updated. Is it deployable? Yes, in two very different ways. Via API, it is deployable by almost anyone: DeepSeek’s pricing page lists deepseek-v4-flash at $0.14 per 1M input tokens on a cache miss, $0.0028 on a cache hit, and $0.28 per 1M output tokens, with a 2,500 concurrency limit. That is roughly a third of deepseek-v4-pro output pricing ($0.87). Seed-stage startups, indie developers, and internal platform teams can run agent loops at this price without a GPU budget. Via self-hosting, the bar is much higher: The weights are MIT-licensed and ungated, but every expert stays resident in memory even though only 13B activate per token. DeepSeek’s vLLM example serves it on a single 4×GB300 node. Unsloth’s dynamic GGUFs put the lossless 8-bit build at 162 GB and a 3-bit build at 103 GB, needing roughly 110 GB of combined RAM plus VRAM. Self-hosting suits mid-size and large enterprises with a serving cluster, or one well-specced workstation at aggressive quantization. Architecture Per the DeepSeek-V4 technical report, V4-Flash is a 284B-parameter MoE with 13B activated per token and a 1M-token context window. Each MoE layer holds 1 shared expert and 256 routed experts with an intermediate dimension of 2048, and 6 routed experts fire per token. The first three MoE layers use hash routing. Multi-token prediction depth is 1. Attention is hybrid, combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). Manifold-Constrained Hyper-Connections (mHC) replace conventional residual connections, with expansion factor 4 and 20 Sinkhorn-Knopp iterations. Pre-training used more than 32T tokens and the Muon optimizer. The paper’s headline efficiency figure — 27% of single-token inference FLOPs and 10% of KV cache versus DeepSeek-V3.2 at 1M context — is stated for V4-Pro, not Flash. <!– EMBED HERE: paste wordpress-embed.html into a Custom HTML block –> Benchmarks All figures below are DeepSeek-reported, from the 0731 model card. Benchmark V4-Flash-0731 V4-Flash (Preview) V4-Pro (Preview) GLM-5.2 Opus-4.8 Terminal Bench 2.1 82.7 61.8 72.1 81.0 85.0 NL2Repo 54.2 39.4 38.5 48.9 69.7 Cybergym 76.7 38.7 52.7 — 83.1 DeepSWE 54.4 7.3 12.8 46.2 58.0 Toolathlon-Verified 70.3 49.7 55.9 59.9 76.2 Agents’ Last Exam 25.2 15.8 16.5 23.8 25.7 AutomationBench Public 25.1 10.8 12.8 12.9 27.2 Two important things to note: Code Agent tasks were run with the minimal mode of DeepSeek Harness, which has not been released. DSBench-FullStack (68.7) and DSBench-Hard (59.6) are internal test sets. Agent scores are harness-sensitive, so independent runs may diverge. Serving it DSpark is enabled with one vLLM flag: –speculative-config ‘{“method”:”dspark”,”num_speculative_tokens”:7,”draft_sample_method”:”greedy”}’. The DSpark paper reports 60–85% faster per-user generation on V4-Flash versus the MTP-1 baseline at matched aggregate throughput. There is no Jinja chat template. DeepSeek ships an encoding/ folder with encode_messages and parse_message_from_completion_text instead. reasoning_effort takes low, high, or max. DeepSeek recommends temperature = 1.0, top_p = 0.95 for agentic use and 1.0 otherwise, with up to 384K output tokens at high and max. Key Takeaways Same 284B/13B architecture as the April preview: the jump is post-training only. Beats V4-Pro (Preview) on every agentic benchmark DeepSeek published, at a third of the output price. MIT-licensed and ungated, so on-premise commercial deployment is unblocked. Self-hosting needs ~110 GB memory at 3-bit, or a 4×GB300 node for full-precision serving. All benchmark numbers are vendor-reported on an unreleased harness — run your own evals first. Check out the Model Update on HF. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post DeepSeek Upgrades DeepSeek-V4-Flash-0731 with Major Agentic and Coding Gains appeared first on MarkTechPost.

Supabase has open sourced Supabase Evals, its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, then scores the result. It powers the public leaderboard at supabase.com/evals and an internal regression suite monitored daily. Is it deployable? Yes, today. supabase/evals is public under Apache-2.0 and runs locally via pnpm. Industries: Developer tooling, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing a wrong RLS policy is a security incident. Applications: Regression-testing docs and skill edits, gating SDK releases, and comparing agent harnesses head to head. Constraints: Local-stack runs need a Docker daemon, provider API keys, and ports 54321–54329 free. How the harness works Supabase defined three dimensions: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema), and stages (build, deploy, investigate, resolve). It then picked the smallest scenario set touching each dimension once, grounded in support tickets, bug reports, and GitHub issues. Scenarios split into two suites. Benchmark scenarios cover breadth and are published. Regression scenarios cover known failure modes, refresh daily, and do not move published scores. Every scenario runs against a real environment. The framework boots a hosted-like stack and a local CLI project in containers, so agents call the actual MCP server and CLI. A platform-lite runtime exposes a Management API-compatible surface backed by @supabase/lite. Scoring combines deterministic checks with LLM-as-a-judge. Agents get one retry before grading. Each eval directory holds PROMPT.md (task plus frontmatter), EVAL.ts (the scorer), and optional remote/ and local/ starting states. Shipping a local/ workspace, or declaring interface: cli, boots a Docker sandbox with the real CLI installed. Run the pipeline</button> </div> <!– ANATOMY –> <div class=”pane” id=”p2″> <div class=”hint”>Every eval lives at <b>evals/&lt;id&gt;/</b>. Click a file to see what it holds.</div> <div class=”tree” id=”tree”></div> <div class=”detail” id=”det2″></div> </div> <!– RUNTIMES –> <div class=”pane” id=”p3″> <div class=”hint”>The harness picks a runtime <b>automatically</b>, per eval. Toggle to compare.</div> <div class=”tog”> <button class=”tg on” data-r=”0″>Tools evals</button> <button class=”tg” data-r=”1″>Local-stack evals</button> </div> <div class=”lanes” id=”lanes”></div> <div class=”note” id=”rnote”></div> </div> <!– FINDINGS –> <div class=”pane” id=”p4″> <div class=”hint”>Published <b>Build stage</b> pass rates. Toggle the Supabase agent skill on and off.</div> <div class=”tog”> <button class=”tg” data-s=”0″>No skill loaded</button> <button class=”tg on” data-s=”1″>Skill loaded</button> </div> <div class=”rows” id=”rows”></div> <div class=”note”> <b>Also measured:</b> Codex / GPT-5.6 reads about 8 docs pages per scenario, versus roughly 2 for Claude Code, which checks the docs in under 40% of scenarios even with skills loaded. Rewriting the Postgres best-practices skill description lifted its activation from about 1 in 10 sessions to 60%.<br><br> Figures are a snapshot from Supabase’s launch post (31 Jul 2026). Results move as models change — check the live page. </div> </div> <div class=”ft”> <span>Source: <a href=”https://supabase.com/blog/introducing-supabase-evals” target=”_blank” rel=”noopener”>Supabase blog</a> · <a href=”https://github.com/supabase/evals” target=”_blank” rel=”noopener”>supabase/evals</a> · Apache-2.0</span> <span><b>Marktechpost</b></span> </div> </div> <script> var STAGES=[ {i:”“,l:”Scenario”,h:”1 · A real scenario”,t:”Each eval is grounded in a real problem — a support ticket, bug report, or GitHub issue. <code>PROMPT.md</code> carries the task the agent sees plus frontmatter tagging its stage, product, and topic.”}, {i:”“,l:”Environments”,h:”2 · Two real environments”,t:”The framework boots a hosted-like Supabase stack and a local CLI project in containers. <code>platform-lite</code> serves a Management API-compatible surface backed by <code>@supabase/lite</code>.”}, {i:”“,l:”Agent runs”,h:”3 · The agent works”,t:”Claude Code, Codex, OpenCode, or an AI SDK agent invokes the real Supabase MCP server and CLI — not mocks. Skills load lazily: only name and description sit in the system prompt.”}, {i:”“,l:”One retry”,h:”4 · One retry allowed”,t:”To cut false negatives while keeping runs sustainable, agents may retry once after a failure before they are graded.”}, {i:”“,l:”Scoring”,h:”5 · Deterministic + judge”,t:”<code>EVAL.ts</code> exports the scorer. Deterministic checks confirm things like whether a user can reach certain data or an Edge Function returns the expected result; an LLM judge handles semantic calls.”}, {i:”“,l:”Results”,h:”6 · Benchmark or regression”,t:”Benchmark scenarios go to the public site and run when assessing new changes or harnesses. Regression scenarios track known failure modes and refresh daily, without moving published scores.”} ]; var FILES=[ {n:”PROMPT.md”,d:”Task + frontmatter”,h:”PROMPT.md”,t:”Frontmatter plus the task description the agent sees. Keys drive discovery and the site filters: <code>stage</code>, <code>suite</code>, <code>product</code>, <code>topic</code>, <code>motivation</code>. <code>suite</code> is required on every eval.”}, {n:”EVAL.ts”,d:”The scorer”,h:”EVAL.ts”,t:”A default-exported scorer. Scorers check what the agent produced, never what the harness provisioned — with <code>projectRunning: true</code>, only the agent’s deltas are scored.”}, {n:”remote/”,d:”Hosted project state”,h:”remote/ — optional”,t:”The hosted project’s starting state, seeded into platform-lite: <code>project.sql</code> for the database, <code>logs.jsonl</code> for observability logs, and <code>functions/</code> for already-deployed Edge Functions.”}, {n:”local/”,d:”Agent workspace”,h:”local/ — optional”,t:”The developer’s working directory, copied into the sandbox before the agent starts. Its presence is also a runtime switch: ship a <code>local/</code> and the eval boots a Docker sandbox.”} ]; var RUN=[ {lanes:[[“PROMPT”,”Agent gets the task, with no local/ directory and no interface: cli”],[“TOOLS”,”It works through the experiment’s MCP / tool surface only — there is no filesystem”],[“SKILLS”,”A load_skill tool returns a skill’s full instructions on demand”],[“SCORE”,”The resulting project state or report is graded”]], note:”<b>Tools evals</b> exercise the MCP surface in isolation. Because the agent has no filesystem, skills are fetched through a tool call rather than read from disk.”}, {lanes:[[“PROMPT”,”Eval ships a local/ workspace or declares interface: cli”],[“SANDBOX”,”A fresh Docker container boots per attempt, with the real Supabase CLI installed”],[“STACK”,”The agent runs supabase init / start / db / test against a live local stack”],[“EXPORT”,”The workspace is copied back to the host so scorers run vite / vitest against it”]], note:”<b>Local-stack evals</b> need a running Docker daemon, and default ports 54321–54329 free. A <code>services:</code> list keeps stack boots fast by starting only what the scenario needs.”} ]; var SCORES=[ {n:”Opus 5″,off:100,on:100}, {n:”Kimi K3″,off:100,on:100}, {n:”GPT-5.6 Sol”,off:89,on:100}, {n:”Sonnet 5″,off:78,on:100}, {n:”GPT-5.4 mini”,off:78,on:89} ]; function $(s){return document.querySelector(s)} function all(s){return [].slice.call(document.querySelectorAll(s))} /* tabs */ all(‘.tab’).forEach(function(b){b.onclick=function(){ all(‘.tab’).forEach(function(x){x.classList.remove(‘on’)}); all(‘.pane’).forEach(function(x){x.classList.remove(‘on’)}); b.classList.add(‘on’);$(‘#’+b.dataset.p).classList.add(‘on’); }}); /* pipeline */ var flow=$(‘#flow’); STAGES.forEach(function(s,i){ var d=document.createElement(‘div’);d.className=’node’;d.dataset.i=i; d.innerHTML='<span class=”dot”></span><div class=”nnum”>0’+(i+1)+'</div><div class=”nico”>’+s.i+'</div><div class=”nlab”>’+s.l+'</div>’; d.onclick=function(){pick(i)};flow.appendChild(d); }); function pick(i){