Committee

In this tutorial, we use GEPA as a reflective prompt-evolution framework to improve the way a language model solves arithmetic word problems. We begin with a weak seed prompt, create a small deterministic benchmark, define a structured evaluator, and pass actionable feedback to GEPA so it can understand why a candidate prompt fails. We also use a multi-component prompt setup in which both the instruction field and the output-format rules evolve together. By the end, we compare the baseline prompt with the optimized prompt on a held-out validation set and inspect how the evolutionary process improves performance. Installing GEPA and LiteLLM and Configuring the Task and Reflection Models Copy CodeCopiedUse a different Browser !pip install -q gepa litellm import os, re, json, random, getpass, textwrap import litellm import gepa.optimize_anything as oa from gepa.optimize_anything import ( optimize_anything, GEPAConfig, EngineConfig, ReflectionConfig, ) litellm.suppress_debug_info = True if not os.environ.get(“OPENAI_API_KEY”): os.environ[“OPENAI_API_KEY”] = getpass.getpass(“Enter your OpenAI API key: “) TASK_LM = “openai/gpt-4o-mini” REFLECTION_LM = “openai/gpt-4.1” MAX_METRIC_CALLS = 100 We install GEPA and LiteLLM, then import the required libraries for prompt optimization and model calls. We securely set up the OpenAI API key and define two models: a task model that solves the problem and a reflection model that improves the prompt. We also set the maximum metric-call budget to keep the optimization process under control. Building a Deterministic Arithmetic Benchmark Dataset Copy CodeCopiedUse a different Browser def make_problems(n, seed=0): rng = random.Random(seed) out = [] for _ in range(n): t = rng.choice([“discount”, “travel”, “wallet”, “chain”]) if t == “discount”: unit = rng.choice([40, 60, 80, 120]) qty = rng.choice([5, 6, 8, 10]) disc = rng.choice([10, 20, 25, 50]) total = unit * qty gold = total – total * disc // 100 q = (f”A shop sells notebooks at {unit} rupees each. You buy {qty} ” f”notebooks and get a {disc}% discount on the total bill. ” f”How many rupees do you pay in total?”) elif t == “travel”: s1, h1 = rng.choice([40, 50, 60]), rng.choice([2, 3]) s2, h2 = rng.choice([30, 45, 70]), rng.choice([1, 2, 3]) gold = s1 * h1 + s2 * h2 q = (f”A car drives at {s1} km/h for {h1} hours, then at {s2} km/h ” f”for {h2} hours. What is the total distance travelled, in km?”) elif t == “wallet”: tens = rng.choice([3, 5, 7, 9]) fifties= rng.choice([2, 4, 6]) spent = rng.choice([50, 80, 110, 150]) gold = tens * 10 + fifties * 50 – spent q = (f”You have {tens} ten-rupee notes and {fifties} fifty-rupee ” f”notes. You spend {spent} rupees. How many rupees are left?”) else: x = rng.choice([6, 9, 12, 15]); y = rng.choice([4, 7, 10]); z = rng.choice([3, 8, 11]) gold = x * 2 – y + z q = (f”Start with the number {x}. Double it, then subtract {y}, ” f”then add {z}. What number do you end with?”) out.append({“question”: q, “answer”: gold}) return out all_problems = make_problems(18, seed=42) random.Random(1).shuffle(all_problems) trainset = all_problems[:12] valset = all_problems[12:] print(f”Dataset: {len(trainset)} train / {len(valset)} val problemsn”) We create a small deterministic dataset of arithmetic word problems covering discounts, travel distance, wallet calculations, and chained operations. We generate the correct answer for each problem programmatically, which keeps the benchmark reliable and easy to evaluate. We then shuffle the examples and split them into a training set for optimization and a validation set for testing generalization. Defining the Evaluator and Structured Feedback for GEPA Copy CodeCopiedUse a different Browser def build_system_prompt(candidate: dict) -> str: return (f”{candidate[‘instructions’]}nn” f”OUTPUT FORMAT RULES:n{candidate[‘format_rules’]}”) def call_task_lm(system_prompt: str, question: str) -> str: for attempt in range(3): try: r = litellm.completion( model=TASK_LM, messages=[{“role”: “system”, “content”: system_prompt}, {“role”: “user”, “content”: question}], temperature=0, max_tokens=600, timeout=60, ) return r[“choices”][0][“message”][“content”] or “” except Exception as e: if attempt == 2: return f”[LM_ERROR] {e}” return “” def parse_answers(text: str): formatted = re.search(r”####s*(-?d+)”, text) all_nums = re.findall(r”-?d+”, text) fmt_val = int(formatted.group(1)) if formatted else None last_val = int(all_nums[-1]) if all_nums else None return fmt_val, last_val def evaluate(candidate: dict, example: dict): system = build_system_prompt(candidate) raw = call_task_lm(system, example[“question”]) gold = example[“answer”] fmt_val, last_val = parse_answers(raw) if fmt_val is not None and fmt_val == gold: score, fb = 1.0, “Correct and correctly formatted.” elif fmt_val is not None and fmt_val != gold: score, fb = 0.0, (f”WRONG ANSWER. You output ‘#### {fmt_val}’ but the ” f”correct answer is {gold}. Re-check the arithmetic and ” f”the order of the steps.”) elif last_val == gold: score, fb = 0.5, (f”Right number ({gold}) but FORMAT VIOLATION: the final ” f”line was not exactly ‘#### {gold}’. Always end with a ” f”line of the form ‘#### <integer>’ and nothing else.”) else: score, fb = 0.0, (f”WRONG. Correct answer is {gold}. The model’s final ” f”number was {last_val}. Likely a multi-step reasoning ” f”slip; show each step and verify before answering.”) oa.log(f”score={score} gold={gold} parsed_fmt={fmt_val} parsed_last={last_val}”) side_info = { “feedback”: fb, “problem”: example[“question”], “gold_answer”: gold, “model_output”: raw[:500], } return score, side_info def eval_set(candidate, dataset, label=””): scores, exact, formatted = [], 0, 0 for ex in dataset: s, info = evaluate(candidate, ex) scores.append(s) if s == 1.0: exact += 1; formatted += 1 elif s == 0.5: formatted += 0 acc = exact / len(dataset) avg = sum(scores) / len(dataset) print(f” [{label}] avg_score={avg:.3f} exact_correct+formatted={exact}/{len(dataset)}”) return avg, acc We define how the candidate prompt is converted into a system prompt and how the task model receives each question. We also create the evaluator that parses the model output, checks whether the final answer follows the required #### <integer> format, and assigns a score. We return structured feedback as actionable side information so that GEPA can determine whether the issue is incorrect reasoning, poor formatting, or both. Configuring GEPA and Running the Prompt Optimization Copy CodeCopiedUse a different Browser seed_candidate = { “instructions”: “Solve the math problem.”, “format_rules”: “Give the answer.”, } print(“=== BASELINE (seed prompt) ===”) print(“Train:”); base_train = eval_set(seed_candidate, trainset, “train”) print(“Val: “); base_val = eval_set(seed_candidate, valset, “val”) print() objective = ( “Evolve a system prompt (the ‘instructions’ and ‘format_rules’ fields) so a ” “small LLM reliably solves multi-step

Google DeepMind released Quantization-Aware Training (QAT) checkpoints for the Gemma 4 family. The release targets local deployment on edge devices and consumer GPUs. It follows the Gemma 4 launch in April and a 12B model two days earlier. We compared the available Gemma 4 edge-model formats using only published numbers. The goal was simple. Show what each precision level costs in memory. Then show what QAT actually changes. What QAT actually does Quantization shrinks a model by lowering weight precision. Standard Post-Training Quantization (PTQ) compresses a finished model. That often degrades quality. QAT instead simulates quantization during training. The model learns to compensate for the precision loss. Google’s AI team states its QAT results yield higher overall quality than standard PTQ baselines. Google did not publish Gemma 4 QAT benchmark scores in the announcement. For context, Gemma 3 QAT cut the Q4_0 perplexity drop by 54% using llama.cpp evaluation. We cite that only as prior-generation precedent. The comparison task Compare Gemma 4 E2B and E4B across three formats. The formats are BF16, Q4_0 QAT, and the new mobile QAT schema. Rank them on memory footprint, quality preservation, and on-device accessibility. Use published figures only. Memory results Format E2B E4B Basis BF16 (16-bit) 9.6 GB 15 GB Official Gemma 4 docs Q4_0 (4-bit, QAT) 3.2 GB 5 GB Official Gemma 4 docs Mobile (QAT, E2B) ~1 GB — QAT announcement The Q4_0 figures match the footprint of PTQ Q4_0. QAT does not change the size at a given format. It improves quality at that size. The new mobile schema delivers the additional reduction. Using that mobile schema, Google reduced Gemma 4 E2B to about 1GB. Developers can go lower still. The text-only model without Per-Layer Embeddings needs under 1GB, dropping the audio and vision encoders. Per-format breakdown BF16 is the quality baseline. E2B needs 9.6 GB and E4B needs 15 GB. It is the reference point, not a phone deployment target. Q4_0 QAT is the general-purpose local format. E2B drops to 3.2 GB and E4B to 5 GB. QAT preserves more quality here than PTQ at the same size. This format fits consumer GPUs. Earlier E2B testing also ran on a Raspberry Pi 5 at INT4. The mobile format is the edge-specialized schema. It brings E2B to about 1 GB. It uses static activations, channel-wise quantization, and targeted 2-bit compression. How the mobile schema works Google AI team engineered four techniques for mobile hardware. Static activations pre-calculate scaling during training, reducing on-device work. Channel-wise quantization fits the design of mobile accelerators. Targeted 2-bit quantization compresses only the token-generation layers. Embedding and KV cache optimization shrinks the active memory footprint. Core reasoning layers stay at higher precision. That protects capability while cutting storage. Developers can also deploy text-only and drop the audio and vision encoders. That trims memory further for use cases that need no multimodality. Dimension breakdown Scores are a qualitative ranking of the formats for on-device use. Memory is the only hard-measured axis. Quality reflects Google’s disclosed design, not measured Gemma 4 numbers. Each score has a one-line basis. Dimension BF16 Q4_0 QAT Mobile QAT Memory footprint 1 — heaviest, 9.6 GB E2B 4 — 3.2 GB E2B 5 — ~1 GB E2B text-only Quality preservation 5 — full-precision baseline 4 — QAT-preserved, near baseline 3 — 2-bit token layers, core kept higher Decode speed 2 — no quantization speedup 4 — 4-bit accelerates decode 5 — mobile-optimized static activations Deployment breadth 4 — loadable but heavy 5 — llama.cpp, Ollama, LM Studio, vLLM, MLX 3 — LiteRT-LM, Transformers.js, edge-focused On-device accessibility 1 — needs large GPU 4 — consumer GPU, Raspberry Pi 5 5 — runs on phones Total (/25) 13 21 21 Winner The result is a tie by design. Q4_0 QAT and mobile QAT both score 21, but for different hardware. For phones, the mobile format leads. It reaches about 1GB on E2B and targets mobile accelerators directly. For laptops and consumer GPUs, Q4_0 QAT is the practical default. BF16 stays the quality reference, not a local choice. Methodology and limits Memory figures come from Google’s Gemma 4 documentation. The ~1GB E2B figure comes from the QAT announcement. Quality is Google’s stated claim. No independent Gemma 4 QAT quality numbers were published at release. We did not run the models locally for this comparison. Developers should test at their own quantization and workload before building. Key Takeaways Q4_0 QAT cuts Gemma 4 E2B to 3.2 GB and E4B to 5 GB, from 9.6 GB and 15 GB at BF16. A new mobile QAT schema brings E2B to about 1 GB; text-only without PLE goes under 1 GB. QAT changes quality at a given size, not the size itself; the mobile format drives the extra memory cut. Google claims higher quality than PTQ but published no Gemma 4 QAT benchmark numbers at release. Weights ship today on Hugging Face with llama.cpp, Ollama, LM Studio, vLLM, MLX, and LiteRT-LM support. Marktechpost’s Visual Explainer Marktechpost · Benchmark Gemma 4 QAT: Comparing Q4_0 and the New Mobile Format Google DeepMind released Quantization-Aware Training checkpoints for Gemma 4. We compared three edge-model formats on published numbers. Formats compared BF16 (16-bit)  ·  Q4_0 QAT (4-bit)  ·  Mobile QAT June 5, 2026 The Comparison Task What we ranked $ compare gemma-4 –models E2B,E4B –formats BF16,Q4_0-QAT,MOBILE-QAT –rank memory,quality,accessibility –source published-only –no-self-run Memory from official Gemma 4 docs. Quality from Google’s stated claim. No models run locally. Format 1 of 3 · Reference BF16 (16-bit) 13 / 25 The full-precision quality baseline. E2B needs 9.6 GB and E4B needs 15 GB. Top observation: a reference point, not a phone or laptop deployment target. Format 2 of 3 · Laptop / GPU Q4_0 QAT (4-bit) 21 / 25 The general-purpose local format. E2B drops to 3.2 GB and E4B to 5 GB. Top observation: QAT preserves more quality than PTQ at the same 4-bit size. Format 3 of 3 · Mobile Mobile QAT 21 / 25 The edge-specialized schema. Brings E2B to about 1 GB. Top

In this tutorial, we work through an end-to-end workflow for Qualcomm AI Hub Models. We start by setting up the required package, discovering the available model collection, and loading MobileNet-V2 for local PyTorch inference. We also handle an important input-shape issue by converting NHWC image tensors into the NCHW format expected by the model. From there, we run inference on both the model’s built-in sample input and a real image, inspect top predictions, execute the official Qualcomm AI Hub CLI demo, and extend the workflow with a YOLOv7 object detection example. Also, we include an optional cloud-device section where we compile, profile, and run the model on a real Qualcomm device when an API token is available. Copy CodeCopiedUse a different Browser import subprocess, sys, os, glob, textwrap, traceback import numpy as np, torch from PIL import Image import matplotlib.pyplot as plt def pip_install(*pkgs): subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, *pkgs], check=True) pip_install(“qai_hub_models”) OUT_DIR = “/content/qaihm_out”; os.makedirs(OUT_DIR, exist_ok=True) torch.set_grad_enabled(False) def to_nchw(value): arr = value[0] if isinstance(value, (list, tuple)) else value t = torch.from_numpy(np.asarray(arr, dtype=np.float32)) if t.ndim == 3: t = t.unsqueeze(0) if t.ndim == 4 and t.shape[1] != 3 and t.shape[-1] == 3: t = t.permute(0, 3, 1, 2).contiguous() return t We begin by importing libraries and setting up a helper function to install packages directly inside Colab. We install qai_hub_models, create an output directory, and disable gradient tracking since we only need inference. We also define the to_nchw() function to convert any input image tensor to the channel-first format expected by the model. Copy CodeCopiedUse a different Browser import pkgutil, qai_hub_models.models as _m model_ids = sorted(n for _, n, p in pkgutil.iter_modules(_m.__path__) if p and not n.startswith(“_”)) print(f”>>> {len(model_ids)} models available. First 40:n”) print(textwrap.fill(“, “.join(model_ids[:40]), 100), “n”) from qai_hub_models.models.mobilenet_v2 import Model as MobileNetV2 model = MobileNetV2.from_pretrained().eval() spec = model.get_input_spec() input_name = list(spec.keys())[0] print(“>>> Input:”, input_name, spec[input_name].shape, spec[input_name].dtype) from torchvision.models import MobileNet_V2_Weights IMAGENET_CLASSES = MobileNet_V2_Weights.IMAGENET1K_V1.meta[“categories”] def top5(logits): if logits.ndim == 1: logits = logits.unsqueeze(0) probs = torch.softmax(logits, dim=1)[0] conf, idx = probs.topk(5) return [(IMAGENET_CLASSES[i], float(c)) for c, i in zip(conf, idx)] We discover the available Qualcomm AI Hub model packages and print the first set of model IDs to understand what is accessible. We then load the pretrained MobileNet-V2 model, read its input specification, and identify the correct input name. We also prepare the ImageNet class labels and define a top5() function to convert model logits into readable top-5 predictions. Copy CodeCopiedUse a different Browser sample = model.sample_inputs() x = to_nchw(sample[input_name]) print(“>>> fed tensor shape:”, tuple(x.shape)) print(“n>>> Top-5 for the built-in sample input:”) for label, conf in top5(model(x)): print(f” {conf:6.2%} {label}”) from torchvision import transforms preprocess = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), ]) img = None try: import urllib.request p = os.path.join(OUT_DIR, “input.jpg”) urllib.request.urlretrieve( “https://raw.githubusercontent.com/pytorch/hub/master/images/dog.jpg”, p) img = Image.open(p).convert(“RGB”) except Exception as e: print(“>>> photo download skipped:”, e) if img is not None: preds = top5(model(preprocess(img).unsqueeze(0))) print(“n>>> Top-5 for the downloaded photo:”) for label, conf in preds: print(f” {conf:6.2%} {label}”) plt.figure(figsize=(5,5)); plt.imshow(img); plt.axis(“off”) plt.title(f”{preds[0][0]} ({preds[0][1]:.1%})”); plt.show() We first run inference using the model’s built-in sample input and use to_nchw() to fix the tensor shape before passing it to MobileNet-V2. We then download a real image, preprocess it using standard resizing, cropping, and tensor conversion steps, and run another prediction. We finally display the image with the top predicted label to visually connect the model output to the input photo. Copy CodeCopiedUse a different Browser def run_demo(module, extra=None, timeout=900): cmd = [sys.executable, “-m”, module, “–eval-mode”, “fp”, “–output-dir”, OUT_DIR] + (extra or []) print(f”n>>> {‘ ‘.join(cmd)}”) try: r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) print(“n”.join((r.stdout + r.stderr).strip().splitlines()[-25:])) except Exception as e: print(“>>> demo skipped:”, e) run_demo(“qai_hub_models.models.mobilenet_v2.demo”) try: pip_install(“qai_hub_models[yolov7]”) run_demo(“qai_hub_models.models.yolov7.demo”) imgs = sorted(glob.glob(OUT_DIR + “/*.png”) + glob.glob(OUT_DIR + “/*.jpg”), key=os.path.getmtime) if imgs: plt.figure(figsize=(9,9)); plt.imshow(Image.open(imgs[-1]).convert(“RGB”)) plt.axis(“off”); plt.title(“YOLOv7 detections”); plt.show() else: print(“>>> no output image found (results may have printed instead).”) except Exception: print(“>>> YOLOv7 section skipped:n”, traceback.format_exc()) We define a reusable run_demo() function that executes official Qualcomm AI Hub model demos from the command line. We use it to run the MobileNet-V2 demo and then install the YOLOv7 extras for object detection. We run the YOLOv7 demo, search for the generated output image, and visualize the detections if an image is created. Copy CodeCopiedUse a different Browser try: import qai_hub as hub devices = hub.get_devices() print(f”n>>> Authenticated. {len(devices)} cloud devices available.”) device = hub.Device(“Samsung Galaxy S24 (Family)”) sample = model.sample_inputs() nchw = to_nchw(sample[input_name]) traced = torch.jit.trace(model, [nchw]) cloud_inputs = {input_name: [nchw.numpy()]} cj = hub.submit_compile_job(model=traced, device=device, input_specs=model.get_input_spec(), options=”–target_runtime tflite”) target = cj.get_target_model(); print(“>>> compiled:”, cj.url) pj = hub.submit_profile_job(model=target, device=device); print(“>>> profiling:”, pj.url) ij = hub.submit_inference_job(model=target, device=device, inputs=cloud_inputs) out = ij.download_output_data() dev_logits = torch.from_numpy(np.asarray(list(out.values())[0][0])) print(“>>> Top-5 from the REAL device:”) for label, conf in top5(dev_logits): print(f” {conf:6.2%} {label}”) target.download(os.path.join(OUT_DIR, “mobilenet_v2.tflite”)) print(“>>> saved compiled .tflite to”, OUT_DIR) except Exception as e: print(“n>>> Cloud (on-device) section skipped — no API token configured.”) print(” Get one at workbench.aihub.qualcomm.com, then:”) print(” !qai-hub configure –api_token YOUR_TOKEN”) print(” detail:”, (str(e).splitlines() or [type(e).__name__])[0]) print(“n>>> Tutorial complete. Outputs in:”, OUT_DIR) We include an optional Qualcomm AI Hub cloud workflow that runs only when an API token is configured. We retrieve available cloud devices, trace the PyTorch model, compile it for TFLite, profile it on a Qualcomm device, and submit an inference job. We then download the device output, print the top predictions, save the compiled TFLite model, and finish by showing where all tutorial outputs are stored. In conclusion, we have a complete practical workflow for using Qualcomm AI Hub Models inside Colab. We learned how to load pretrained models, prepare inputs correctly, run local inference, visualize classification and detection results, and use the official demos as reproducible reference points. We also saw how the same model can move beyond local PyTorch execution into Qualcomm’s cloud-device pipeline for compilation, profiling, and real-device inference. It provides a path from simple experimentation to hardware-aware deployment with Qualcomm AI Hub. Check out the Full Codes with Notebook here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join

On June 5, 404 Media reported that attackers had been using Meta’s AI customer support agent to steal Instagram accounts. Their approach was simple: They asked the agent to link the accounts to email addresses that they controlled, and the agent complied. One attacker broke into the dormant Obama White House account and made pro-Iran posts; others took over accounts with valuable, single-word handles, possibly in order to sell them. AI cybersecurity concerns are nothing new. Since Anthropic announced in April that its Mythos model was too good at hacking to be released to the general public, commentators, researchers, and federal officials alike have fixated on the idea that superpowered AI systems could lay waste to our computer infrastructure. That’s not quite what this Instagram hack was: There, AI was the target rather than the attacker, and the method was far simpler than anything Mythos would cook up. But as companies offload more work to AI, these comparatively unsophisticated attacks could wreak their own havoc. “As AI becomes more and more widely used—especially when AI is more and more widely used to automate our work flows, like account recovery—I think attackers are going to be more and more motivated to attack AI itself,” says Neil Gong, a professor of electrical and computer engineering at Duke University. Gong and other scholars have been issuing warnings about the security vulnerabilities of AI agents for a while. They publish papers and blog posts detailing exploits such as indirect prompt injection, which involves hijacking agents using commands hidden in websites, emails, or other seemingly anodyne data sources. Compared with these techniques, the Meta hack was practically mindless. The only complication that hackers had to overcome was using a VPN that matched the true account owner’s location; then they directly asked the support agent to change the account’s email address, and it complied. Meta has not commented publicly on how this vulnerability slipped through the cracks. But given the simplicity of the exploit, Gong says, it should have been uncovered easily, before the agent was deployed. “It’s really surprising,” he says. “I don’t understand why they didn’t find this simple problem.” Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology, agrees. “It raises questions like: Were there even guardrails in place?” she says. “Did anyone think to test for this kind of scenario?” She notes that the oversight is particularly striking coming from a company like Meta, which has extensive expertise in both AI and cybersecurity. Meta did not respond to a request for comment for this article, but on Monday a Meta spokesperson said on X that the vulnerability had been resolved. As embarrassing a moment as this might be for Meta in particular, it also highlights some core vulnerabilities shared by all AI agents. Unlike traditional software, agents can respond in flexible—and unexpected—ways to new circumstances, which is why they might be able to substitute for human customer support agents. But AI agents can also be tricked in ways that humans wouldn’t be, and because they can take real-world actions, those mistakes have consequences. “A human would say, ‘Okay, why do you want to change the email address?’ and maybe respond with a security question,” says Somesh Jha, a professor of computer science at the University of Wisconsin–Madison. “What is going on with these agents is they’re very eager to finish the task. It’s almost like some elementary school student who just wants to please the teacher.” There are ways to mitigate the risks. Companies can use traditional software to build guardrails that make sure agents follow strict rules, such as always asking for answers to security questions before sending sensitive account information to a new email address. And the experts consulted for this article all agree that agents should undergo rigorous red-teaming, a process in which developers try their best to attack a system in order to discover its vulnerabilities before it is deployed. But there are also countervailing forces. Companies want to deploy capable agents, and the more power an agent has—and the fewer guardrails it is subject to—the more work it can potentially take on. “Security and utility always have a trade-off,” says Bo Li, a professor of computer science at the  University of Illinois Urbana-Champaign. And adequate red-teaming can be expensive. Defenders have to expend more resources than attackers do, because attackers only need to discover a single exploit, while defenders try to discover and patch as many as they can. When attackers are working toward something as valuable as a single-word Instagram handle, they’ll pour resources into finding exploits, so defenders have to spend even more money to protect that prize.  As AI models continue to improve, hardening their defenses might actually get easier. Though the probabilistic nature of large language models means that LLM agents will always be vulnerable to some forms of attack, a more sophisticated model might have identified an attempt to change the email associated with the Obama White House account as suspicious. And AI systems can be used for agent red-teaming, much as participants in Anthropic’s Project Glasswing use Mythos to identify vulnerabilities in their software.  Still, experts expect that the problem of securing AI agents will only become more pressing in the future. As agents grow more capable, companies that adopt them may want to give them more power, both to provide more services with fewer humans and to avoid being left behind by their competitors. In the fast-moving world of AI, the time needed to carefully secure risky agentic systems might seem like an unconscionable delay. “Everybody wants to be the first to do something and just push things out without careful scrutiny and red-teaming,” Jha says. “I think it’s a very dangerous thing.”

Perplexity AI announced what it calls the first hybrid local-server inference orchestrator at Computex 2026. The system is designed to automatically route AI tasks between a user’s local device and cloud-based frontier models without requiring the user to decide in advance. The feature is expected come to Perplexity Computer in July 2026. What is Hybrid Agentic Inference? To understand what Perplexity built, it helps to understand the three-way tension that AI systems face. Accuracy demands the most capable models, which are expensive to run. Privacy demands that some data never leave the device. Cost and energy efficiency demand that you don’t spend a frontier model’s compute on tasks a smaller model can handle. That routing layer is what Perplexity calls hybrid agentic inference. A compact AI model runs locally on the user’s device. This local model evaluates each incoming task or subtask. It determines whether the task involves sensitive data, whether it requires heavy computation, or whether it can be handled entirely on-device. Based on that evaluation, work is either kept local or sent to a frontier model in the cloud. Perplexity describes this local model as deciding “when sensitive data should also be kept locally.” The system is designed to ask for user permission before sending sensitive tasks to the cloud. That design addresses a specific concern enterprises have about agentic AI: data governance — knowing where data goes and who controls that decision. Examples of data the system is intended to keep local include financial records, health information, and personal files. Work that requires a frontier model’s full capability runs on the server. Most real tasks are a mix, so the system splits them and coordinates the parts. How It Fits into Perplexity Computer Perplexity Computer is the company’s cloud-based multi-model agentic product, launched in February 2026. It originally ran entirely in the cloud on the Perplexity Max subscription tier ($200/month). Personal Computer is a separate, related product that brought Computer’s capabilities onto the local device — with access to local files, native Mac apps, the web, and Perplexity’s secure servers. Personal Computer launched on Mac in April 2026. Windows support is planned; a waitlist is open. The new hybrid local-server inference orchestrator is the next step for Personal Computer. Previously, even within Personal Computer, the division was relatively fixed: local file access happened on-device, heavy computation ran on Perplexity’s servers. The orchestrator changes that. The system now reasons about where each piece of a task should execute — not just which model to use, but which physical location should process it. Perplexity Computer coordinates up to 20 AI models in a single workflow. The system is one that creates a team of agents and orchestrates across models, tools and files in one single system. The hybrid orchestrator extends that orchestration to compute location itself. Key Takeaways Perplexity AI announced the first hybrid local-server inference orchestrator at Computex 2026, routing AI tasks automatically between on-device and cloud models. A compact local model acts as the router — classifying each subtask by data sensitivity and compute requirements before dispatching it. Sensitive data (financial records, health files) stays on-device; compute-heavy tasks go to frontier cloud models — no manual configuration required. The orchestration framework is model-agnostic and chip-agnostic, confirmed to run on Intel Core Ultra Series 3 and NVIDIA RTX Spark hardware. The feature arrives in Perplexity Computer in July 2026, initially on Windows; Personal Computer is already available on Mac with a Windows waitlist open. Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Perplexity AI Introduces Hybrid Local-Server Inference Orchestrator for Personal Computer: Automatic On-Device and Cloud Task Routing appeared first on MarkTechPost.