AI

As of this week in Montana, any biotech company with an experimental drug has a clear path to selling it to consumers. Companies whose drugs have been through preliminary testing—sometimes in as few as 10 healthy people—can pay $12,500 to apply to a newly established review board for approval. Once its treatment is rubber-stamped, the company can set the price of the drug and sell it via experimental treatment clinics, the first of which is likely to be up and running around the end of this year.   Montana’s latest right-to-try legislation is unique. While other jurisdictions with similar laws limit access to drugs to people with terminal illness, in Montana access is theoretically available to anyone who gives informed consent and can pay. That includes people desperate for treatments for rare diseases. It also includes those who are interested in longevity and want to try out drugs pitched as preventive therapies. The state’s Department of Health and Human Services recently finalized rules to implement the law. The rules stipulate that patient consumers provide fully informed consent and that each application be reviewed by a board that includes a Montana-certified doctor, expert scientists, and an ethicist. Supporters of the law stress that they want the process to be responsible. “It will be done in a very rigorous way, with qualified medical professionals and appropriate oversight,” says Matt Kaeberlein, a scientist on the first board, which was formed independently of the state health department. But other experts are worried about the potential for harm in selling unproven treatments to people without oversight from the US Food and Drug Administration. “I would be concerned,” says Aaron Kesselheim, a professor of medicine at Harvard Medical School with expertise in health policy and drug regulation. There has been a growing movement to make unapproved drugs more accessible in the US. But the story of Montana’s law is unique. It’s been driven and drafted by longevity enthusiasts instead of the usual libertarian and patient groups. An unusual origin story Montana first passed a right-to-try law in 2015. In 2023, with the support of state senator Ken Bogner, the state expanded the law to include all patients, not those just with terminal disease. Last year, Bogner told MIT Technology Review that his vision was to focus “more on preventative medicine” rather than “just treating diseases once they show up.” Bogner says he had “started working on a bill” that would become the 2023 law when the Alliance for Longevity Initiatives (A4LI), a nonprofit “dedicated to advancing legislation and policies aimed at increasing healthy human lifespan,” got in touch. A4LI connected Bogner with others who helped draft the bill and testified in support of it. Once that law was in place, the tech entrepreneur and longevity enthusiast Niklas Anzinger got involved. Anzinger has been working to establish a jurisdiction to fast-track the search for drugs that might deliver radical life extension. He is based in Próspera—a private city and “special economic zone” in Roatán, Honduras, which is already home to a separate clinic that sells experimental stem-cell and gene therapies. Anzinger founded a community there called Infinita City; he has also founded an investment company and a “service providing” company, both of which include the name Infinita. Over the last couple of years, Anzinger has switched his focus to the US. “Now we think that Montana is a better model, because it’s building on … existing regulatory precedents,” he says. Once Montana’s 2023 law was passed, he adds, he worked with a handful of unnamed biotech companies to draft a second bill—one that laid out the specific terms under which clinics can offer unapproved drugs. That law was passed in April 2025 and adopted the following month. Since then, Anzinger, Bogner, and others have been waiting for the state’s Department of Health and Human Services to finalize specific rules for treatment centers—a set of operational guidelines and requirements that any clinic offering treatments unapproved by the FDA must meet under Montana’s law. “The rules have been taking a very long time,” says Anzinger. “Then on Friday, we heard they were effective … from Saturday [July 25].” The rules have since been published online. Following the rules With the new rules in hand, Anzinger and his colleague Stephen Martin, Infinita’s US lead, got to work. The first step was to establish an independent experimental treatment review board—a panel of five experts to evaluate applications for access. Anzinger and Martin started recruiting candidates months ago. The state’s first board, named the Montana ETRB, was officially announced by Infinita earlier this week. For the time being, it is the state’s only review board, although Anzinger says that other groups are free to establish their own. After Bogner raised concerns that the board’s website wrongly implied that it was an official state body, the site was updated to note that “It is a private service run by Montana Governance Services Inc.” That company is “a local Montana registered entity, but it is under the Infinita umbrella,” says Anzinger.  Infinita will pay board members a flat fee, funded by the $12,500 companies will have to pay to have their applications reviewed. Anzinger stresses that the board members, and their decisions, will be independent of Infinita. In accordance with the rules, the board includes a Montana-licensed doctor: James Burke, an oncologist. It also includes a bioethicist: Jessica Flanigan, a libertarian who is known for her strong views in support of self-medication and her book Pharmaceutical Freedom. The other three members are familiar faces in the longevity community—all of whom are well respected in the field. “When we looked at our own network, these were some of the best guys,” says Martin. They include Felipe Sierra, who formerly held a senior role at the National Institutes of Health’s arm focused on aging. More recently, Sierra served as chief scientific officer at Hevolution Foundation, a nonprofit that funds research into extending healthy lifespan with the support of the government of Saudi Arabia. Matt Kaeberlein, who

Google DeepMind has released Gemini Robotics 2, the intelligence layer for its next generation of robots. The release moves the stack past table-top manipulation into whole body control, five finger dexterity and multi robot teamwork. It ships as three separate models with three different access tiers. Most robots today are pre-programmed or tele-operated for narrow, repetitive task sequences. They do not adapt to unpredictable environments, and skills rarely transfer between robot bodies. Gemini Robotics 2 targets all three limits at once. TL;DR Three models ship together: a VLA, an embodied reasoning VLM, and an on-device VLA. One checkpoint drives Apollo 2 with two different hands plus a Franka Duo gripper. Gemini Robotics ER 2 is public preview; the VLA and on-device models stay gated. Multi-finger dexterity is still the weak axis, ranging from 32% to 92%. ASIMOV-Agentic, a new safety benchmark, is on Hugging Face under CC-BY-4.0. 3 Models and What They Do Gemini Robotics 2 Gemini Robotics 2 is the vision-language-action (VLA) model. It converts vision and language input into motor control. It can drive full humanoids from feet to fingertips, and other bi-arm robots. It also handles dexterous manipulation on both multi-finger hands and parallel grippers. Gemini Robotics ER 2 Gemini Robotics ER 2 is the embodied reasoning (ER) model. It is a vision language model that acts as the high level brain. It communicates with humans, understands the physical world, and plans multi-step tasks lasting several minutes. Per its model card, ER 2 is based on Gemini 3.5 Flash. It accepts interleaved text, image, video and audio with a context window of up to 128k, and emits text up to 64K tokens. Gemini Robotics On-Device 2 Gemini Robotics On-Device 2 is the efficient VLA optimized to run locally on the robot. Its model card states it is built on Gemini Robotics 1.5 technology and Google’s on-device Gemma models. Inputs are text, images and robot proprioception as numerical values. Outputs are robot actions as numerical values. The division of labor matters for system design. ER 2 plans and tracks the task, then hands motor execution to a VLA declared as a tool. Developers register low level control interfaces, such as VLA models or navigation APIs, as callable tools. They then stream multimodal video, audio or text directly into the model. Whole body control on Apptronik Apollo 2 Previous Gemini Robotics models controlled only the humanoid’s upper body for table-top tasks. Gemini Robotics 2 extends control to whole body motion for the first time. The worked example uses Apptronik’s Apollo 2. Given the instruction “put the watering can into the green bin in the bottom shelf,” Apollo walks to the table and picks up the watering can. It then takes a few steps to the shelves and places the object at its destination. Google DeepMind is direct about the remaining gap. It states that its robots have more to advance in movement speed. Dexterity across hands and grippers Gemini Robotics 2 can control the five-fingered, 22 degree-of-freedom SharpaWave hand on Apollo 2. Reported actions include tying knots and sealing a ziplock bag. The same model also operates standard two-fingered parallel grippers on a Franka Duo platform for tasks like tight packing. The reported success rates come from one model checkpoint controlling three embodiments: Apollo 2 with SharpaWave hands, Apollo 2 with Inspire hands, and Franka Duo with a Robotiq gripper. Category Embodiment Task Success General whole body manipulation Apollo 2 + Inspire hands Pick up from shelf 76.3% General whole body manipulation Apollo 2 + Inspire hands Pick up from table 68.4% General whole body manipulation Apollo 2 + Inspire hands Pick up from floor 45.7% Multi-finger dexterity Apollo 2 + Sharpa hands Unscrew bulb 92% Multi-finger dexterity Apollo 2 + Sharpa hands Tie trash bag 44% Multi-finger dexterity Apollo 2 + Sharpa hands Ziplock 40% Multi-finger dexterity Apollo 2 + Sharpa hands Screw bulb 36% Multi-finger dexterity Apollo 2 + Sharpa hands Dustpan 32% Gripper dexterity Franka Duo Precise insertion tasks 89.6% Gripper dexterity Franka Duo Diverse tool kitting 78.9% Gripper dexterity Franka Duo General pick and place 74.2% ER 2: Temporal Intelligence and Tool Orchestration The developer X post focuses on a problem that rarely gets benchmarked: knowing when a task is actually done. Progress classification: Each frame in a video feed is assigned to one of five progress levels, from 0-20% through 80-100%. Gemini Robotics ER 2 reaches 57.4% accuracy on this task. Google DeepMind reports this outperforms previous generation models and competing frontier models. Moment finding: This measures whether the model can identify the exact frame where a critical event occurs. An example is the moment to stop pouring coffee into a cup. ER 2 reaches 91.3% accuracy with a 0.96 second mean absolute distance. Google DeepMind reports it competing closely with much larger model categories at 4x the execution speed. Tool orchestration: ER 2 is evaluated across three control modes: real VLA, sim VLA, and human tele-op. It outperforms Gemini Robotics ER 1.6 on all three. ER 2 integrates with the Gemini Live API through a bidirectional streaming endpoint. The stated purpose is removing the stop-and-think pauses that break multi-step execution. It can also natively call tools like Google Search or any user-defined function. Three spatial capabilities were upgraded. Success and failure detection now runs on raw video feeds instead of static snapshots, which catches mid-execution spills and slips. General instrument reading extends beyond circular dials to digital displays, linear scales, rulers and liquid thermometers, tested across 10 instrument types. Spatial visual question answering improves through Gemini’s multimodal advances. Google DeepMind built a demo with Spot from Boston Dynamics, using ER 2 to orchestrate Spot navigation and manipulator APIs. Sample code is available in the robotics-samples repository. Multi-Robot Collaboration Gemini Robotics 2 introduces collaboration between different robot types. The reasoning is that no single robot fits every task. A wheeled rover suits indoor work, while a humanoid handles uneven terrain better. Robots communicate through a shared semantic understanding to hand off subtasks. The

Liquid AI has released two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. Both are masked language models built on the LFM2 hybrid backbone. Both carry an 8,192-token context. Encoders sit underneath classifiers, intent routers, safety filters, and PII detectors. Those jobs run continuously, usually without a GPU, and increasingly on longer inputs. BERT established the class. ModernBERT pushed its accuracy, speed, and context. Liquid AI’s argument is that the LFM2 architecture continues that line, because its cost grows more slowly as inputs get longer. How a decoder became an encoder The encoders are not trained from scratch. They are initialized from the LFM2.5-230M and LFM2.5-350M decoder backbones, then converted with three changes: First, the causal attention mask is replaced with a bidirectional one, so every token attends to both sides. Second, the LFM2 short convolutions are made non-causal using symmetric center padding, so each token’s convolution mixes in neighbours on both sides. Third, the model is trained with a masked language modeling objective at a 30% mask rate. That is denser than BERT’s 15%, following evidence Liquid AI cites that a higher mask rate helps at this scale. Training runs in two stages: Stage one establishes general language competence with a short-context MLM objective on a large web corpus at 1,024 tokens. Stage two extends context to 8,192 tokens on the full data mix, strengthening factual, legal, and multilingual competence. Architecturally, the backbone interleaves gated short-convolution blocks with grouped-query attention, the same design described in the LFM2 technical report. Both checkpoints use a hidden size of 1024 and a 65,536-token vocabulary, and support 15 languages. The license is the LFM Open License v1.0. The embed below breaks down the conversion, the full ranking table, the per-task scores, and the CPU latency figures. The benchmark results Liquid AI evaluated 14 models on 17 tasks pulled from GLUE, SuperGLUE, and multilingual classification. Every model is fully fine-tuned per task, and the reported score is that fine-tuned model’s result. LFM2.5-Encoder-350M posts a 17-task mean of 81.02 (±1.00), ranking fourth. The three models ahead of it are all larger. They are XLM-R XL at 3.5B (83.06), ModernBERT-large at 395M (81.68), and XLM-R large at 560M (81.34). The top model is nearly 10 times its size. LFM2.5-Encoder-230M posts 79.29 (±1.02), ranking sixth. It beats ModernBERT-base at 78.19 and every EuroBERT model in the table, including EuroBERT-610M (75.87) and EuroBERT-2.1B (72.19). Both new encoders also score above Liquid AI’s own retrieval siblings, LFM2.5-ColBERT-350M (76.18) and LFM2.5-Embedding-350M (75.68). That gap is the stated reason Liquid AI built a general-purpose encoder instead of reusing the retrievers. The methodology is the more interesting part, and it is open-sourced under Apache-2.0. Every model is loaded with fp32 master weights and bf16 autocast, so the table compares models rather than number formats. Every model uses the same AdamW recipe, taken from the EuroBERT card. Learning rate is selected per model and task across 10 rates and 3 seeds. Scores are then reported as the mean over 5 fresh seeds that never touched selection. The transformers version is pinned to 4.56.2 so dependency drift is not an uncontrolled variable. Use Cases and Deployment Environments The release names three settings. Edge and embedded devices come first. A car’s onboard compute or an industrial controller has no spare GPU, and cannot afford a cloud round trip. Regulated and on-premise systems in finance, healthcare, and legal, where documents are long, sensitive, and cannot leave in-house infrastructure. And high-volume cost-sensitive pipelines, where a small encoder acts as a cheap first pass in front of a larger model. Liquid AI also puts a useful number on the context window: 8,192 tokens is roughly 13 to 15 pages. One forward pass covers a full contract or a complete patient record. To show what a fine-tuned encoder looks like, the research team shipped five demos. Each runs in a CPU-only Hugging Face Space. They cover zero-shot prompt routing, zero-shot policy linting, and spell checking. A PII detector handles 40 information types across 16 languages. A bonus masked-diffusion demo runs the encoder as a chatbot that generates by iteratively unmasking. Getting them running Both encoders load through transformers. The body is exposed as Lfm2BidirectionalModel and masked-LM loading uses Lfm2BidirectionalForMaskedLM. Both are wired through auto_map, so trust_remote_code=True is required on every load call. A base encoder produces general-purpose representations, not task outputs, so fine-tuning is mandatory. Liquid AI’s fine-tuning tutorial walks through long legal documents at an 8k context configuration. The model selection guidance is straightforward: 350M when accuracy matters most, 230M for tighter hardware or higher throughput. Key Takeaways Liquid AI released LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, open-weight bidirectional encoders with 8,192-token context. Both convert LFM2.5 decoders into encoders via bidirectional masking, non-causal short convolutions, and 30% MLM. LFM2.5-Encoder-350M ranks 4th of 14 models at 81.02; the three above it are all larger. At 8K tokens on CPU, the 230M runs one forward pass in ~28s versus over 90s for ModernBERT-base. Check out the Liquid AI blog post, the Hugging Face write-up, and the model cards for LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. The post Liquid AI Releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: Bidirectional Encoders That Stay Fast at 8K Context on CPU appeared first on MarkTechPost.

In June 2024, a small company called Zanskar purchased a geothermal power plant in New Mexico that was failing fast. The water coming from the underground reservoir was getting colder by the day, making the plant uneconomical to run. Now, two years later, that plant is running at full capacity again, thanks to a new well. With the help of advanced modeling and modern drilling technology, the company was able to identify a better potential well site, drill down thousands of feet, and revive the entire operation. As the world looks for more sources of emissions-free electricity that are available 24-7, Lightning Dock shows there’s still hidden potential deep beneath our feet.  Conventional geothermal power plants rely on having the right underground conditions. Water flows through fractured hot rocks to harvest heat and then through a power plant to generate electricity. If the water isn’t hot enough or not enough flows, the power plant can’t work efficiently. Lightning Dock came online in 2013, and the site historically had two production wells that were used to feed the power plant. It’s common to see temperatures drop at a well site over time, typically at a rate of 1 to 2 °F per year. In the period before Zanskar took over the site, Lightning Dock saw temperatures drop by 50 °F over five years, a rate of 10 °F per year. When the company purchased the facility, the water going into the power plant was just 250 °F, while the plant was designed to operate at temperatures of at least 310 °F.  When Zanskar mapped the underground conditions at Lightning Dock using advanced modeling techniques, they discovered that the wells were only hitting the very top of the reservoir there. Those production wells were quite shallow (at just 2,500 feet deep) and not in the best location, says Joel Edwards, Zanskar’s cofounder and CEO. The company’s modeling predicted that if the company were to drill another, deeper well in a new spot, the plant would be able to run efficiently. The team drilled a new well that reaches a depth of 8,000 feet and started operation in May 2025. After a full year, the well is still flowing at more than 4,000 gallons per minute. The plant has “completely turned around,” Edwards says. “It looks really exciting.” The data so far shows that the plant should be successful for years to come. “Ultimately you need to run these things for long time frames to get confidence in their performance over long time frames,” Edwards says. The progress at Lightning Dock could be good news for other geothermal sites too. The conventional wisdom in geothermal energy is that the deeper you go, the hotter it gets. But there’s usually a trade-off: With those depths come rocks that are packed tighter together. So drilling deeper could mean the hot water can’t flow as effectively, a problem for anyone trying to use it to generate electricity in a power plant. What the team found, however, was that the flow actually increased in the area where they drilled the new well. “That fundamentally changes how you think about not just Lightning Dock but all hydrothermal assets in America and what the potential can be for all of them,” says Ben Brenner, director of federal affairs at Zanskar.  Over the past year of operation, Lightning Dock generated over twice the electricity it would have with the old wells. This is a relatively small power plant, with a capacity of 15 megawatts going to the local grid (about enough to power 11,000 US homes). Oil and gas developers have chased resources deeper underground over the past few decades. Operations started relatively close to the surface, but oil and gas production can stretch down 20,000 feet or more today. “I think that arc is going to play out in geothermal,” Edwards says. While typical geothermal well fields range from 3,000 to 5,000 feet deep, it could become more common to go deeper in the future, he adds.   The company plans to do further development at the Lightning Dock site: with a few years of development and an upgrade to the power plant, it could get even more electricity out of this area, Edwards says. There’s been a lot of buzz and investment in enhanced geothermal systems—projects that aim to expand where geothermal resources can be used. Fervo Energy, for example, uses fracking techniques to open up rock that would otherwise be too closed off for traditional geothermal energy. But there’s still plenty of “low-hanging fruit” in the geothermal world, Edwards says. Many of these conventional resources have plenty of potential—they just need a second look. This article is from The Spark, MIT Technology Review’s weekly climate newsletter. To receive it in your inbox every Wednesday, sign up here. 

Three terms now compete for the same line in AI engineering job descriptions. Prompt engineering is the established one. Loop engineering entered the AI vocabulary in late 2025 and dominated developer discussion through June 2026. Graph engineering followed roughly six weeks later. They get used interchangeably. Should they be? The three are not competing techniques. They are three different units of control, stacked. A prompt controls one model response. A loop controls one agent’s behavior cycle. A graph controls the organization of many agents. Each layer preserves the layer beneath it. A prompt does not disappear once a loop is built around it. it stops being the thing typed by hand. This article separates the three: what gets designed at each layer, what the published claim says about when the higher layers pay for themselves, and where the skepticism is warranted. One task, three layers Interactive explainer · 01 One task, three layers — watch what changes The three terms are not competing techniques. They are three different units of control. Here is the same job handled at each layer, step by step. Orange dots mark the moments a human is required. Task Fix the failing tests in the auth module, then open a pull request. Run all three Reset Layer 1 Prompt engineering You control one model response. You are the loop. Your turns 0 · Model calls 0 Layer 2 Loop engineering You control one agent’s cycle. The loop does the prompting. Your turns 0 · Model calls 0 Layer 3 Graph engineering You control how many agents are organised. Your turns 0 · Nodes 0 · Parallel 0 What actually differs   Prompt Loop Graph Unit of control One model response One agent’s behaviour cycle An organisation of agents What you write Instructions, examples, output format Trigger, tools, stop condition, retry budget Nodes, edges, shared state, failure routes Who says “again?” A human, every turn A verifier the loop calls itself A routing rule written in advance Where it breaks Ambiguous or overstuffed instruction It cannot tell done from stuck Context never crossed an edge you forgot to draw Enough when One shot, a person reads the result Repetitive, machine-checkable, one domain Cross-domain work with parallel branches Illustrative walkthrough · step counts are not benchmarks Built by Marktechpost The stack, in order Each step in the progression was named in practice before it appeared in vendor documentation. Prompt engineering covers writing and structuring the instruction for a single call. Anthropic’s guidance is to separate a system prompt into labeled sections — background information, instructions, tool guidance, output description — delineated with XML tags or Markdown headers. The recommendation is to supply the minimal set of information that fully specifies the expected behavior. Minimal does not mean short. Context engineering came next. Anthropic describes it as the natural progression of prompt engineering. The question moves from finding the right words to deciding what configuration of tokens belongs in the window at all. Context is a finite resource, and the engineering problem is optimizing the utility of those tokens against model constraints. Harness engineering covers the environment a single agent runs inside: files, tools, memory, feedback. Loop engineering sits one floor above the harness. A June 2026 arXiv paper on agentic AI in building engineering, Buildrix, sets out the same four-step progression explicitly — prompt, then context, then harness, then loop — with the final layer defining how a system repeatedly observes, acts, verifies and recovers. Graph engineering is the newest label and the least settled. One enterprise writeup notes that the term’s provenance is unresolved and that it collides with an older knowledge-graph usage of the same word. The underlying practice, graph-based orchestration, has a documented lineage in multi-agent systems research. Layer 1: Prompt Engineering The defining assumption is that a human is present at every iteration. A prompt is written, the model responds, the output is judged, the prompt is revised. That assumption is what breaks. High volume. Multi-step tasks. No human available to grade the output. Results that feed the next step automatically. Any one of these, and the prompt alone stops being sufficient. Nothing about the prompt got worse. The surrounding conditions changed. Prompt engineering also does not vanish inside the higher layers. Anthropic’s multi-agent research writeup reports that prompt engineering was the primary lever for fixing coordination failures. Early versions spawned 50 subagents for simple queries, and the fix was prompting rather than topology. Prompt engineering, explained Interactive explainer · 02 · Layer 1 What prompt engineering is Prompt engineering is designing the text of a single call to a model. One input, one forward pass, one output — and a human who reads the result and decides whether to run it again. That last part is the assumption everything above this layer exists to remove. Prompt what you write Model one forward pass Response one output you you judge it, then rewrite the prompt yourself Swipe the diagram sideways → The cycle exists here too — but you are the one closing it, every single turn. Anatomy of a prompt Tap a section. Anthropic’s guidance is to delineate parts with XML tags or Markdown headers rather than writing one undifferentiated block. <background_information>What the model needs to know before it starts <instructions>The task itself, and the rules for doing it ## Tool guidanceWhich tool to reach for, and when ## Output descriptionThe exact shape the answer should take <examples>A few cases that cover the edges you actually hit What this section does Background information The assumption that breaks Prompt engineering assumes a human is present at every iteration. Tap a condition to see what happens when that stops being true. high volume multi-step task nobody available to grade it output feeds the next step Layer 1 of 3 · the layer most tasks stop at Built by Marktechpost Layer 2: loop engineering The framing is that a coding agent is a brute-force tool for finding solutions. The craft is designing the goal, the

In this tutorial, we deploy the 1-bit Bonsai-27B language model using the PrismML fork of llama.cpp, which provides the specialized CUDA kernels required to decode the model’s Q1_0_g128 GGUF quantization format. We begin by validating the GPU runtime, installing the required Python dependencies, compiling the CUDA-enabled inference binaries, and downloading the compressed model weights from Hugging Face. We then test the model through llama-cli, launch an OpenAI-compatible local inference server, and interact with it through a reusable Python client that supports standard completions, streamed responses, multi-turn conversations, and code generation. We also examine optional configurations for throughput benchmarking, quantized key-value caching, long-context inference, speculative decoding, and multimodal extensions. Copy CodeCopiedUse a different Browser import os import sys import time import json import shutil import subprocess import multiprocessing WORK_DIR = “/content” REPO_URL = “https://github.com/PrismML-Eng/llama.cpp” REPO_DIR = os.path.join(WORK_DIR, “llama.cpp”) BUILD_DIR = os.path.join(REPO_DIR, “build”) BIN_DIR = os.path.join(BUILD_DIR, “bin”) HF_REPO = “prism-ml/Bonsai-27B-gguf” MODEL_FILE = “Bonsai-27B-Q1_0.gguf” MODEL_PATH = os.path.join(WORK_DIR, MODEL_FILE) SERVER_HOST = “127.0.0.1” SERVER_PORT = 8080 SERVER_URL = f”http://{SERVER_HOST}:{SERVER_PORT}” GEN_PARAMS = {“temperature”: 0.7, “top_p”: 0.95, “top_k”: 20} CTX_SIZE = 8192 N_GPU_LAYERS = 99 USE_KV_Q4 = False def sh(cmd, check=True, **kw): “””Run a shell command, streaming output to the notebook.””” print(f”n$ {cmd}”) return subprocess.run(cmd, shell=True, check=check, **kw) print(“=” * 70) print(“[1/7] Checking environment”) print(“=” * 70) gpu = subprocess.run(“nvidia-smi –query-gpu=name,memory.total –format=csv,noheader”, shell=True, capture_output=True, text=True) if gpu.returncode != 0: sys.exit(“No GPU detected. In Colab: Runtime -> Change runtime type -> GPU (T4).”) print(f”GPU detected: {gpu.stdout.strip()}”) print(“Bonsai-27B needs only ~5.2 GB peak at 4K context — any Colab GPU works.”) sh(“pip -q install huggingface_hub requests”) We configure the Colab workspace, model repository, server endpoint, inference parameters, context size, and GPU offloading settings required throughout the tutorial. We define a reusable shell-command function and verify that the runtime exposes a compatible NVIDIA GPU before continuing. We then install the Hugging Face Hub and HTTP client dependencies needed for model retrieval and API communication. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[2/7] Building PrismML llama.cpp fork with CUDA (cached after 1st run)”) print(“=” * 70) if not os.path.isdir(REPO_DIR): sh(f”git clone –depth 1 {REPO_URL} {REPO_DIR}”) else: print(“Repo already cloned — skipping.”) cli_bin = os.path.join(BIN_DIR, “llama-cli”) server_bin = os.path.join(BIN_DIR, “llama-server”) bench_bin = os.path.join(BIN_DIR, “llama-bench”) if not (os.path.exists(cli_bin) and os.path.exists(server_bin)): jobs = multiprocessing.cpu_count() sh(f”cmake -S {REPO_DIR} -B {BUILD_DIR} -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release”) sh(f”cmake –build {BUILD_DIR} -j{jobs} –target llama-cli llama-server llama-bench”) else: print(“Binaries already built — skipping.”) We clone the PrismML fork of llama.cpp, which provides the specialized kernels required for the model’s Q1_0_g128 quantization format. We configure a CUDA-enabled release build with CMake and compile the command-line, server, and benchmarking executables. We also reuse previously generated binaries when they already exist, reducing repeated setup time in the same Colab session. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[3/7] Downloading weights from Hugging Face”) print(“=” * 70) from huggingface_hub import hf_hub_download if not os.path.exists(MODEL_PATH): downloaded = hf_hub_download(repo_id=HF_REPO, filename=MODEL_FILE, local_dir=WORK_DIR) print(f”Downloaded to: {downloaded}”) else: print(“Model already on disk — skipping.”) print(f”Model size on disk: {os.path.getsize(MODEL_PATH) / 1e9:.2f} GB”) We connect to the Hugging Face Hub and download the Bonsai-27B GGUF model into the Colab workspace. We skip the transfer when the model file is already available locally, allowing subsequent runs to proceed more efficiently. We then calculate and display the deployed model size to confirm that the compressed weights are stored correctly. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[4/7] Smoke test with llama-cli”) print(“=” * 70) sh( f'{cli_bin} -m {MODEL_PATH} ‘ f’-p “Explain in two sentences why 1-bit quantization saves memory.” ‘ f’-n 128 -ngl {N_GPU_LAYERS} ‘ f’–temp {GEN_PARAMS[“temperature”]} ‘ f’–top-p {GEN_PARAMS[“top_p”]} –top-k {GEN_PARAMS[“top_k”]} ‘ f’-no-cnv 2>/dev/null’, check=False, ) print(“=” * 70) print(“[5/7] Starting llama-server (OpenAI-compatible API)”) print(“=” * 70) import requests kv_flags = “-ctk q4_0 -ctv q4_0” if USE_KV_Q4 else “” server_cmd = ( f”{server_bin} -m {MODEL_PATH} ” f”–host {SERVER_HOST} –port {SERVER_PORT} ” f”-ngl {N_GPU_LAYERS} -c {CTX_SIZE} {kv_flags}” ) print(f”$ {server_cmd} (background)”) server_log = open(os.path.join(WORK_DIR, “server.log”), “w”) server_proc = subprocess.Popen(server_cmd, shell=True, stdout=server_log, stderr=server_log) for _ in range(120): try: if requests.get(f”{SERVER_URL}/health”, timeout=2).status_code == 200: print(“Server is up.”) break except requests.exceptions.RequestException: pass time.sleep(2) else: server_proc.kill() sys.exit(“Server failed to start — check /content/server.log”) We perform a command-line smoke test to verify that the compiled runtime can load the quantized model and generate a valid response. We then start llama-server with full GPU layer offloading, the selected context window, and optional quantized KV-cache settings. We repeatedly query the health endpoint until the OpenAI-compatible inference service becomes ready for client requests. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[6/7] Talking to Bonsai-27B via the OpenAI-compatible API”) print(“=” * 70) def chat(messages, stream=False, max_tokens=512, **overrides): “””Minimal OpenAI-compatible chat client for the local llama-server.””” payload = { “model”: “bonsai-27b”, “messages”: messages, “max_tokens”: max_tokens, “stream”: stream, **GEN_PARAMS, **overrides, } if not stream: r = requests.post(f”{SERVER_URL}/v1/chat/completions”, json=payload) r.raise_for_status() return r.json()[“choices”][0][“message”][“content”] r = requests.post(f”{SERVER_URL}/v1/chat/completions”, json=payload, stream=True) r.raise_for_status() full = [] for line in r.iter_lines(): if not line or not line.startswith(b”data: “): continue chunk = line[len(b”data: “):] if chunk == b”[DONE]”: break delta = json.loads(chunk)[“choices”][0][“delta”].get(“content”, “”) full.append(delta) print(delta, end=””, flush=True) print() return “”.join(full) SYSTEM = {“role”: “system”, “content”: “You are a helpful assistant”} print(“n— 6a: basic completion —“) answer = chat([SYSTEM, {“role”: “user”, “content”: “What is the capital of France? One sentence.”}]) print(answer) print(“n— 6b: math reasoning, streamed token-by-token —“) chat([SYSTEM, {“role”: “user”, “content”: “A train travels 120 km at 80 km/h, then 90 km at ” “60 km/h. What is its average speed for the whole ” “trip? Show your reasoning briefly.”}], stream=True, max_tokens=700) print(“n— 6c: multi-turn chat —“) history = [SYSTEM] for user_msg in [“My name is Priya and I love graph algorithms.”, “Suggest one project idea that combines my interest with LLMs.”, “What was my name again?”]: history.append({“role”: “user”, “content”: user_msg}) reply = chat(history, max_tokens=300) history.append({“role”: “assistant”, “content”: reply}) print(f”nUSER: {user_msg}nBONSAI: {reply}”) print(“n— 6d: code generation —“) print(chat([SYSTEM, {“role”: “user”, “content”: “Write a Python function that returns the n-th ” “Fibonacci number using memoization. Code only.”}], max_tokens=400)) We define a reusable Python chat client that sends OpenAI-compatible requests to the locally hosted Bonsai-27B server. We support both standard