{"id":107456,"date":"2026-07-28T19:49:38","date_gmt":"2026-07-28T19:49:38","guid":{"rendered":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/"},"modified":"2026-07-28T19:49:38","modified_gmt":"2026-07-28T19:49:38","slug":"deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows","status":"publish","type":"post","link":"https:\/\/youzum.net\/es\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/","title":{"rendered":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows"},"content":{"rendered":"

In this tutorial, we deploy the 1-bit Bonsai-27B<\/strong><\/a> language model using the PrismML fork of llama.cpp, which provides the specialized CUDA kernels required to decode the model\u2019s Q1_0_g128 GGUF quantization format. We begin by validating the GPU runtime, installing the required Python dependencies, compiling the CUDA-enabled inference binaries, and downloading the compressed model weights from Hugging Face. We then test the model through llama-cli, launch an OpenAI-compatible local inference server, and interact with it through a reusable Python client that supports standard completions, streamed responses, multi-turn conversations, and code generation. We also examine optional configurations for throughput benchmarking, quantized key-value caching, long-context inference, speculative decoding, and multimodal extensions.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import os\nimport sys\nimport time\nimport json\nimport shutil\nimport subprocess\nimport multiprocessing\nWORK_DIR      = \"\/content\"\nREPO_URL      = \"https:\/\/github.com\/PrismML-Eng\/llama.cpp\"\nREPO_DIR      = os.path.join(WORK_DIR, \"llama.cpp\")\nBUILD_DIR     = os.path.join(REPO_DIR, \"build\")\nBIN_DIR       = os.path.join(BUILD_DIR, \"bin\")\nHF_REPO       = \"prism-ml\/Bonsai-27B-gguf\"\nMODEL_FILE    = \"Bonsai-27B-Q1_0.gguf\"\nMODEL_PATH    = os.path.join(WORK_DIR, MODEL_FILE)\nSERVER_HOST   = \"127.0.0.1\"\nSERVER_PORT   = 8080\nSERVER_URL    = f\"http:\/\/{SERVER_HOST}:{SERVER_PORT}\"\nGEN_PARAMS    = {\"temperature\": 0.7, \"top_p\": 0.95, \"top_k\": 20}\nCTX_SIZE      = 8192\nN_GPU_LAYERS  = 99\nUSE_KV_Q4     = False\ndef sh(cmd, check=True, **kw):\n   \"\"\"Run a shell command, streaming output to the notebook.\"\"\"\n   print(f\"n$ {cmd}\")\n   return subprocess.run(cmd, shell=True, check=check, **kw)\nprint(\"=\" * 70)\nprint(\"[1\/7] Checking environment\")\nprint(\"=\" * 70)\ngpu = subprocess.run(\"nvidia-smi --query-gpu=name,memory.total --format=csv,noheader\",\n                    shell=True, capture_output=True, text=True)\nif gpu.returncode != 0:\n   sys.exit(\"No GPU detected. In Colab: Runtime -> Change runtime type -> GPU (T4).\")\nprint(f\"GPU detected: {gpu.stdout.strip()}\")\nprint(\"Bonsai-27B needs only ~5.2 GB peak at 4K context \u2014 any Colab GPU works.\")\nsh(\"pip -q install huggingface_hub requests\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We configure the Colab workspace, model repository, server endpoint, inference parameters, context size, and GPU offloading settings required throughout the tutorial. We define a reusable shell-command function and verify that the runtime exposes a compatible NVIDIA GPU before continuing. We then install the Hugging Face Hub and HTTP client dependencies needed for model retrieval and API communication.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"=\" * 70)\nprint(\"[2\/7] Building PrismML llama.cpp fork with CUDA (cached after 1st run)\")\nprint(\"=\" * 70)\nif not os.path.isdir(REPO_DIR):\n   sh(f\"git clone --depth 1 {REPO_URL} {REPO_DIR}\")\nelse:\n   print(\"Repo already cloned \u2014 skipping.\")\ncli_bin    = os.path.join(BIN_DIR, \"llama-cli\")\nserver_bin = os.path.join(BIN_DIR, \"llama-server\")\nbench_bin  = os.path.join(BIN_DIR, \"llama-bench\")\nif not (os.path.exists(cli_bin) and os.path.exists(server_bin)):\n   jobs = multiprocessing.cpu_count()\n   sh(f\"cmake -S {REPO_DIR} -B {BUILD_DIR} -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release\")\n   sh(f\"cmake --build {BUILD_DIR} -j{jobs} --target llama-cli llama-server llama-bench\")\nelse:\n   print(\"Binaries already built \u2014 skipping.\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We clone the PrismML fork of llama.cpp, which provides the specialized kernels required for the model\u2019s Q1_0_g128 quantization format. We configure a CUDA-enabled release build with CMake and compile the command-line, server, and benchmarking executables. We also reuse previously generated binaries when they already exist, reducing repeated setup time in the same Colab session.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"=\" * 70)\nprint(\"[3\/7] Downloading weights from Hugging Face\")\nprint(\"=\" * 70)\nfrom huggingface_hub import hf_hub_download\nif not os.path.exists(MODEL_PATH):\n   downloaded = hf_hub_download(repo_id=HF_REPO, filename=MODEL_FILE,\n                                local_dir=WORK_DIR)\n   print(f\"Downloaded to: {downloaded}\")\nelse:\n   print(\"Model already on disk \u2014 skipping.\")\nprint(f\"Model size on disk: {os.path.getsize(MODEL_PATH) \/ 1e9:.2f} GB\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We connect to the Hugging Face Hub and download the Bonsai-27B GGUF model into the Colab workspace. We skip the transfer when the model file is already available locally, allowing subsequent runs to proceed more efficiently. We then calculate and display the deployed model size to confirm that the compressed weights are stored correctly.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"=\" * 70)\nprint(\"[4\/7] Smoke test with llama-cli\")\nprint(\"=\" * 70)\nsh(\n   f'{cli_bin} -m {MODEL_PATH} '\n   f'-p \"Explain in two sentences why 1-bit quantization saves memory.\" '\n   f'-n 128 -ngl {N_GPU_LAYERS} '\n   f'--temp {GEN_PARAMS[\"temperature\"]} '\n   f'--top-p {GEN_PARAMS[\"top_p\"]} --top-k {GEN_PARAMS[\"top_k\"]} '\n   f'-no-cnv 2>\/dev\/null',\n   check=False,\n)\nprint(\"=\" * 70)\nprint(\"[5\/7] Starting llama-server (OpenAI-compatible API)\")\nprint(\"=\" * 70)\nimport requests\nkv_flags = \"-ctk q4_0 -ctv q4_0\" if USE_KV_Q4 else \"\"\nserver_cmd = (\n   f\"{server_bin} -m {MODEL_PATH} \"\n   f\"--host {SERVER_HOST} --port {SERVER_PORT} \"\n   f\"-ngl {N_GPU_LAYERS} -c {CTX_SIZE} {kv_flags}\"\n)\nprint(f\"$ {server_cmd}  (background)\")\nserver_log = open(os.path.join(WORK_DIR, \"server.log\"), \"w\")\nserver_proc = subprocess.Popen(server_cmd, shell=True,\n                              stdout=server_log, stderr=server_log)\nfor _ in range(120):\n   try:\n       if requests.get(f\"{SERVER_URL}\/health\", timeout=2).status_code == 200:\n           print(\"Server is up.\")\n           break\n   except requests.exceptions.RequestException:\n       pass\n   time.sleep(2)\nelse:\n   server_proc.kill()\n   sys.exit(\"Server failed to start \u2014 check \/content\/server.log\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We perform a command-line smoke test to verify that the compiled runtime can load the quantized model and generate a valid response. We then start llama-server with full GPU layer offloading, the selected context window, and optional quantized KV-cache settings. We repeatedly query the health endpoint until the OpenAI-compatible inference service becomes ready for client requests.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"=\" * 70)\nprint(\"[6\/7] Talking to Bonsai-27B via the OpenAI-compatible API\")\nprint(\"=\" * 70)\ndef chat(messages, stream=False, max_tokens=512, **overrides):\n   \"\"\"Minimal OpenAI-compatible chat client for the local llama-server.\"\"\"\n   payload = {\n       \"model\": \"bonsai-27b\",\n       \"messages\": messages,\n       \"max_tokens\": max_tokens,\n       \"stream\": stream,\n       **GEN_PARAMS,\n       **overrides,\n   }\n   if not stream:\n       r = requests.post(f\"{SERVER_URL}\/v1\/chat\/completions\", json=payload)\n       r.raise_for_status()\n       return r.json()[\"choices\"][0][\"message\"][\"content\"]\n   r = requests.post(f\"{SERVER_URL}\/v1\/chat\/completions\", json=payload, stream=True)\n   r.raise_for_status()\n   full = []\n   for line in r.iter_lines():\n       if not line or not line.startswith(b\"data: \"):\n           continue\n       chunk = line[len(b\"data: \"):]\n       if chunk == b\"[DONE]\":\n           break\n       delta = json.loads(chunk)[\"choices\"][0][\"delta\"].get(\"content\", \"\")\n       full.append(delta)\n       print(delta, end=\"\", flush=True)\n   print()\n   return \"\".join(full)\nSYSTEM = {\"role\": \"system\", \"content\": \"You are a helpful assistant\"}\nprint(\"n--- 6a: basic completion ---\")\nanswer = chat([SYSTEM, {\"role\": \"user\",\n                       \"content\": \"What is the capital of France? One sentence.\"}])\nprint(answer)\nprint(\"n--- 6b: math reasoning, streamed token-by-token ---\")\nchat([SYSTEM, {\"role\": \"user\",\n              \"content\": \"A train travels 120 km at 80 km\/h, then 90 km at \"\n                         \"60 km\/h. What is its average speed for the whole \"\n                         \"trip? Show your reasoning briefly.\"}],\n    stream=True, max_tokens=700)\nprint(\"n--- 6c: multi-turn chat ---\")\nhistory = [SYSTEM]\nfor user_msg in [\"My name is Priya and I love graph algorithms.\",\n                \"Suggest one project idea that combines my interest with LLMs.\",\n                \"What was my name again?\"]:\n   history.append({\"role\": \"user\", \"content\": user_msg})\n   reply = chat(history, max_tokens=300)\n   history.append({\"role\": \"assistant\", \"content\": reply})\n   print(f\"nUSER: {user_msg}nBONSAI: {reply}\")\nprint(\"n--- 6d: code generation ---\")\nprint(chat([SYSTEM, {\"role\": \"user\",\n                    \"content\": \"Write a Python function that returns the n-th \"\n                               \"Fibonacci number using memoization. Code only.\"}],\n          max_tokens=400))\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We define a reusable Python chat client that sends OpenAI-compatible requests to the locally hosted Bonsai-27B server. We support both standard and server-sent-event streaming responses while applying the configured temperature, top-p, and top-k sampling parameters. We then evaluate basic factual answering, mathematical reasoning, multi-turn conversational memory, and Python code generation.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"=\" * 70)\nprint(\"[7\/7] Optional extras\")\nprint(\"=\" * 70)\nRUN_BENCHMARK = False\nif RUN_BENCHMARK:\n   sh(f\"{bench_bin} -m {MODEL_PATH} -ngl {N_GPU_LAYERS}\", check=False)\nprint(\"\"\"\nNOTES & NEXT STEPS\n------------------\n* Long context: the model supports up to 262K tokens. On Colab, raise\n CTX_SIZE and set USE_KV_Q4 = True (4-bit KV cache) \u2014 with it, 100K-token\n contexts fit in roughly 6.8 GB peak, well inside a T4's 16 GB.\n* Speculative decoding: the repo ships a DSpark drafter (Q4_1, ~1.79 GB)\n that gives a lossless ~1.37x decode speedup on CUDA. See the PrismML\n llama.cpp fork's README for the serving flags, and download the drafter\n pack from the same HF repo if you want to try it.\n* Vision: an optional ~0.63 GB mmproj pack adds image input; it is only\n loaded when an image arrives, so text-only serving never pays for it.\n* Quality vs size: if you want more headroom, the ternary sibling\n (prism-ml\/Ternary-Bonsai-27B-gguf, ~5.9 GB, ~95% of FP16) is a drop-in\n swap \u2014 just change HF_REPO \/ MODEL_FILE above.\n* Shutting down: run  server_proc.kill()  in a later cell to free the GPU.\n\"\"\")\nprint(\"Done. The server is still running \u2014 call chat([...]) from new cells!\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We expose an optional benchmarking switch that measures prompt-processing and token-generation performance with the compiled llama-bench executable. We review advanced deployment options, including long-context inference, 4-bit KV caching, speculative decoding, vision support, and a higher-capacity ternary model variant. We finish while keeping the server process active so that we can continue calling the chat function from additional Colab cells.<\/p>\n
In conclusion, we established a complete local inference workflow for running Bonsai-27B. We used the PrismML implementation to preserve compatibility with the model\u2019s highly compressed 1.125-bit weight representation while keeping the full inference pipeline accessible through both command-line and OpenAI-compatible interfaces. We validated reasoning, conversational memory, streaming generation, and programming capabilities, while retaining control over sampling parameters, context length, GPU offloading, and KV-cache precision. This setup gives us a platform for experimenting with low-bit large language models, evaluating their efficiency and output quality, and integrating them into Python applications without relying on an external hosted inference service.<\/p>\n
\n
\n<\/p>
\n<\/p>
Check out the\u00a0Full Code here<\/a>.\u00a0<\/strong>Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0150k+ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
The post Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we deploy the 1-bit Bonsai-27B language model using the PrismML fork of llama.cpp, which provides the specialized CUDA kernels required to decode the model\u2019s Q1_0_g128 GGUF quantization format. We begin by validating the GPU runtime, installing the required Python dependencies, compiling the CUDA-enabled inference binaries, and downloading the compressed model weights from Hugging Face. We then test the model through llama-cli, launch an OpenAI-compatible local inference server, and interact with it through a reusable Python client that supports standard completions, streamed responses, multi-turn conversations, and code generation. We also examine optional configurations for throughput benchmarking, quantized key-value caching, long-context inference, speculative decoding, and multimodal extensions. Copy CodeCopiedUse a different Browser import os import sys import time import json import shutil import subprocess import multiprocessing WORK_DIR = “\/content” REPO_URL = “https:\/\/github.com\/PrismML-Eng\/llama.cpp” REPO_DIR = os.path.join(WORK_DIR, “llama.cpp”) BUILD_DIR = os.path.join(REPO_DIR, “build”) BIN_DIR = os.path.join(BUILD_DIR, “bin”) HF_REPO = “prism-ml\/Bonsai-27B-gguf” MODEL_FILE = “Bonsai-27B-Q1_0.gguf” MODEL_PATH = os.path.join(WORK_DIR, MODEL_FILE) SERVER_HOST = “127.0.0.1” SERVER_PORT = 8080 SERVER_URL = f”http:\/\/{SERVER_HOST}:{SERVER_PORT}” GEN_PARAMS = {“temperature”: 0.7, “top_p”: 0.95, “top_k”: 20} CTX_SIZE = 8192 N_GPU_LAYERS = 99 USE_KV_Q4 = False def sh(cmd, check=True, **kw): “””Run a shell command, streaming output to the notebook.””” print(f”n$ {cmd}”) return subprocess.run(cmd, shell=True, check=check, **kw) print(“=” * 70) print(“[1\/7] Checking environment”) print(“=” * 70) gpu = subprocess.run(“nvidia-smi –query-gpu=name,memory.total –format=csv,noheader”, shell=True, capture_output=True, text=True) if gpu.returncode != 0: sys.exit(“No GPU detected. In Colab: Runtime -> Change runtime type -> GPU (T4).”) print(f”GPU detected: {gpu.stdout.strip()}”) print(“Bonsai-27B needs only ~5.2 GB peak at 4K context \u2014 any Colab GPU works.”) sh(“pip -q install huggingface_hub requests”) We configure the Colab workspace, model repository, server endpoint, inference parameters, context size, and GPU offloading settings required throughout the tutorial. We define a reusable shell-command function and verify that the runtime exposes a compatible NVIDIA GPU before continuing. We then install the Hugging Face Hub and HTTP client dependencies needed for model retrieval and API communication. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[2\/7] Building PrismML llama.cpp fork with CUDA (cached after 1st run)”) print(“=” * 70) if not os.path.isdir(REPO_DIR): sh(f”git clone –depth 1 {REPO_URL} {REPO_DIR}”) else: print(“Repo already cloned \u2014 skipping.”) cli_bin = os.path.join(BIN_DIR, “llama-cli”) server_bin = os.path.join(BIN_DIR, “llama-server”) bench_bin = os.path.join(BIN_DIR, “llama-bench”) if not (os.path.exists(cli_bin) and os.path.exists(server_bin)): jobs = multiprocessing.cpu_count() sh(f”cmake -S {REPO_DIR} -B {BUILD_DIR} -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release”) sh(f”cmake –build {BUILD_DIR} -j{jobs} –target llama-cli llama-server llama-bench”) else: print(“Binaries already built \u2014 skipping.”) We clone the PrismML fork of llama.cpp, which provides the specialized kernels required for the model\u2019s Q1_0_g128 quantization format. We configure a CUDA-enabled release build with CMake and compile the command-line, server, and benchmarking executables. We also reuse previously generated binaries when they already exist, reducing repeated setup time in the same Colab session. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[3\/7] Downloading weights from Hugging Face”) print(“=” * 70) from huggingface_hub import hf_hub_download if not os.path.exists(MODEL_PATH): downloaded = hf_hub_download(repo_id=HF_REPO, filename=MODEL_FILE, local_dir=WORK_DIR) print(f”Downloaded to: {downloaded}”) else: print(“Model already on disk \u2014 skipping.”) print(f”Model size on disk: {os.path.getsize(MODEL_PATH) \/ 1e9:.2f} GB”) We connect to the Hugging Face Hub and download the Bonsai-27B GGUF model into the Colab workspace. We skip the transfer when the model file is already available locally, allowing subsequent runs to proceed more efficiently. We then calculate and display the deployed model size to confirm that the compressed weights are stored correctly. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[4\/7] Smoke test with llama-cli”) print(“=” * 70) sh( f'{cli_bin} -m {MODEL_PATH} ‘ f’-p “Explain in two sentences why 1-bit quantization saves memory.” ‘ f’-n 128 -ngl {N_GPU_LAYERS} ‘ f’–temp {GEN_PARAMS[“temperature”]} ‘ f’–top-p {GEN_PARAMS[“top_p”]} –top-k {GEN_PARAMS[“top_k”]} ‘ f’-no-cnv 2>\/dev\/null’, check=False, ) print(“=” * 70) print(“[5\/7] Starting llama-server (OpenAI-compatible API)”) print(“=” * 70) import requests kv_flags = “-ctk q4_0 -ctv q4_0” if USE_KV_Q4 else “” server_cmd = ( f”{server_bin} -m {MODEL_PATH} ” f”–host {SERVER_HOST} –port {SERVER_PORT} ” f”-ngl {N_GPU_LAYERS} -c {CTX_SIZE} {kv_flags}” ) print(f”$ {server_cmd} (background)”) server_log = open(os.path.join(WORK_DIR, “server.log”), “w”) server_proc = subprocess.Popen(server_cmd, shell=True, stdout=server_log, stderr=server_log) for _ in range(120): try: if requests.get(f”{SERVER_URL}\/health”, timeout=2).status_code == 200: print(“Server is up.”) break except requests.exceptions.RequestException: pass time.sleep(2) else: server_proc.kill() sys.exit(“Server failed to start \u2014 check \/content\/server.log”) We perform a command-line smoke test to verify that the compiled runtime can load the quantized model and generate a valid response. We then start llama-server with full GPU layer offloading, the selected context window, and optional quantized KV-cache settings. We repeatedly query the health endpoint until the OpenAI-compatible inference service becomes ready for client requests. Copy CodeCopiedUse a different Browser print(“=” * 70) print(“[6\/7] Talking to Bonsai-27B via the OpenAI-compatible API”) print(“=” * 70) def chat(messages, stream=False, max_tokens=512, **overrides): “””Minimal OpenAI-compatible chat client for the local llama-server.””” payload = { “model”: “bonsai-27b”, “messages”: messages, “max_tokens”: max_tokens, “stream”: stream, **GEN_PARAMS, **overrides, } if not stream: r = requests.post(f”{SERVER_URL}\/v1\/chat\/completions”, json=payload) r.raise_for_status() return r.json()[“choices”][0][“message”][“content”] r = requests.post(f”{SERVER_URL}\/v1\/chat\/completions”, json=payload, stream=True) r.raise_for_status() full = [] for line in r.iter_lines(): if not line or not line.startswith(b”data: “): continue chunk = line[len(b”data: “):] if chunk == b”[DONE]”: break delta = json.loads(chunk)[“choices”][0][“delta”].get(“content”, “”) full.append(delta) print(delta, end=””, flush=True) print() return “”.join(full) SYSTEM = {“role”: “system”, “content”: “You are a helpful assistant”} print(“n— 6a: basic completion —“) answer = chat([SYSTEM, {“role”: “user”, “content”: “What is the capital of France? One sentence.”}]) print(answer) print(“n— 6b: math reasoning, streamed token-by-token —“) chat([SYSTEM, {“role”: “user”, “content”: “A train travels 120 km at 80 km\/h, then 90 km at ” “60 km\/h. What is its average speed for the whole ” “trip? Show your reasoning briefly.”}], stream=True, max_tokens=700) print(“n— 6c: multi-turn chat —“) history = [SYSTEM] for user_msg in [“My name is Priya and I love graph algorithms.”, “Suggest one project idea that combines my interest with LLMs.”, “What was my name again?”]: history.append({“role”: “user”, “content”: user_msg}) reply = chat(history, max_tokens=300) history.append({“role”: “assistant”, “content”: reply}) print(f”nUSER: {user_msg}nBONSAI: {reply}”) print(“n— 6d: code generation —“) print(chat([SYSTEM, {“role”: “user”, “content”: “Write a Python function that returns the n-th ” “Fibonacci number using memoization. Code only.”}], max_tokens=400)) We define a reusable Python chat client that sends OpenAI-compatible requests to the locally hosted Bonsai-27B server. We support both standard<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"pmpro_default_level":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_pvb_checkbox_block_on_post":false,"footnotes":""},"categories":[52,5,7,1],"tags":[],"class_list":["post-107456","post","type-post","status-publish","format-standard","hentry","category-ai-club","category-committee","category-news","category-uncategorized","pmpro-has-access"],"acf":[],"yoast_head":"\nDeploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum<\/title>\n<meta name=\"description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/youzum.net\/es\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\" \/>\n<meta property=\"og:locale\" content=\"es_ES\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum\" \/>\n<meta property=\"og:description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta property=\"og:url\" content=\"https:\/\/youzum.net\/es\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\" \/>\n<meta property=\"og:site_name\" content=\"YouZum\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DroneAssociationTH\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-07-28T19:49:38+00:00\" \/>\n<meta name=\"author\" content=\"admin NU\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Escrito por\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin NU\" \/>\n\t<meta name=\"twitter:label2\" content=\"Tiempo de lectura\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutos\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\"},\"author\":{\"name\":\"admin NU\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\"},\"headline\":\"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows\",\"datePublished\":\"2026-07-28T19:49:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\"},\"wordCount\":690,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"articleSection\":[\"AI\",\"Committee\",\"News\",\"Uncategorized\"],\"inLanguage\":\"es\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\",\"url\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\",\"name\":\"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum\",\"isPartOf\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#website\"},\"datePublished\":\"2026-07-28T19:49:38+00:00\",\"description\":\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\",\"breadcrumb\":{\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#breadcrumb\"},\"inLanguage\":\"es\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/youzum.net\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/yousum.gpucore.co\/#website\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"name\":\"YouSum\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/yousum.gpucore.co\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"es\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\",\"name\":\"Drone Association Thailand\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"es\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"width\":300,\"height\":300,\"caption\":\"Drone Association Thailand\"},\"image\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DroneAssociationTH\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\",\"name\":\"admin NU\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"es\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"caption\":\"admin NU\"},\"url\":\"https:\/\/youzum.net\/es\/members\/adminnu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/youzum.net\/es\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/","og_locale":"es_ES","og_type":"article","og_title":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum","og_description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","og_url":"https:\/\/youzum.net\/es\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/","og_site_name":"YouZum","article_publisher":"https:\/\/www.facebook.com\/DroneAssociationTH\/","article_published_time":"2026-07-28T19:49:38+00:00","author":"admin NU","twitter_card":"summary_large_image","twitter_misc":{"Escrito por":"admin NU","Tiempo de lectura":"8 minutos"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#article","isPartOf":{"@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/"},"author":{"name":"admin NU","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c"},"headline":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows","datePublished":"2026-07-28T19:49:38+00:00","mainEntityOfPage":{"@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/"},"wordCount":690,"commentCount":0,"publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"articleSection":["AI","Committee","News","Uncategorized"],"inLanguage":"es","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/","url":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/","name":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows - YouZum","isPartOf":{"@id":"https:\/\/yousum.gpucore.co\/#website"},"datePublished":"2026-07-28T19:49:38+00:00","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","breadcrumb":{"@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#breadcrumb"},"inLanguage":"es","potentialAction":[{"@type":"ReadAction","target":["https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/youzum.net\/deploying-a-1-bit-bonsai-27b-model-with-prismml-llama-cpp-and-openai-compatible-local-inference-workflows\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/youzum.net\/"},{"@type":"ListItem","position":2,"name":"Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows"}]},{"@type":"WebSite","@id":"https:\/\/yousum.gpucore.co\/#website","url":"https:\/\/yousum.gpucore.co\/","name":"YouSum","description":"","publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/yousum.gpucore.co\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"es"},{"@type":"Organization","@id":"https:\/\/yousum.gpucore.co\/#organization","name":"Drone Association Thailand","url":"https:\/\/yousum.gpucore.co\/","logo":{"@type":"ImageObject","inLanguage":"es","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","width":300,"height":300,"caption":"Drone Association Thailand"},"image":{"@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DroneAssociationTH\/"]},{"@type":"Person","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c","name":"admin NU","image":{"@type":"ImageObject","inLanguage":"es","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","caption":"admin NU"},"url":"https:\/\/youzum.net\/es\/members\/adminnu\/"}]}},"rttpg_featured_image_url":null,"rttpg_author":{"display_name":"admin NU","author_link":"https:\/\/youzum.net\/es\/members\/adminnu\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/youzum.net\/es\/category\/ai-club\/\" rel=\"category tag\">AI<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/committee\/\" rel=\"category tag\">Committee<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/news\/\" rel=\"category tag\">News<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/uncategorized\/\" rel=\"category tag\">Uncategorized<\/a>","rttpg_excerpt":"In this tutorial, we deploy the 1-bit Bonsai-27B language model using the PrismML fork of llama.cpp, which provides the specialized CUDA kernels required to decode the model\u2019s Q1_0_g128 GGUF quantization format. We begin by validating the GPU runtime, installing the required Python dependencies, compiling the CUDA-enabled inference binaries, and downloading the compressed model weights from…","_links":{"self":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts\/107456","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/comments?post=107456"}],"version-history":[{"count":0,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts\/107456\/revisions"}],"wp:attachment":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/media?parent=107456"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/categories?post=107456"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/tags?post=107456"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}