{"id":118937,"date":"2026-09-20T01:56:11","date_gmt":"2026-09-20T01:56:11","guid":{"rendered":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/"},"modified":"2026-09-20T01:56:11","modified_gmt":"2026-09-20T01:56:11","slug":"gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026","status":"publish","type":"post","link":"https:\/\/youzum.net\/es\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/","title":{"rendered":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)"},"content":{"rendered":"<h2 class=\"wp-block-heading\"><strong>First, separate 2 ideas: containers vs. quantization methods<\/strong><\/h2>\n<p class=\"wp-block-paragraph\">Most confusion comes from mixing 2 layers.<\/p>\n<p class=\"wp-block-paragraph\">A <strong>container<\/strong> defines how tensors are stored on disk. A <strong>quantization method<\/strong> defines how weights are squeezed into fewer bits.<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Containers:<\/strong> safetensors, GGUF, PyTorch pickle (<code>.bin<\/code> \/ <code>.pt<\/code>).<\/li>\n<li><strong>Methods:<\/strong> GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants.<\/li>\n<li><strong>Both at once:<\/strong> EXL2 and EXL3 are a method plus a storage layout tied to one inference library.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\"><strong>A quick memory rule of thumb<\/strong><\/h3>\n<p class=\"wp-block-paragraph\"><strong>Weight memory \u2248 parameters \u00d7 bits-per-weight \u00f7 8.<\/strong><\/p>\n<figure class=\"wp-block-table\">\n<table class=\"has-fixed-layout\">\n<thead>\n<tr>\n<th>Model<\/th>\n<th>16-bit<\/th>\n<th>~4.5 bits per weight<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>8B<\/td>\n<td>~16 GB<\/td>\n<td>~4.5 GB<\/td>\n<\/tr>\n<tr>\n<td>70B<\/td>\n<td>~140 GB<\/td>\n<td>~39 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p class=\"wp-block-paragraph\"><em>This is arithmetic, not a vendor benchmark. It covers weights only. The KV cache and runtime overhead add more on top.<\/em><\/p>\n<h2 class=\"wp-block-heading\"><strong>1. Full precision: safetensors and PyTorch<\/strong> <strong>.bin<\/strong><\/h2>\n<p class=\"wp-block-paragraph\">Unquantized models usually ship as 16-bit weights, in either <code>pytorch_model.bin<\/code> or <code>model.safetensors<\/code>.<\/p>\n<p class=\"wp-block-paragraph\">The older <code>.bin<\/code> \/ <code>.pt<\/code> files use Python pickle. Loading a pickle file can execute arbitrary code, which makes untrusted checkpoints a security risk.<\/p>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/safetensors\/safetensors\">Safetensors<\/a>, created at Hugging Face, removes that risk. A file is a small JSON header plus raw tensor buffers, with nothing executable inside. Tensors can be memory-mapped and loaded one at a time without reading the whole file. Safetensors is now listed as a <a href=\"https:\/\/pytorch.org\/projects\/safetensors\/\">PyTorch Foundation project<\/a>.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Important nuance:<\/strong> most GPTQ, AWQ, EXL2, EXL3, and MLX models are also stored in <code>.safetensors<\/code> files. The quantization lives in the tensor contents and a config file, not in a new container.<\/p>\n<h2 class=\"wp-block-heading\"><strong>2. GGUF (llama.cpp)<\/strong><\/h2>\n<h3 class=\"wp-block-heading\"><strong>What it is<\/strong><\/h3>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/ggml-org\/ggml\/blob\/master\/docs\/gguf.md\">GGUF<\/a> is a binary format for running models with GGML and GGML-based executors such as llama.cpp. It was created by Georgi Gerganov, who also leads llama.cpp (<a href=\"https:\/\/huggingface.co\/docs\/hub\/gguf\">Hugging Face docs<\/a>). It was introduced on <a href=\"https:\/\/huggingface.co\/247labs\/Llama-2-7b-Verse-Bot\">August 21, 2023<\/a> as the replacement for the older GGML format.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Why it replaced GGML<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">The older GGML, GGMF, and GGJT files could not say which architecture a model belonged to. Adding a new hyperparameter broke every existing file. GGUF switched to typed key-value metadata, so new fields can be added without breaking old files.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Design goals<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">The spec lists 5 goals: single-file deployment, extensibility, <code>mmap<\/code> compatibility, easy loading, and complete information inside the file. Unlike tensor-only formats, GGUF can carry the tokenizer, special tokens, and a Jinja chat template alongside the weights.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Reading GGUF quant names<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">The suffix in a name like <code>Q4_K_M.gguf<\/code> tells you the scheme. Figures below come from the <a href=\"https:\/\/huggingface.co\/docs\/hub\/gguf\">Hugging Face GGUF docs<\/a>.<\/p>\n<figure class=\"wp-block-table\">\n<table class=\"has-fixed-layout\">\n<thead>\n<tr>\n<th>Type<\/th>\n<th>How it works<\/th>\n<th>Bits per weight<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Q4_0 \/ Q4_1 (legacy)<\/td>\n<td>4-bit round-to-nearest in 32-weight blocks; Q4_1 adds a block minimum<\/td>\n<td>4.5 \/ 5.0*<\/td>\n<\/tr>\n<tr>\n<td>Q8_0 (legacy label)<\/td>\n<td>8-bit round-to-nearest in 32-weight blocks<\/td>\n<td>8.5*<\/td>\n<\/tr>\n<tr>\n<td>Q2_K<\/td>\n<td>16 blocks \u00d7 16 weights per super-block, 4-bit scales and mins<\/td>\n<td>2.625<\/td>\n<\/tr>\n<tr>\n<td>Q3_K<\/td>\n<td>16 blocks \u00d7 16 weights, 6-bit scales<\/td>\n<td>3.4375<\/td>\n<\/tr>\n<tr>\n<td>Q4_K<\/td>\n<td>8 blocks \u00d7 32 weights, 6-bit scales and mins<\/td>\n<td>4.5<\/td>\n<\/tr>\n<tr>\n<td>Q5_K<\/td>\n<td>8 blocks \u00d7 32 weights, 6-bit scales and mins<\/td>\n<td>5.5<\/td>\n<\/tr>\n<tr>\n<td>Q6_K<\/td>\n<td>16 blocks \u00d7 16 weights, 8-bit scales<\/td>\n<td>6.5625<\/td>\n<\/tr>\n<tr>\n<td>IQ4_XS<\/td>\n<td>256-weight super-blocks, uses an importance matrix<\/td>\n<td>4.25<\/td>\n<\/tr>\n<tr>\n<td>IQ3_XXS<\/td>\n<td>Same I-quant family<\/td>\n<td>3.06<\/td>\n<\/tr>\n<tr>\n<td>IQ2_XXS<\/td>\n<td>Same I-quant family<\/td>\n<td>2.06<\/td>\n<\/tr>\n<tr>\n<td>IQ1_S<\/td>\n<td>Same I-quant family<\/td>\n<td>1.56<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p class=\"wp-block-paragraph\"><em>*Derived by hand, not listed in the HF table: 32 weights plus a 16-bit scale (and a 16-bit minimum for Q4_1).<\/em><\/p>\n<p class=\"wp-block-paragraph\"><strong>Checking the Q4_K math<\/strong>: A super-block holds 256 weights. 256 \u00d7 4 bits = 1,024 bits. Add 8 blocks \u00d7 12 bits of scales and minimums (96 bits). Add a 16-bit super-scale and 16-bit super-minimum (32 bits). Total: 1,152 \u00f7 256 = 4.5 bits per weight.<\/p>\n<p class=\"wp-block-paragraph\"><strong>What <code>_S<\/code>, <code>_M<\/code>, <code>_L<\/code> mean<\/strong>: These are mixes, not new types. For example, llama.cpp describes Q4_K_M as using Q6_K for half of the <code>attention.wv<\/code> and <code>feed_forward.w2<\/code> tensors and Q4_K elsewhere (<a href=\"https:\/\/unsloth.ai\/docs\/basics\/inference-and-deployment\/saving-to-gguf\">Unsloth docs<\/a>). That is why a Q4_K_M file averages above 4.5 bits per weight.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Newer types<\/strong>: The HF table also lists TQ1_0 and TQ2_0 for ternary weights, plus MXFP4, a 4-bit microscaling floating-point type.<\/p>\n<p class=\"wp-block-paragraph\"><strong>A labeling quirk<\/strong>: Hugging Face files Q8_0 under \u201clegacy\u201d types. In practice, Q8_0 remains the standard near-lossless GGUF choice.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Quality vs. size<\/strong><\/h3>\n<p class=\"wp-block-paragraph\"><strong>Hugging Face\u2019s reference table for a <a href=\"https:\/\/github.com\/huggingface\/skills\/blob\/main\/skills\/huggingface-local-models\/references\/quantization.md\">Llama-2-7B-class model shows the trade-off<\/a>:<\/strong><\/p>\n<figure class=\"wp-block-table\">\n<table class=\"has-fixed-layout\">\n<thead>\n<tr>\n<th>Quant<\/th>\n<th>Perplexity<\/th>\n<th>Change vs FP16<\/th>\n<th>Size<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>FP16<\/td>\n<td>5.9565<\/td>\n<td>baseline<\/td>\n<td>13.0 GB<\/td>\n<\/tr>\n<tr>\n<td>Q8_0<\/td>\n<td>5.9584<\/td>\n<td>+0.03%<\/td>\n<td>7.0 GB<\/td>\n<\/tr>\n<tr>\n<td>Q6_K<\/td>\n<td>5.9642<\/td>\n<td>+0.13%<\/td>\n<td>5.5 GB<\/td>\n<\/tr>\n<tr>\n<td>Q5_K_M<\/td>\n<td>5.9796<\/td>\n<td>+0.39%<\/td>\n<td>4.8 GB<\/td>\n<\/tr>\n<tr>\n<td>Q4_K_M<\/td>\n<td>6.0565<\/td>\n<td>+1.68%<\/td>\n<td>4.1 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p class=\"wp-block-paragraph\"><em>Illustrative only. These numbers come from a 2023-era 7B model; newer models can react differently.<\/em><\/p>\n<h3 class=\"wp-block-heading\"><strong>Importance matrix (imatrix)<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">GGUF quantization <strong>can<\/strong> use calibration data. llama.cpp\u2019s <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/blob\/master\/tools\/imatrix\/README.md\"><code>llama-imatrix<\/code><\/a> computes an importance matrix from a text file. <code>llama-quantize --imatrix<\/code> then uses it to improve quality. For 1-bit and 2-bit mixes, <code>llama-quantize<\/code> <a href=\"https:\/\/qwen.readthedocs.io\/en\/latest\/quantization\/llama.cpp.html\">warns if no imatrix is supplied<\/a>.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Naming convention<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">The spec defines filenames as base name, size label, fine-tune, version, encoding, type, and shard. Shards use a 5-digit counter such as <code>00003-of-00009<\/code>. Optional <code>mmproj-<\/code> and <code>mtp-<\/code> prefixes mark vision projectors and multi-token-prediction draft modules.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Where GGUF runs<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">GGUF is native to llama.cpp and its ecosystem. Hugging Face documents use with <a href=\"https:\/\/huggingface.co\/docs\/hub\/gguf\">llama.cpp, LM Studio, GPT4All, and Ollama<\/a>.<\/p>\n<p class=\"wp-block-paragraph\">vLLM support exists but is limited. vLLM calls it highly experimental and under-optimized, and GGUF now needs the out-of-tree <code><a href=\"https:\/\/docs.vllm.ai\/en\/latest\/features\/quantization\/gguf\/\">vllm-gguf-plugin<\/a><\/code>.<\/p>\n<h2 class=\"wp-block-heading\"><strong>3. GPTQ<\/strong><\/h2>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/arxiv.org\/abs\/2210.17323\">GPTQ<\/a> was written by Elias Frantar (IST Austria), Saleh Ashkboos and Torsten Hoefler (ETH Zurich), and Dan Alistarh (IST Austria &amp; Neural Magic). It first appeared on arXiv on October 31, 2022. It was published at <a href=\"https:\/\/github.com\/ist-daslab\/gptq\">ICLR 2023<\/a>.<\/p>\n<h3 class=\"wp-block-heading\"><strong>How it works<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">GPTQ is a one-shot, post-training weight quantization method. It uses approximate second-order (Hessian) information to decide how to round weights. Rounding error in one column is compensated by adjusting weights not yet quantized. It needs a small calibration dataset but no retraining.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Main results<\/strong><\/h3>\n<ul class=\"wp-block-list\">\n<li>Quantized 175B-parameter models in about 4 GPU hours, down to 3 or 4 bits per weight (<a href=\"https:\/\/arxiv.org\/abs\/2210.17323\">arXiv<\/a>).<\/li>\n<li>Reported negligible accuracy loss at those bit widths.<\/li>\n<li>End-to-end speedups over FP16 of about 3.25x on NVIDIA A100 and 4.5x on A6000 (<a href=\"https:\/\/huggingface.co\/papers\/2210.17323\">HF paper page<\/a>).<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Reading GPTQ names<\/h3>\n<p class=\"wp-block-paragraph\">GPTQ repos often include <code>GPTQ<\/code> or tags like <code>4bit-128g<\/code> in the name.<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Group size (<code>128g<\/code>):<\/strong> one scale per 128 weights. Smaller groups improve accuracy but add a little size.<\/li>\n<li><strong>Act-order (<code>desc_act<\/code>):<\/strong> quantizes columns in order of importance, usually improving accuracy.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\"><strong>Tooling status in 2026<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">The original <a href=\"https:\/\/github.com\/vllm-project\/vllm\/issues\/30136\">AutoGPTQ library is no longer maintained<\/a>.<\/p>\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/github.com\/ModelCloud\/GPTQModel\">GPTQModel<\/a> states it has fully supplanted AutoGPTQ and AutoAWQ for Transformers, Optimum, and PEFT. Its output runs in Transformers, vLLM, and SGLang.<\/li>\n<li><a href=\"https:\/\/github.com\/vllm-project\/llm-compressor\">llm-compressor<\/a> also implements GPTQ, but saves results in the <code>compressed-tensors<\/code> format.<\/li>\n<li>Hugging Face estimates GPTQ calibration for an <a href=\"https:\/\/huggingface.co\/docs\/transformers\/quantization\/selecting\">8B model at about 20 minutes on 1 A100<\/a>.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\"><strong>4. AWQ<\/strong><\/h2>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/arxiv.org\/abs\/2306.00978\">AWQ<\/a> (Activation-aware Weight Quantization) comes from Song Han\u2019s group at MIT. It first appeared on arXiv on June 1, 2023. It won the <a href=\"https:\/\/ui.adsabs.harvard.edu\/abs\/2023arXiv230600978L\/abstract\">MLSys 2024 Best Paper Award<\/a>.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Core idea<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">Not all weights matter equally. Protecting roughly 1% of \u2018salient\u2019 <a href=\"https:\/\/proceedings.mlsys.org\/paper_files\/paper\/2024\/hash\/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html\">weights sharply reduces quantization error<\/a>.<\/p>\n<p class=\"wp-block-paragraph\"><strong>The twist:<\/strong> AWQ finds those salient channels by looking at <strong>activation<\/strong> magnitudes, not the weights themselves.<\/p>\n<p class=\"wp-block-paragraph\">It does not store those channels at higher precision. Instead, it scales them up through a mathematically equivalent transformation, keeping a uniform, hardware-friendly format. AWQ uses no backpropagation or reconstruction, so it is less likely to overfit its calibration set.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Speed and cost<\/strong><\/h3>\n<ul class=\"wp-block-list\">\n<li>The paper\u2019s TinyChat runtime ran more than 3x faster than the Hugging Face FP16 implementation on desktop and mobile GPUs.<\/li>\n<li>Hugging Face estimates AWQ calibration for an 8B model at <a href=\"https:\/\/huggingface.co\/docs\/transformers\/quantization\/selecting\">about 10 minutes on 1 A100, roughly half of GPTQ\u2019s estimate<\/a>.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\"><strong>Tooling status in 2026<\/strong><\/h3>\n<ul class=\"wp-block-list\">\n<li>AutoAWQ is officially deprecated. Its last tested setup was <a href=\"https:\/\/pypi.org\/project\/autoawq\/\">Torch 2.6.0 and Transformers 4.51.3<\/a>.<\/li>\n<li>vLLM adopted the functionality into <a href=\"https:\/\/docs.vllm.ai\/en\/stable\/features\/quantization\/auto_awq\/\">llm-compressor<\/a>, now the recommended AWQ workflow.<\/li>\n<li>MLX-LM also supports AWQ on Apple Silicon.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\"><strong>5. EXL2 (ExLlamaV2)<\/strong><\/h2>\n<h3 class=\"wp-block-heading\"><strong>What it is<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">EXL2 is the native format of <a href=\"https:\/\/github.com\/turboderp-org\/exllamav2\">ExLlamaV2<\/a>, an inference library by turboderp for consumer GPUs. It uses the same optimization method as GPTQ and supports 2, 3, 4, 5, 6, and 8-bit quantization.<\/p>\n<h3 class=\"wp-block-heading\"><strong>What makes it different<\/strong><\/h3>\n<ul class=\"wp-block-list\">\n<li><strong>Any average bitrate from 2 to 8 bits per weight<\/strong>: Quantization levels can be mixed across and within layers.<\/li>\n<li><strong>Column-level mixing<\/strong>: More important columns inside a layer can get more bits.<\/li>\n<li><strong>Automatic allocation<\/strong>: The converter quantizes each matrix several ways and measures error against calibration data. It then picks settings that minimize the worst-case error while hitting the target bitrate.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">That is why EXL2 files carry names like <code>4.65bpw<\/code> instead of <code>4-bit<\/code>.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Runtime<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">TabbyAPI is the official recommended server, providing an OpenAI-compatible API. EXL2 renames some tensors so <a href=\"https:\/\/github.com\/turboderp-org\/exllamav3\">every model looks like a Llama variant internally<\/a>. That makes EXL2 hard to reuse in other frameworks.<\/p>\n<h2 class=\"wp-block-heading\"><strong>6. EXL3 (ExLlamaV3)<\/strong><\/h2>\n<h3 class=\"wp-block-heading\"><strong>What it is<\/strong><\/h3>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/turboderp-org\/exllamav3\/blob\/master\/doc\/exl3.md\">EXL3<\/a> is the successor format, built on <a href=\"https:\/\/arxiv.org\/abs\/2406.11235\">QTIP<\/a> from Cornell RelaxML. QTIP uses trellis-coded quantization with incoherence processing and was published at NeurIPS 2024.<\/p>\n<p class=\"wp-block-paragraph\">EXL3 keeps QTIP\u2019s procedural codebook and trellis encoding. It changes how tensors are regularized and packed.<\/p>\n<h3 class=\"wp-block-heading\"><strong>Why it matters<\/strong><\/h3>\n<ul class=\"wp-block-list\">\n<li><strong>Simple conversion<\/strong>: You supply a Hugging Face model and a target bitrate. Hessians are computed on the fly during conversion (<a href=\"https:\/\/github.com\/turboderp-org\/exllamav3\">README<\/a>).<\/li>\n<li><strong>Reasonable cost<\/strong>: Conversion takes minutes for small models and a few hours for 70B+ on 1 RTX 4090-class GPU. For contrast, the README says AQLM on a 70B model takes about 720 A100 GPU-hours.<\/li>\n<li><strong>Very low bitrates<\/strong>: Llama-3.1-70B stays coherent at 1.6 bits per weight. With a 3-bit output layer and a 4,096-token cache, it fits in under 16 GB of VRAM.<\/li>\n<li><strong>Portable layout<\/strong>: EXL3 largely keeps the original tensor structure, unlike EXL2.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\"><strong>Features<\/strong><\/h3>\n<p class=\"wp-block-paragraph\">ExLlamaV3 adds 2\u20138 bit KV-cache quantization, tensor- and expert-parallel inference, speculative decoding, multimodal support, and a Transformers plugin. Recent releases add <a href=\"https:\/\/github.com\/turboderp-org\/exllamav3\">CPU offloading for large MoE models<\/a>.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Hardware note:<\/strong> ExLlamaV3 requires CUDA 12.4 or later. Its README has listed ROCm support as a to-do item.<\/p>\n<h2 class=\"wp-block-heading\"><strong>7. bitsandbytes (NF4 \/ INT8): quantize at load time<\/strong><\/h2>\n<p class=\"wp-block-paragraph\">bitsandbytes is usually not something you download pre-quantized. You load a 16-bit model and quantize it on the fly.<\/p>\n<p class=\"wp-block-paragraph\">Its 4-bit mode comes from <a href=\"https:\/\/arxiv.org\/abs\/2305.14314\">QLoRA<\/a>:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>NF4 (4-bit NormalFloat):<\/strong> a data type designed for normally distributed weights.<\/li>\n<li><strong>Double quantization:<\/strong> the quantization constants themselves are quantized to save more memory.<\/li>\n<li><strong>Result:<\/strong> fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning performance.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\"><strong><a href=\"https:\/\/huggingface.co\/docs\/transformers\/quantization\/selecting\">Hugging Face\u2019s guidance<\/a>:<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>No calibration dataset needed.<\/li>\n<li>Inference speedup is not guaranteed.<\/li>\n<li>It remains the standard path for QLoRA fine-tuning via PEFT.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\"><strong>8. MLX (Apple Silicon)<\/strong><\/h2>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/github.com\/ml-explore\/mlx-lm\">MLX-LM<\/a> is a Python package for running and fine-tuning LLMs on Apple Silicon with MLX. MLX comes from <a href=\"https:\/\/huggingface.co\/docs\/hub\/en\/mlx\">Apple Machine Learning Research<\/a>.<\/p>\n<p class=\"wp-block-paragraph\">MLX models are safetensors with MLX-specific quantized weights. <code>mlx_lm.convert<\/code> with <code>-q<\/code> quantizes a Hugging Face model and can upload it to the <code>mlx-community<\/code> organization.<\/p>\n<p class=\"wp-block-paragraph\">On a Mac, both GGUF (via llama.cpp) and MLX are strong options.<\/p>\n<h2 class=\"wp-block-heading\"><strong>9. Other names you will meet<\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>compressed-tensors \/ FP8<\/strong>: The on-disk format written by <a href=\"https:\/\/docs.vllm.ai\/projects\/llm-compressor\/en\/latest\/guides\/saving_a_model\/\">llm-compressor<\/a>. It covers FP8, INT4\/INT8 weight-only schemes, NVFP4, and sparsity. FP8 needs newer hardware such as NVIDIA H100\/H200\/B100 or AMD MI300 to deliver its full benefit (<a href=\"https:\/\/huggingface.co\/docs\/transformers\/quantization\/concept_guide\">concept guide<\/a>).<\/li>\n<li><strong>HQQ<\/strong>: Fast, calibration-free quantization from 8 down to 1 bit. Accuracy can drop sharply below 4 bits.<\/li>\n<li><strong>SINQ<\/strong>: Another calibration-free, on-the-fly method now listed in Transformers.<\/li>\n<li><strong>AQLM, SpQR, VPTQ, HIGGS<\/strong>: Research methods pushing below 2 bits per weight.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\"><strong>Comparison table<\/strong><\/h2>\n<figure class=\"wp-block-table\">\n<table class=\"has-fixed-layout\">\n<thead>\n<tr>\n<th>Format<\/th>\n<th>What it is<\/th>\n<th>Calibration<\/th>\n<th>Best hardware<\/th>\n<th>Main runtimes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Safetensors (16-bit)<\/td>\n<td>Container<\/td>\n<td>None<\/td>\n<td>GPUs with enough VRAM<\/td>\n<td>Transformers, vLLM, SGLang<\/td>\n<\/tr>\n<tr>\n<td>GGUF<\/td>\n<td>Container + quant types<\/td>\n<td>Optional (imatrix)<\/td>\n<td>CPU, Apple Silicon, CPU+GPU split<\/td>\n<td>llama.cpp, Ollama, LM Studio<\/td>\n<\/tr>\n<tr>\n<td>GPTQ<\/td>\n<td>Method (in safetensors)<\/td>\n<td>Required<\/td>\n<td>GPUs<\/td>\n<td>vLLM, SGLang, Transformers<\/td>\n<\/tr>\n<tr>\n<td>AWQ<\/td>\n<td>Method (in safetensors)<\/td>\n<td>Required<\/td>\n<td>GPUs<\/td>\n<td>vLLM, SGLang, Transformers<\/td>\n<\/tr>\n<tr>\n<td>EXL2<\/td>\n<td>Method + layout<\/td>\n<td>Required<\/td>\n<td>Consumer NVIDIA GPUs<\/td>\n<td>ExLlamaV2, TabbyAPI<\/td>\n<\/tr>\n<tr>\n<td>EXL3<\/td>\n<td>Method + layout<\/td>\n<td>Built into conversion<\/td>\n<td>Consumer NVIDIA GPUs<\/td>\n<td>ExLlamaV3, TabbyAPI<\/td>\n<\/tr>\n<tr>\n<td>bitsandbytes NF4<\/td>\n<td>On-the-fly method<\/td>\n<td>None<\/td>\n<td>NVIDIA (and Intel) GPUs<\/td>\n<td>Transformers, PEFT<\/td>\n<\/tr>\n<tr>\n<td>MLX<\/td>\n<td>Method (in safetensors)<\/td>\n<td>None by default<\/td>\n<td>Apple Silicon<\/td>\n<td>MLX-LM<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n<h2 class=\"wp-block-heading\"><strong>Which should you pick?<\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>Mac, CPU-only, or a model bigger than your VRAM:<\/strong> GGUF. Start at Q4_K_M; move to Q5_K_M or Q6_K if memory allows.<\/li>\n<li><strong>Serving many users on data-center GPUs:<\/strong> AWQ or GPTQ in vLLM\/SGLang, or FP8 on Hopper\/Blackwell-class cards.<\/li>\n<li><strong>One user, consumer NVIDIA GPUs, maximum tokens per second:<\/strong> EXL3 via TabbyAPI (EXL2 for older setups).<\/li>\n<li><strong>Fine-tuning on a budget:<\/strong> bitsandbytes NF4 with QLoRA.<\/li>\n<li><strong>Apple Silicon with a Python workflow or fine-tuning:<\/strong> MLX.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\"><strong>Key Takeaways<\/strong><\/h2>\n<ul class=\"wp-block-list\">\n<li>A file format (GGUF, safetensors) is not the same thing as a quantization method (GPTQ, AWQ).<\/li>\n<li>GGUF is the default for CPU, Apple Silicon, and mixed CPU+GPU local inference.<\/li>\n<li>GPTQ and AWQ are the 4-bit workhorses for GPU serving in vLLM, SGLang, and Transformers.<\/li>\n<li>EXL2 and EXL3 target fast single-user inference and fine-grained bitrates on consumer GPUs.<\/li>\n<li>AutoGPTQ and AutoAWQ are unmaintained; use GPTQModel or llm-compressor instead.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n<\/p><p class=\"wp-block-paragraph\">\n<h4 class=\"wp-block-heading\"><strong>Sources<\/strong>:<\/h4>\n<\/p><p class=\"wp-block-paragraph\"><strong>Specs and official docs<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>GGUF specification \u2014 https:\/\/github.com\/ggml-org\/ggml\/blob\/master\/docs\/gguf.md<\/li>\n<li>Hugging Face Hub: GGUF \u2014 https:\/\/huggingface.co\/docs\/hub\/gguf<\/li>\n<li>llama.cpp imatrix README \u2014 https:\/\/github.com\/ggml-org\/llama.cpp\/blob\/master\/tools\/imatrix\/README.md<\/li>\n<li>llama.cpp quantize README \u2014 https:\/\/github.com\/ggml-org\/llama.cpp\/blob\/master\/tools\/quantize\/README.md<\/li>\n<li>Qwen docs: llama.cpp quantization \u2014 https:\/\/qwen.readthedocs.io\/en\/latest\/quantization\/llama.cpp.html<\/li>\n<li>Unsloth docs: saving to GGUF \u2014 https:\/\/unsloth.ai\/docs\/basics\/inference-and-deployment\/saving-to-gguf<\/li>\n<li>Hugging Face skills: quantization reference \u2014 https:\/\/github.com\/huggingface\/skills\/blob\/main\/skills\/huggingface-local-models\/references\/quantization.md<\/li>\n<li>vLLM: GGUF \u2014 https:\/\/docs.vllm.ai\/en\/latest\/features\/quantization\/gguf\/<\/li>\n<li>vLLM: AutoAWQ \u2014 https:\/\/docs.vllm.ai\/en\/stable\/features\/quantization\/auto_awq\/<\/li>\n<li>vLLM RFC #30136 (legacy quantization formats) \u2014 https:\/\/github.com\/vllm-project\/vllm\/issues\/30136<\/li>\n<li>llm-compressor \u2014 https:\/\/github.com\/vllm-project\/llm-compressor<\/li>\n<li>llm-compressor: saving a model \u2014 https:\/\/docs.vllm.ai\/projects\/llm-compressor\/en\/latest\/guides\/saving_a_model\/<\/li>\n<li>Transformers: selecting a quantization method \u2014 https:\/\/huggingface.co\/docs\/transformers\/quantization\/selecting<\/li>\n<li>Transformers: quantization concepts \u2014 https:\/\/huggingface.co\/docs\/transformers\/quantization\/concept_guide<\/li>\n<li>Safetensors \u2014 https:\/\/github.com\/safetensors\/safetensors<\/li>\n<li>PyTorch Foundation: Safetensors \u2014 https:\/\/pytorch.org\/projects\/safetensors\/<\/li>\n<li>Hugging Face Hub: MLX \u2014 https:\/\/huggingface.co\/docs\/hub\/en\/mlx<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\"><strong>Papers<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>GPTQ (arXiv 2210.17323) \u2014 https:\/\/arxiv.org\/abs\/2210.17323<\/li>\n<li>GPTQ official code (ICLR 2023) \u2014 https:\/\/github.com\/ist-daslab\/gptq<\/li>\n<li>AWQ (arXiv 2306.00978) \u2014 https:\/\/arxiv.org\/abs\/2306.00978<\/li>\n<li>AWQ (MLSys 2024) \u2014 https:\/\/proceedings.mlsys.org\/paper_files\/paper\/2024\/hash\/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html<\/li>\n<li>QLoRA (arXiv 2305.14314) \u2014 https:\/\/arxiv.org\/abs\/2305.14314<\/li>\n<li>QTIP (arXiv 2406.11235) \u2014 https:\/\/arxiv.org\/abs\/2406.11235<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\"><strong>Libraries<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>ExLlamaV2 \u2014 https:\/\/github.com\/turboderp-org\/exllamav2<\/li>\n<li>ExLlamaV3 \u2014 https:\/\/github.com\/turboderp-org\/exllamav3<\/li>\n<li>EXL3 format notes \u2014 https:\/\/github.com\/turboderp-org\/exllamav3\/blob\/master\/doc\/exl3.md<\/li>\n<li>GPTQModel \u2014 https:\/\/github.com\/ModelCloud\/GPTQModel<\/li>\n<li>AutoAWQ (deprecated) \u2014 https:\/\/pypi.org\/project\/autoawq\/<\/li>\n<li>MLX-LM \u2014 https:\/\/github.com\/ml-explore\/mlx-lm<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\"><strong>Community discussion<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>r\/LocalLLaMA thread on model formats \u2014 https:\/\/www.reddit.com\/r\/LocalLLaMA\/comments\/1ayd4xr\/for_those_who_dont_know_what_different_model\/<\/li>\n<\/ul>\n<p>The post <a href=\"https:\/\/www.marktechpost.com\/2026\/09\/18\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\">GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)<\/a> appeared first on <a href=\"https:\/\/www.marktechpost.com\/\">MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>First, separate 2 ideas: containers vs. quantization methods Most confusion comes from mixing 2 layers. A container defines how tensors are stored on disk. A quantization method defines how weights are squeezed into fewer bits. Containers: safetensors, GGUF, PyTorch pickle (.bin \/ .pt). Methods: GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants. Both at once: EXL2 and EXL3 are a method plus a storage layout tied to one inference library. A quick memory rule of thumb Weight memory \u2248 parameters \u00d7 bits-per-weight \u00f7 8. Model 16-bit ~4.5 bits per weight 8B ~16 GB ~4.5 GB 70B ~140 GB ~39 GB This is arithmetic, not a vendor benchmark. It covers weights only. The KV cache and runtime overhead add more on top. 1. Full precision: safetensors and PyTorch .bin Unquantized models usually ship as 16-bit weights, in either pytorch_model.bin or model.safetensors. The older .bin \/ .pt files use Python pickle. Loading a pickle file can execute arbitrary code, which makes untrusted checkpoints a security risk. Safetensors, created at Hugging Face, removes that risk. A file is a small JSON header plus raw tensor buffers, with nothing executable inside. Tensors can be memory-mapped and loaded one at a time without reading the whole file. Safetensors is now listed as a PyTorch Foundation project. Important nuance: most GPTQ, AWQ, EXL2, EXL3, and MLX models are also stored in .safetensors files. The quantization lives in the tensor contents and a config file, not in a new container. 2. GGUF (llama.cpp) What it is GGUF is a binary format for running models with GGML and GGML-based executors such as llama.cpp. It was created by Georgi Gerganov, who also leads llama.cpp (Hugging Face docs). It was introduced on August 21, 2023 as the replacement for the older GGML format. Why it replaced GGML The older GGML, GGMF, and GGJT files could not say which architecture a model belonged to. Adding a new hyperparameter broke every existing file. GGUF switched to typed key-value metadata, so new fields can be added without breaking old files. Design goals The spec lists 5 goals: single-file deployment, extensibility, mmap compatibility, easy loading, and complete information inside the file. Unlike tensor-only formats, GGUF can carry the tokenizer, special tokens, and a Jinja chat template alongside the weights. Reading GGUF quant names The suffix in a name like Q4_K_M.gguf tells you the scheme. Figures below come from the Hugging Face GGUF docs. Type How it works Bits per weight Q4_0 \/ Q4_1 (legacy) 4-bit round-to-nearest in 32-weight blocks; Q4_1 adds a block minimum 4.5 \/ 5.0* Q8_0 (legacy label) 8-bit round-to-nearest in 32-weight blocks 8.5* Q2_K 16 blocks \u00d7 16 weights per super-block, 4-bit scales and mins 2.625 Q3_K 16 blocks \u00d7 16 weights, 6-bit scales 3.4375 Q4_K 8 blocks \u00d7 32 weights, 6-bit scales and mins 4.5 Q5_K 8 blocks \u00d7 32 weights, 6-bit scales and mins 5.5 Q6_K 16 blocks \u00d7 16 weights, 8-bit scales 6.5625 IQ4_XS 256-weight super-blocks, uses an importance matrix 4.25 IQ3_XXS Same I-quant family 3.06 IQ2_XXS Same I-quant family 2.06 IQ1_S Same I-quant family 1.56 *Derived by hand, not listed in the HF table: 32 weights plus a 16-bit scale (and a 16-bit minimum for Q4_1). Checking the Q4_K math: A super-block holds 256 weights. 256 \u00d7 4 bits = 1,024 bits. Add 8 blocks \u00d7 12 bits of scales and minimums (96 bits). Add a 16-bit super-scale and 16-bit super-minimum (32 bits). Total: 1,152 \u00f7 256 = 4.5 bits per weight. What _S, _M, _L mean: These are mixes, not new types. For example, llama.cpp describes Q4_K_M as using Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K elsewhere (Unsloth docs). That is why a Q4_K_M file averages above 4.5 bits per weight. Newer types: The HF table also lists TQ1_0 and TQ2_0 for ternary weights, plus MXFP4, a 4-bit microscaling floating-point type. A labeling quirk: Hugging Face files Q8_0 under \u201clegacy\u201d types. In practice, Q8_0 remains the standard near-lossless GGUF choice. Quality vs. size Hugging Face\u2019s reference table for a Llama-2-7B-class model shows the trade-off: Quant Perplexity Change vs FP16 Size FP16 5.9565 baseline 13.0 GB Q8_0 5.9584 +0.03% 7.0 GB Q6_K 5.9642 +0.13% 5.5 GB Q5_K_M 5.9796 +0.39% 4.8 GB Q4_K_M 6.0565 +1.68% 4.1 GB Illustrative only. These numbers come from a 2023-era 7B model; newer models can react differently. Importance matrix (imatrix) GGUF quantization can use calibration data. llama.cpp\u2019s llama-imatrix computes an importance matrix from a text file. llama-quantize &#8211;imatrix then uses it to improve quality. For 1-bit and 2-bit mixes, llama-quantize warns if no imatrix is supplied. Naming convention The spec defines filenames as base name, size label, fine-tune, version, encoding, type, and shard. Shards use a 5-digit counter such as 00003-of-00009. Optional mmproj- and mtp- prefixes mark vision projectors and multi-token-prediction draft modules. Where GGUF runs GGUF is native to llama.cpp and its ecosystem. Hugging Face documents use with llama.cpp, LM Studio, GPT4All, and Ollama. vLLM support exists but is limited. vLLM calls it highly experimental and under-optimized, and GGUF now needs the out-of-tree vllm-gguf-plugin. 3. GPTQ GPTQ was written by Elias Frantar (IST Austria), Saleh Ashkboos and Torsten Hoefler (ETH Zurich), and Dan Alistarh (IST Austria &amp; Neural Magic). It first appeared on arXiv on October 31, 2022. It was published at ICLR 2023. How it works GPTQ is a one-shot, post-training weight quantization method. It uses approximate second-order (Hessian) information to decide how to round weights. Rounding error in one column is compensated by adjusting weights not yet quantized. It needs a small calibration dataset but no retraining. Main results Quantized 175B-parameter models in about 4 GPU hours, down to 3 or 4 bits per weight (arXiv). Reported negligible accuracy loss at those bit widths. End-to-end speedups over FP16 of about 3.25x on NVIDIA A100 and 4.5x on A6000 (HF paper page). Reading GPTQ names GPTQ repos often include GPTQ or tags like 4bit-128g in the name. Group size (128g): one scale per 128 weights. Smaller groups improve accuracy but add a little size. Act-order (desc_act):<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"pmpro_default_level":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_pvb_checkbox_block_on_post":false,"footnotes":""},"categories":[52,5,7,1],"tags":[],"class_list":["post-118937","post","type-post","status-publish","format-standard","hentry","category-ai-club","category-committee","category-news","category-uncategorized","pmpro-has-access"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum<\/title>\n<meta name=\"description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/youzum.net\/es\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\" \/>\n<meta property=\"og:locale\" content=\"es_ES\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum\" \/>\n<meta property=\"og:description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta property=\"og:url\" content=\"https:\/\/youzum.net\/es\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\" \/>\n<meta property=\"og:site_name\" content=\"YouZum\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DroneAssociationTH\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-20T01:56:11+00:00\" \/>\n<meta name=\"author\" content=\"admin NU\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Escrito por\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin NU\" \/>\n\t<meta name=\"twitter:label2\" content=\"Tiempo de lectura\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutos\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\"},\"author\":{\"name\":\"admin NU\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\"},\"headline\":\"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)\",\"datePublished\":\"2026-09-20T01:56:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\"},\"wordCount\":2321,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"articleSection\":[\"AI\",\"Committee\",\"News\",\"Uncategorized\"],\"inLanguage\":\"es\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\",\"url\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\",\"name\":\"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum\",\"isPartOf\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#website\"},\"datePublished\":\"2026-09-20T01:56:11+00:00\",\"description\":\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\",\"breadcrumb\":{\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#breadcrumb\"},\"inLanguage\":\"es\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/youzum.net\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/yousum.gpucore.co\/#website\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"name\":\"YouSum\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/yousum.gpucore.co\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"es\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\",\"name\":\"Drone Association Thailand\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"es\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"width\":300,\"height\":300,\"caption\":\"Drone Association Thailand\"},\"image\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DroneAssociationTH\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\",\"name\":\"admin NU\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"es\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"caption\":\"admin NU\"},\"url\":\"https:\/\/youzum.net\/es\/members\/adminnu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/youzum.net\/es\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/","og_locale":"es_ES","og_type":"article","og_title":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum","og_description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","og_url":"https:\/\/youzum.net\/es\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/","og_site_name":"YouZum","article_publisher":"https:\/\/www.facebook.com\/DroneAssociationTH\/","article_published_time":"2026-09-20T01:56:11+00:00","author":"admin NU","twitter_card":"summary_large_image","twitter_misc":{"Escrito por":"admin NU","Tiempo de lectura":"12 minutos"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#article","isPartOf":{"@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/"},"author":{"name":"admin NU","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c"},"headline":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)","datePublished":"2026-09-20T01:56:11+00:00","mainEntityOfPage":{"@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/"},"wordCount":2321,"commentCount":0,"publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"articleSection":["AI","Committee","News","Uncategorized"],"inLanguage":"es","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/","url":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/","name":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026) - YouZum","isPartOf":{"@id":"https:\/\/yousum.gpucore.co\/#website"},"datePublished":"2026-09-20T01:56:11+00:00","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","breadcrumb":{"@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#breadcrumb"},"inLanguage":"es","potentialAction":[{"@type":"ReadAction","target":["https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/youzum.net\/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/youzum.net\/"},{"@type":"ListItem","position":2,"name":"GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)"}]},{"@type":"WebSite","@id":"https:\/\/yousum.gpucore.co\/#website","url":"https:\/\/yousum.gpucore.co\/","name":"YouSum","description":"","publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/yousum.gpucore.co\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"es"},{"@type":"Organization","@id":"https:\/\/yousum.gpucore.co\/#organization","name":"Drone Association Thailand","url":"https:\/\/yousum.gpucore.co\/","logo":{"@type":"ImageObject","inLanguage":"es","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","width":300,"height":300,"caption":"Drone Association Thailand"},"image":{"@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DroneAssociationTH\/"]},{"@type":"Person","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c","name":"admin NU","image":{"@type":"ImageObject","inLanguage":"es","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","caption":"admin NU"},"url":"https:\/\/youzum.net\/es\/members\/adminnu\/"}]}},"rttpg_featured_image_url":null,"rttpg_author":{"display_name":"admin NU","author_link":"https:\/\/youzum.net\/es\/members\/adminnu\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/youzum.net\/es\/category\/ai-club\/\" rel=\"category tag\">AI<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/committee\/\" rel=\"category tag\">Committee<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/news\/\" rel=\"category tag\">News<\/a> <a href=\"https:\/\/youzum.net\/es\/category\/uncategorized\/\" rel=\"category tag\">Uncategorized<\/a>","rttpg_excerpt":"First, separate 2 ideas: containers vs. quantization methods Most confusion comes from mixing 2 layers. A container defines how tensors are stored on disk. A quantization method defines how weights are squeezed into fewer bits. Containers: safetensors, GGUF, PyTorch pickle (.bin \/ .pt). Methods: GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants. Both at once:&hellip;","_links":{"self":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts\/118937","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/comments?post=118937"}],"version-history":[{"count":0,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/posts\/118937\/revisions"}],"wp:attachment":[{"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/media?parent=118937"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/categories?post=118937"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/youzum.net\/es\/wp-json\/wp\/v2\/tags?post=118937"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}