Most KV quantizers are fighting the same underlying enemy: outlier channels<\/strong> \u2014 a handful of channels with disproportionately large magnitudes that dominate the quantization range and squeeze the rest of the signal into just a few representable levels. This is why naive INT2 quantization (only four levels) collapses to near-zero accuracy.<\/p>\n KIVI<\/a> established the standard baseline here. It showed that key vectors have fixed outlier channels across tokens while value vectors do not, so it quantizes keys per-channel<\/em> and values per-token<\/em>. That tuning-free 2-bit recipe cuts end-to-end peak memory (weights included) by about 2.6\u00d7, and it is the reference point the newer methods build on.<\/p>\n TurboQuant handles outliers without ever looking at your data, in two stages:<\/p>\n The selling point is theoretical: TurboQuant\u2019s distortion is provably within a small constant factor (\u2248 2.7\u00d7) of the information-theoretic lower bound. In practice it reaches essentially full-precision recall on Needle-in-a-Haystack at 4\u00d7 compression, and the paper reports absolute quality neutrality at 3.5 bits and only marginal degradation at 2.5 bits per channel. Because it needs no calibration, it works on any model untouched and doubles as a fast vector-database quantizer.<\/p>\n One caveat worth flagging: the widely repeated \u201c8\u00d7 faster attention on H100\u201d figure comes from Google\u2019s blog<\/a>, not the paper, and refers to a narrow attention-logit microbenchmark. TurboQuant\u2019s documented sweet spot is the 3\u20134 bit near-lossless regime.<\/p>\n OSCAR bets the opposite way. Its premise is that at INT2\u2019s four levels, a data-oblivious rotation is the wrong tool \u2014 blindly smoothing ranges isn\u2019t enough when there\u2019s almost no precision to spare. So OSCAR computes an attention-aware<\/em> rotation from a one-time offline calibration pass: keys are rotated into the eigenbasis of the query covariance, values into the score-weighted value covariance. A Hadamard transform plus a bit-reversal permutation then spread channel importance evenly across the quantization groups.<\/p>\n What sets OSCAR apart is that it ships as a complete system, not just an algorithm:<\/p>\n At an effective 2.28 bits, OSCAR lands within 1.42 points of BF16 on Qwen3-8B and is essentially on par on Qwen3-32B (a 0.02-point gap). On GLM-4.7-FP8 \u2014 where naive INT2 collapses to zero and data-oblivious baselines reach only low single digits \u2014 OSCAR matches BF16 and even edges slightly ahead on the reported benchmarks (within noise). Together AI reports up to 7.83\u00d7 job-level throughput and roughly 8\u00d7 KV-cache memory reduction at 100K context, with up to ~3\u00d7 faster decoding.<\/p>\n Neither \u2014 and that\u2019s the honest answer. For deployable INT2 at 128K tokens on supported models<\/strong>, OSCAR is currently the only demonstrated option that doesn\u2019t collapse, and it comes with production-ready SGLang support. For training-free, model-agnostic quantization in the 3\u20134 bit regime<\/strong>, TurboQuant offers far broader generality.<\/p>\n OSCAR\u2019s paper reports that TurboQuant drops by more than 40 points at a comparable budget \u2014 but that evaluation runs inside OSCAR\u2019s own framework, quantizes all layers, uses a single random seed, and operates well below TurboQuant\u2019s intended bit-width, so it\u2019s a weak basis for a head-to-head verdict. The more interesting possibility is that the two are complementary<\/strong>: pairing a calibration-aware rotation with an optimal scalar quantizer is a promising combination nobody has shipped yet. (Both teams have publicly noted the same idea.)<\/p>\n TurboQuant and OSCAR are both built for a single long context. Neither handles extended multi-turn conversations<\/strong>, where history piles up across many exchanges. Apple\u2019s EpiCache<\/a> is a training-free KV-cache management framework aimed exactly at that gap:<\/p>\n Across LongMemEval, RealTalk, and LoCoMo, EpiCache reports up to 40% higher accuracy than eviction baselines, near-full-cache accuracy at 4\u20136\u00d7 compression, and up to 3.5\u00d7 lower peak memory (and ~2.4\u00d7 lower latency). Because it decides which<\/em> tokens to keep rather than how precisely<\/em> to store them, it composes directly with OSCAR or TurboQuant for compounding savings.<\/p>\n \n The post The KV Cache Compression Race: TurboQuant vs OSCAR vs EpiCache<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":" Long-context large language models (LLMs) face a memory bottleneck that has nothing to do with model weights. During decoding, transformers cache the key and value (KV) vectors for every token at every layer so they don\u2019t have to recompute attention. This cache grows linearly with sequence length and batch size, and at long context with high concurrency it can dwarf the model\u2019s own footprint. Consider Llama-3.1-70B in BF16. Its KV cache costs about 0.31 MB per token (80 layers \u00d7 8 KV heads \u00d7 128 head-dim \u00d7 2 tensors \u00d7 2 bytes). At 128K tokens that is ~40 GB; at 1M tokens it exceeds 300 GB \u2014 more than the 140 GB of weights themselves. Worse, every newly decoded token has to stream the entire cache out of high-bandwidth memory (HBM), which makes decoding memory-bandwidth-bound rather than compute-bound. Shrinking the KV cache is therefore the most direct lever for cutting both cost and decode latency. Current approaches fall into roughly five families: token eviction (H2O, SnapKV), quantization (KIVI, GEAR), low-rank projection (Palu), merging (KVMerger), and architectural sharing (MLA). Recent 2026 work has pushed hard on the ultra-low-bit quantization frontier. Google and NYU\u2019s TurboQuant (ICLR 2026) and Together AI\u2019s OSCAR attack the same problem from opposite directions, while Apple\u2019s EpiCache tackles a problem neither one addresses. Most KV quantizers are fighting the same underlying enemy: outlier channels \u2014 a handful of channels with disproportionately large magnitudes that dominate the quantization range and squeeze the rest of the signal into just a few representable levels. This is why naive INT2 quantization (only four levels) collapses to near-zero accuracy. KIVI established the standard baseline here. It showed that key vectors have fixed outlier channels across tokens while value vectors do not, so it quantizes keys per-channel and values per-token. That tuning-free 2-bit recipe cuts end-to-end peak memory (weights included) by about 2.6\u00d7, and it is the reference point the newer methods build on. TurboQuant: data-oblivious and theoretically optimal TurboQuant handles outliers without ever looking at your data, in two stages: Stage one: each vector is randomly rotated so its coordinates become nearly independent and approximately Gaussian, which lets an optimal precomputed scalar (Lloyd\u2013Max) quantizer be applied per coordinate. Stage two: a 1-bit Quantized Johnson\u2013Lindenstrauss (QJL) transform is applied to the residual, giving a provably unbiased estimate of attention logits with no normalization-constant overhead. The selling point is theoretical: TurboQuant\u2019s distortion is provably within a small constant factor (\u2248 2.7\u00d7) of the information-theoretic lower bound. In practice it reaches essentially full-precision recall on Needle-in-a-Haystack at 4\u00d7 compression, and the paper reports absolute quality neutrality at 3.5 bits and only marginal degradation at 2.5 bits per channel. Because it needs no calibration, it works on any model untouched and doubles as a fast vector-database quantizer. One caveat worth flagging: the widely repeated \u201c8\u00d7 faster attention on H100\u201d figure comes from Google\u2019s blog, not the paper, and refers to a narrow attention-logit microbenchmark. TurboQuant\u2019s documented sweet spot is the 3\u20134 bit near-lossless regime. Image source: Data from the TurboQuant paper \u2013 https:\/\/arxiv.org\/abs\/2504.19874 OSCAR: attention-aware and deployment-ready OSCAR bets the opposite way. Its premise is that at INT2\u2019s four levels, a data-oblivious rotation is the wrong tool \u2014 blindly smoothing ranges isn\u2019t enough when there\u2019s almost no precision to spare. So OSCAR computes an attention-aware rotation from a one-time offline calibration pass: keys are rotated into the eigenbasis of the query covariance, values into the score-weighted value covariance. A Hadamard transform plus a bit-reversal permutation then spread channel importance evenly across the quantization groups. What sets OSCAR apart is that it ships as a complete system, not just an algorithm: Mixed-precision paged cache: sink and recent tokens stay in BF16 while the history compresses to INT2 \u2014 at 128K context only ~0.24% of tokens remain in BF16. Fused Triton kernels with full SGLang integration (paged-attention and prefix-cache compatible). Precomputed rotations (a \u201cRotationZoo\u201d) for Qwen3-4B\/8B\/32B, GLM-4.7-FP8, and MiniMax-M2.7 \u2014 no recalibration needed. At an effective 2.28 bits, OSCAR lands within 1.42 points of BF16 on Qwen3-8B and is essentially on par on Qwen3-32B (a 0.02-point gap). On GLM-4.7-FP8 \u2014 where naive INT2 collapses to zero and data-oblivious baselines reach only low single digits \u2014 OSCAR matches BF16 and even edges slightly ahead on the reported benchmarks (within noise). Together AI reports up to 7.83\u00d7 job-level throughput and roughly 8\u00d7 KV-cache memory reduction at 100K context, with up to ~3\u00d7 faster decoding. Image Source- Data from the OSCAR paper: https:\/\/arxiv.org\/abs\/2605.17757 So which one wins? Neither \u2014 and that\u2019s the honest answer. For deployable INT2 at 128K tokens on supported models, OSCAR is currently the only demonstrated option that doesn\u2019t collapse, and it comes with production-ready SGLang support. For training-free, model-agnostic quantization in the 3\u20134 bit regime, TurboQuant offers far broader generality. OSCAR\u2019s paper reports that TurboQuant drops by more than 40 points at a comparable budget \u2014 but that evaluation runs inside OSCAR\u2019s own framework, quantizes all layers, uses a single random seed, and operates well below TurboQuant\u2019s intended bit-width, so it\u2019s a weak basis for a head-to-head verdict. The more interesting possibility is that the two are complementary: pairing a calibration-aware rotation with an optimal scalar quantizer is a promising combination nobody has shipped yet. (Both teams have publicly noted the same idea.) Image source: Data from the OSCAR paper- https:\/\/arxiv.org\/abs\/2605.17757 The third axis: EpiCache TurboQuant and OSCAR are both built for a single long context. Neither handles extended multi-turn conversations, where history piles up across many exchanges. Apple\u2019s EpiCache is a training-free KV-cache management framework aimed exactly at that gap: Block-wise prefill processes history in blocks to keep peak memory bounded. Episodic clustering segments the conversation into coherent semantic \u201cepisodes,\u201d each with its own compressed cache. Episode-matched retrieval routes each query to the most relevant episode at inference time. Adaptive layer-wise budget allocation measures each layer\u2019s sensitivity to eviction and distributes the memory budget accordingly. Across LongMemEval, RealTalk, and LoCoMo, EpiCache reports up to 40% higher accuracy than eviction baselines, near-full-cache accuracy at 4\u20136\u00d7 compression, and<\/p>","protected":false},"author":2,"featured_media":98294,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"pmpro_default_level":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_pvb_checkbox_block_on_post":false,"footnotes":""},"categories":[52,5,7,1],"tags":[],"class_list":["post-98293","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-club","category-committee","category-news","category-uncategorized","pmpro-has-access"],"acf":[],"yoast_head":"\nTurboQuant: data-oblivious and theoretically optimal<\/strong><\/h2>\n
\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/06\/image-3.jpeg\")
OSCAR: attention-aware and deployment-ready<\/strong><\/h2>\n
\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/06\/image-4.jpeg\")
So which one wins?<\/strong><\/h2>\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/06\/image-6.jpeg\")
The third axis: EpiCache<\/strong><\/h2>\n
\n
Key Takeaways<\/strong><\/h2>\n
\n
\nSources<\/strong><\/h3>\n
\n