What is Tokenization Drift and How to Fix It?
A model can behave perfectly one moment and degrade the next—without any change to your data, pipeline, or logic. The root cause often lies in something far more subtle: how your input is tokenized. Before a model processes text, it converts it into token IDs, and even minor formatting differences—like spacing, line breaks, or punctuation—can produce entirely different token sequences. This phenomenon is known as tokenization drift: when small surface-level changes push your input into a different region of token space, leading to unpredictable shifts in model behavior. The impact goes deeper than just token IDs. During instruction tuning, models learn not only tasks but also the structure in which those tasks are presented—specific separators, prefixes, and formatting patterns. When your prompt deviates from these learned patterns, you are no longer operating within the model’s familiar distribution. The result isn’t confusion—it’s a model doing its best on inputs it was never optimized to handle. In this article, we’ll break this down using the GPT-2 tokenizer to show how small formatting changes affect tokens, and build a simple metric to measure drift across prompts. Then, we’ll implement a lightweight prompt optimization loop to pick formats that keep your inputs consistent and reliable. Setting up the dependencies Copy CodeCopiedUse a different Browser import numpy as np import matplotlib.pyplot as plt import matplotlib.patches as mpatches import seaborn as sns from collections import defaultdict from sklearn.decomposition import PCA In this code block, we are loading the GPT-2 tokenizer — the same Byte-Pair Encoding scheme used by GPT-4, LLaMA, and Mistral. We use GPT-2 specifically because it requires no auth token and demonstrates the space-prefix artifact identically to every modern production tokenizer. Copy CodeCopiedUse a different Browser from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained(“gpt2”) print(“Tokenizer loaded:”, tokenizer.__class__.__name__) print(“Vocab size:”, tokenizer.vocab_size) Tokenization Artifact Demo We take seven words and test each in two forms—once with a leading space and once without—then encode them using the GPT-2 tokenizer. By setting add_special_tokens=False, we ensure we’re only measuring the token IDs for the words themselves, without any extra padding or special markers. The results are striking. Not a single pair produces the same token ID—every word is treated as completely different depending on whether it has a leading space. Even more interesting, some words without the space don’t map to a single token at all. For example, “classify” becomes two tokens [4871, 1958], while “ classify” is a single token [36509]. This means the model doesn’t just see a different ID—it sees a different sequence length, which shifts how attention is computed for everything that follows. Copy CodeCopiedUse a different Browser pairs = [ (” classify”, “classify”), (” answer”, “answer”), (” positive”, “positive”), (” negative”, “negative”), (” sentiment”, “sentiment”), (” output”, “output”), (” label”, “label”), ] print(“=” * 60) print(f”{‘Token (with space)’:<22} {‘ID’:>6} {‘Token (no space)’:<20} {‘ID’:>6} {‘Same?’:>6}”) print(“=” * 60) for with_space, without_space in pairs: id_ws = tokenizer.encode(with_space, add_special_tokens=False) id_nws = tokenizer.encode(without_space, add_special_tokens=False) match = “✓” if id_ws == id_nws else “✗ DIFFERENT” print(f”{repr(with_space):<22} {str(id_ws):>8} {repr(without_space):<20} {str(id_nws):>8} {match}”) print() print(“Key takeaway: Leading spaces create DIFFERENT token IDs.”) print(“To the model, ‘ classify’ and ‘classify’ are as distinct as ‘apple’ and ‘orange’.”) Visualising the Token ID Shift We plot two charts to make the token ID gap visual. The left chart shows the raw IDs side-by-side — blue for space-prefixed, red for bare — and the right chart plots the absolute distance between each pair. Copy CodeCopiedUse a different Browser words = [p[1] for p in pairs] ids_ws = [tokenizer.encode(” ” + w, add_special_tokens=False)[0] for w in words] ids_nws = [tokenizer.encode(w, add_special_tokens=False)[0] for w in words] delta = [abs(a – b) for a, b in zip(ids_ws, ids_nws)] x = np.arange(len(words)) width = 0.35 fig, axes = plt.subplots(1, 2, figsize=(14, 5)) fig.patch.set_facecolor(“#FAFAF8”) # Left: side-by-side token IDs ax = axes[0] ax.set_facecolor(“#FAFAF8″) bars1 = ax.bar(x – width/2, ids_ws, width, label=’With leading space’, color=”#3B6FE0″, alpha=0.85) bars2 = ax.bar(x + width/2, ids_nws, width, label=’Without leading space’, color=”#E05C3B”, alpha=0.85) ax.set_xticks(x) ax.set_xticklabels(words, rotation=30, ha=”right”, fontsize=9) ax.set_ylabel(“Token ID”, fontsize=10) ax.set_title(“Token IDs: ‘ word’ vs ‘word'”, fontsize=12, fontweight=”bold”, pad=12) ax.legend(fontsize=9) ax.spines[[“top”, “right”]].set_visible(False) ax.grid(axis=”y”, alpha=0.3) for bar in bars1: ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 50, str(int(bar.get_height())), ha=”center”, va=”bottom”, fontsize=7, color=”#3B6FE0″) for bar in bars2: ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 50, str(int(bar.get_height())), ha=”center”, va=”bottom”, fontsize=7, color=”#E05C3B”) # Right: delta ax2 = axes[1] ax2.set_facecolor(“#FAFAF8”) color_bars = [“#E05C3B” if d > 500 else “#F0A070” if d > 100 else “#A8C4F0” for d in delta] bars3 = ax2.bar(words, delta, color=color_bars, alpha=0.9) ax2.set_ylabel(“Absolute Token ID Distance”, fontsize=10) ax2.set_title(“How Far Apart Are the Token IDs?”, fontsize=12, fontweight=”bold”, pad=12) ax2.set_xticklabels(words, rotation=30, ha=”right”, fontsize=9) ax2.spines[[“top”, “right”]].set_visible(False) ax2.grid(axis=”y”, alpha=0.3) for bar, d in zip(bars3, delta): ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 10, str(d), ha=”center”, va=”bottom”, fontsize=9, fontweight=”bold”) high = mpatches.Patch(color=”#E05C3B”, alpha=0.9, label=”> 500 apart”) med = mpatches.Patch(color=”#F0A070″, alpha=0.9, label=”100-500 apart”) low = mpatches.Patch(color=”#A8C4F0″, alpha=0.9, label=”< 100 apart”) ax2.legend(handles=[high, med, low], fontsize=8) plt.tight_layout(pad=2) plt.suptitle(“Tokenization Artifacts: One Space, Completely Different Token”, fontsize=14, fontweight=”bold”, y=1.02) plt.savefig(“tokenization_artifact.png”, dpi=150, bbox_inches=”tight”, facecolor=”#FAFAF8″) plt.show() Simulating Accuracy Drop Demo We start with a standard SFT prompt format and create a few variations by making small changes—like removing newlines, tweaking punctuation, or rewording the instruction. We then measure how similar each version is using token overlap. The key insight is that even small formatting changes can significantly alter the token sequence. For example, removing newlines drops similarity to around 80%, showing that these aren’t just cosmetic—they’re signals the model relies on. The biggest impact comes from rewording the instruction, which cuts overlap to nearly 50%, meaning the prompt no longer resembles what the model was trained on, increasing the risk of unpredictable behavior. Copy CodeCopiedUse a different Browser def tokenize_prompt(text): return tokenizer.encode(text, add_special_tokens=False) # The canonical SFT (fine-tuning) template — what the model was trained on sft_template = ( “Below is a customer review. Classify the sentiment.nn” “Review: {review}nn” “Sentiment:” ) # Prompt variants — small changes, big token consequences variants = { “✓ SFT template (optimal)”: “Below is a customer review. Classify the sentiment.nnReview: {review}nnSentiment:”, “✗ Removed newlines”: “Below is a customer review. Classify the
What is Tokenization Drift and How to Fix It? Leer entrada »




