{"id":108383,"date":"2026-08-01T19:59:41","date_gmt":"2026-08-01T19:59:41","guid":{"rendered":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/"},"modified":"2026-08-01T19:59:41","modified_gmt":"2026-08-01T19:59:41","slug":"accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking","status":"publish","type":"post","link":"https:\/\/youzum.net\/it\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/","title":{"rendered":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking"},"content":{"rendered":"

In this tutorial, we explore how NVIDIA Transformer Engine<\/strong><\/a> <\/strong>accelerates transformer workloads by combining fused GPU kernels, BF16 computation, and hardware-aware FP8 execution. We begin by installing Transformer Engine and detecting the active GPU architecture so that we can determine whether the runtime supports TE kernels, FP8 tensor cores, or only the pure-PyTorch fallback path. We then examine core fused components such as te.Linear, te.LayerNorm, te.LayerNormLinear, te.LayerNormMLP, and te.TransformerLayer, while also configuring a delayed-scaling FP8 recipe that manages tensor scaling, amax history, and hybrid E4M3\/E5M2 formats. Using these components, we construct a compact GPT-style causal language model, train it on deterministic synthetic sequences, compare higher-precision and FP8 execution, measure runtime and peak GPU memory, inspect FP8 metadata, and validate the trained model through autoregressive generation.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import subprocess, sys, os\ndef pip_install(*pkgs):\n   subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n                   \"--no-build-isolation\", *pkgs], check=False)\nprint(\">> Installing transformer_engine[pytorch] (this can take a few minutes)...\")\npip_install(\"transformer_engine[pytorch]\")\nimport time, math, gc\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nassert torch.cuda.is_available(), \"Enable a GPU runtime in Colab first!\"\nDEVICE = \"cuda\"\nprops = torch.cuda.get_device_properties(0)\nCC = (props.major, props.minor)\nGPU_NAME = props.name\nprint(f\">> GPU: {GPU_NAME} | compute capability {CC[0]}.{CC[1]} | \"\n     f\"{props.total_memory\/1e9:.1f} GB\")\nTE_CAPABLE  = CC >= (8, 0)\nFP8_CAPABLE = CC >= (8, 9)\nte = None\nif TE_CAPABLE:\n   try:\n       import transformer_engine.pytorch as te\n       from transformer_engine.common import recipe\n       print(\">> Transformer Engine imported OK:\",\n             getattr(te, \"__version__\", \"unknown version\"))\n   except Exception as e:\n       print(f\">> TE import failed ({e}); using pure-PyTorch fallback.\")\n       TE_CAPABLE = FP8_CAPABLE = False\nelse:\n   print(\">> GPU is pre-Ampere (e.g. T4): TE kernels unsupported -> fallback mode.\")\nif TE_CAPABLE and FP8_CAPABLE and te is not None:\n   try:\n       ok, reason = te.fp8.check_fp8_support()\n       FP8_CAPABLE = bool(ok)\n       if not ok:\n           print(\">> TE reports FP8 unsupported:\", reason)\n   except Exception:\n       pass\nprint(f\">> Mode: TE={'ON' if TE_CAPABLE else 'OFF'} | \"\n     f\"FP8={'ON' if FP8_CAPABLE else 'OFF (will use BF16)'}\")\ntorch.manual_seed(1234)\nif TE_CAPABLE:\n   H = 768\n   x_demo = torch.randn(8, 32, H, device=DEVICE, dtype=torch.bfloat16)\n   lin      = te.Linear(H, H, bias=True, params_dtype=torch.bfloat16).to(DEVICE)\n   ln       = te.LayerNorm(H, params_dtype=torch.bfloat16).to(DEVICE)\n   ln_lin   = te.LayerNormLinear(H, 3 * H, params_dtype=torch.bfloat16).to(DEVICE)\n   ln_mlp   = te.LayerNormMLP(H, 4 * H, params_dtype=torch.bfloat16).to(DEVICE)\n   with torch.no_grad():\n       print(\"n>> Module tour (shapes):\")\n       print(\"   te.Linear         \", tuple(lin(x_demo).shape))\n       print(\"   te.LayerNorm      \", tuple(ln(x_demo).shape))\n       print(\"   te.LayerNormLinear\", tuple(ln_lin(x_demo).shape))\n       print(\"   te.LayerNormMLP   \", tuple(ln_mlp(x_demo).shape))\n   del lin, ln, ln_lin, ln_mlp, x_demo\n   gc.collect(); torch.cuda.empty_cache()\nfp8_recipe = None\nif FP8_CAPABLE:\n   fp8_recipe = recipe.DelayedScaling(\n       fp8_format=recipe.Format.HYBRID,\n       amax_history_len=16,\n       amax_compute_algo=\"max\",\n   )\n   print(\"n>> FP8 recipe:\", fp8_recipe)\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We install NVIDIA Transformer Engine and initialize the PyTorch environment required for GPU-accelerated execution. We inspect the active GPU, compute capability, and memory capacity to determine whether fused TE kernels and FP8 tensor cores are available. We also validate the core fused modules and configure a delayed-scaling FP8 recipe while preserving an automatic PyTorch fallback for unsupported hardware.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
VOCAB, D_MODEL, N_HEADS, N_LAYERS, FFN, SEQ = 96, 768, 12, 4, 3072, 256\nclass MiniGPT_TE(nn.Module):\n   \"\"\"Causal LM where every block is a single fused te.TransformerLayer.\"\"\"\n   def __init__(self):\n       super().__init__()\n       self.emb = nn.Embedding(VOCAB, D_MODEL)\n       self.pos = nn.Embedding(SEQ, D_MODEL)\n       self.blocks = nn.ModuleList([\n           te.TransformerLayer(\n               hidden_size=D_MODEL,\n               ffn_hidden_size=FFN,\n               num_attention_heads=N_HEADS,\n               self_attn_mask_type=\"causal\",\n               layer_number=i + 1,\n               params_dtype=torch.bfloat16,\n               hidden_dropout=0.0,\n               attention_dropout=0.0,\n           )\n           for i in range(N_LAYERS)\n       ])\n       self.ln_f = nn.LayerNorm(D_MODEL)\n       self.head = nn.Linear(D_MODEL, VOCAB, bias=False)\n   def forward(self, idx):\n       B, T = idx.shape\n       h = self.emb(idx) + self.pos(torch.arange(T, device=idx.device))\n       h = h.to(torch.bfloat16)\n       for blk in self.blocks:\n           h = blk(h)\n       h = self.ln_f(h.float())\n       return self.head(h)\nclass Block_PT(nn.Module):\n   \"\"\"Plain-PyTorch transformer block, mirrors te.TransformerLayer.\"\"\"\n   def __init__(self):\n       super().__init__()\n       self.ln1 = nn.LayerNorm(D_MODEL)\n       self.attn = nn.MultiheadAttention(D_MODEL, N_HEADS, batch_first=True)\n       self.ln2 = nn.LayerNorm(D_MODEL)\n       self.mlp = nn.Sequential(nn.Linear(D_MODEL, FFN), nn.GELU(),\n                                nn.Linear(FFN, D_MODEL))\n   def forward(self, x, mask):\n       a, _ = self.attn(self.ln1(x), self.ln1(x), self.ln1(x),\n                        attn_mask=mask, need_weights=False)\n       x = x + a\n       return x + self.mlp(self.ln2(x))\nclass MiniGPT_PT(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.emb = nn.Embedding(VOCAB, D_MODEL)\n       self.pos = nn.Embedding(SEQ, D_MODEL)\n       self.blocks = nn.ModuleList([Block_PT() for _ in range(N_LAYERS)])\n       self.ln_f = nn.LayerNorm(D_MODEL)\n       self.head = nn.Linear(D_MODEL, VOCAB, bias=False)\n   def forward(self, idx):\n       B, T = idx.shape\n       mask = torch.triu(torch.full((T, T), float(\"-inf\"),\n                                    device=idx.device), diagonal=1)\n       h = self.emb(idx) + self.pos(torch.arange(T, device=idx.device))\n       for blk in self.blocks:\n           h = blk(h, mask)\n       return self.head(self.ln_f(h))\nmodel = (MiniGPT_TE() if TE_CAPABLE else MiniGPT_PT()).to(DEVICE)\nn_params = sum(p.numel() for p in model.parameters())\nprint(f\"n>> Model: {'TE fused' if TE_CAPABLE else 'pure PyTorch'} | \"\n     f\"{n_params\/1e6:.1f}M params | {N_LAYERS} layers x {D_MODEL}d\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We define a compact causal language model using fused te.TransformerLayer blocks for Transformer Engine execution. We also implement an equivalent pure-PyTorch transformer architecture with multi-head attention, layer normalization, residual connections, and feed-forward networks. We select the appropriate model dynamically according to GPU support and report the final parameter count and architectural dimensions.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def make_batch(bsz=16):\n   phase  = torch.randint(0, VOCAB, (bsz, 1))\n   stride = torch.randint(1, 7, (bsz, 1))\n   steps  = torch.arange(SEQ + 1).unsqueeze(0)\n   seq = (phase + stride * steps) % VOCAB\n   return seq[:, :-1].to(DEVICE), seq[:, 1:].to(DEVICE)\nopt = torch.optim.AdamW(model.parameters(), lr=3e-4)\ndef run_step(x, y, use_fp8):\n   if TE_CAPABLE and use_fp8:\n       with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):\n           logits = model(x)\n   else:\n       logits = model(x)\n   loss = F.cross_entropy(logits.float().reshape(-1, VOCAB), y.reshape(-1))\n   opt.zero_grad(set_to_none=True)\n   loss.backward()\n   opt.step()\n   return loss.item()\nprint(f\"n>> Training 60 steps ({'FP8' if FP8_CAPABLE else 'BF16\/FP32'})...\")\nt0 = time.time()\nfor step in range(1, 61):\n   x, y = make_batch()\n   loss = run_step(x, y, use_fp8=FP8_CAPABLE)\n   if step % 10 == 0:\n       print(f\"   step {step:3d} | loss {loss:.4f} | \"\n             f\"{(time.time()-t0)\/step*1000:.0f} ms\/step\")\nprint(f\">> Final loss: {loss:.4f} (random guess would be ~{math.log(VOCAB):.2f})\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We create deterministic arithmetic-pattern sequences that allow the model to learn predictable token transitions across the vocabulary. We configure the AdamW optimizer and implement a training step that conditionally wraps the forward pass in te.fp8_autocast when FP8 execution is supported. We train the model for multiple iterations, monitor the loss and step latency, and compare the final loss against the random-guess baseline.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def bench(use_fp8, iters=30, warmup=10):\n   x, y = make_batch(bsz=32)\n   for _ in range(warmup):\n       run_step(x, y, use_fp8)\n   torch.cuda.synchronize()\n   torch.cuda.reset_peak_memory_stats()\n   t = time.time()\n   for _ in range(iters):\n       run_step(x, y, use_fp8)\n   torch.cuda.synchronize()\n   ms = (time.time() - t) \/ iters * 1000\n   mem = torch.cuda.max_memory_allocated() \/ 1e9\n   return ms, mem\nprint(\"n>> Benchmark (batch 32, seq 256, fwd+bwd+optim):\")\nms_hi, mem_hi = bench(use_fp8=False)\nprint(f\"   {'BF16' if TE_CAPABLE else 'FP32'}: {ms_hi:7.1f} ms\/step | \"\n     f\"peak mem {mem_hi:.2f} GB\")\nif FP8_CAPABLE:\n   ms_f8, mem_f8 = bench(use_fp8=True)\n   print(f\"   FP8 : {ms_f8:7.1f} ms\/step | peak mem {mem_f8:.2f} GB\")\n   print(f\"   Speedup: {ms_hi\/ms_f8:.2f}x  \"\n         f\"(gains grow with model size \u2014 try D_MODEL=2048, N_LAYERS=12)\")\nelse:\n   print(\"   FP8 benchmark skipped \u2014 needs an sm_89+ GPU (L4\/H100\/Ada\/Blackwell).\")\nif FP8_CAPABLE:\n   blk = model.blocks[0]\n   for name, m in blk.named_modules():\n       meta = getattr(m, \"fp8_meta\", None)\n       if meta and \"scaling_fwd\" in meta:\n           s = meta[\"scaling_fwd\"]\n           print(f\"n>> FP8 state of block-0 submodule '{name}':\")\n           print(\"   scale       :\", s.scale.flatten()[:4].tolist())\n           print(\"   amax_history:\", s.amax_history[0, :4].tolist())\n           break\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We benchmark forward propagation, backpropagation, and optimizer updates using higher-precision and FP8 execution modes. We measure average training-step latency and peak allocated GPU memory to quantify the performance and memory impact of reduced-precision computation. We also inspect the scaling factors and amax history maintained by Transformer Engine to understand how delayed scaling stabilizes FP8 tensors.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
@torch.no_grad()\ndef generate(prompt_len=8, gen_len=24):\n   x, _ = make_batch(bsz=1)\n   ctx = x[:, :prompt_len]\n   for _ in range(gen_len):\n       inp = ctx[:, -SEQ:]\n       if TE_CAPABLE and FP8_CAPABLE:\n           with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):\n               logits = model(inp)\n       else:\n           logits = model(inp)\n       nxt = logits[:, -1].argmax(-1, keepdim=True)\n       ctx = torch.cat([ctx, nxt], dim=1)\n   return ctx[0].tolist()\nseq = generate()\nprint(\"n>> Greedy generation (should continue the arithmetic pattern):\")\nprint(\"   prompt+gen:\", seq)\ndiffs = [(b - a) % VOCAB for a, b in zip(seq, seq[1:])]\nprint(\"   step diffs:\", diffs, \"<- constant stride = model learned the rule\")\nprint(\"n>> Done! Things to try next:\")\nprint(\"   * Scale up: D_MODEL=2048, N_LAYERS=12 -> FP8 speedup becomes dramatic\")\nprint(\"   * recipe.Format.E4M3 vs HYBRID; amax_history_len=1024\")\nprint(\"   * te.LayerNormMLP \/ te.LayerNormLinear in your own architectures\")\nprint(\"   * fp8_model_init() to store weights themselves in FP8 for inference\")\n<\/code><\/pre>\n<\/div>\n<\/div>\n
We implement greedy autoregressive generation by repeatedly feeding the latest context into the trained causal language model. We compare consecutive generated tokens to verify whether the model preserves the constant arithmetic stride present in the synthetic training data. We conclude by identifying practical extensions, including larger model dimensions, alternative FP8 formats, longer amax histories, fused modules, and FP8 weight initialization.<\/p>\n
In conclusion, we demonstrated how we integrate NVIDIA Transformer Engine into an end-to-end transformer training workflow while preserving compatibility across different Colab GPU environments. We used fused transformer modules to reduce kernel-launch overhead and memory traffic, applied FP8 autocasting with delayed scaling when supported, and retained BF16 or FP32 execution through an automatic PyTorch fallback. By training and benchmarking the same mini causal language model, we observed how hardware capability, numerical format, fused execution, and model scale influence training speed and memory consumption. We also inspected the internal scaling factors and amax history that support stable FP8 computation, which gives us a clearer understanding of how Transformer Engine manages reduced-precision arithmetic.<\/p>\n
\n
\n<\/p>
\n<\/p>
Check out the\u00a0Full Codes<\/a>.\u00a0<\/strong>Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0150k+ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
<\/a><\/p>\n
The post Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we explore how NVIDIA Transformer Engine accelerates transformer workloads by combining fused GPU kernels, BF16 computation, and hardware-aware FP8 execution. We begin by installing Transformer Engine and detecting the active GPU architecture so that we can determine whether the runtime supports TE kernels, FP8 tensor cores, or only the pure-PyTorch fallback path. We then examine core fused components such as te.Linear, te.LayerNorm, te.LayerNormLinear, te.LayerNormMLP, and te.TransformerLayer, while also configuring a delayed-scaling FP8 recipe that manages tensor scaling, amax history, and hybrid E4M3\/E5M2 formats. Using these components, we construct a compact GPT-style causal language model, train it on deterministic synthetic sequences, compare higher-precision and FP8 execution, measure runtime and peak GPU memory, inspect FP8 metadata, and validate the trained model through autoregressive generation. Copy CodeCopiedUse a different Browser import subprocess, sys, os def pip_install(*pkgs): subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, “–no-build-isolation”, *pkgs], check=False) print(“>> Installing transformer_engine[pytorch] (this can take a few minutes)…”) pip_install(“transformer_engine[pytorch]”) import time, math, gc import torch import torch.nn as nn import torch.nn.functional as F assert torch.cuda.is_available(), “Enable a GPU runtime in Colab first!” DEVICE = “cuda” props = torch.cuda.get_device_properties(0) CC = (props.major, props.minor) GPU_NAME = props.name print(f”>> GPU: {GPU_NAME} | compute capability {CC[0]}.{CC[1]} | ” f”{props.total_memory\/1e9:.1f} GB”) TE_CAPABLE = CC >= (8, 0) FP8_CAPABLE = CC >= (8, 9) te = None if TE_CAPABLE: try: import transformer_engine.pytorch as te from transformer_engine.common import recipe print(“>> Transformer Engine imported OK:”, getattr(te, “__version__”, “unknown version”)) except Exception as e: print(f”>> TE import failed ({e}); using pure-PyTorch fallback.”) TE_CAPABLE = FP8_CAPABLE = False else: print(“>> GPU is pre-Ampere (e.g. T4): TE kernels unsupported -> fallback mode.”) if TE_CAPABLE and FP8_CAPABLE and te is not None: try: ok, reason = te.fp8.check_fp8_support() FP8_CAPABLE = bool(ok) if not ok: print(“>> TE reports FP8 unsupported:”, reason) except Exception: pass print(f”>> Mode: TE={‘ON’ if TE_CAPABLE else ‘OFF’} | ” f”FP8={‘ON’ if FP8_CAPABLE else ‘OFF (will use BF16)’}”) torch.manual_seed(1234) if TE_CAPABLE: H = 768 x_demo = torch.randn(8, 32, H, device=DEVICE, dtype=torch.bfloat16) lin = te.Linear(H, H, bias=True, params_dtype=torch.bfloat16).to(DEVICE) ln = te.LayerNorm(H, params_dtype=torch.bfloat16).to(DEVICE) ln_lin = te.LayerNormLinear(H, 3 * H, params_dtype=torch.bfloat16).to(DEVICE) ln_mlp = te.LayerNormMLP(H, 4 * H, params_dtype=torch.bfloat16).to(DEVICE) with torch.no_grad(): print(“n>> Module tour (shapes):”) print(” te.Linear “, tuple(lin(x_demo).shape)) print(” te.LayerNorm “, tuple(ln(x_demo).shape)) print(” te.LayerNormLinear”, tuple(ln_lin(x_demo).shape)) print(” te.LayerNormMLP “, tuple(ln_mlp(x_demo).shape)) del lin, ln, ln_lin, ln_mlp, x_demo gc.collect(); torch.cuda.empty_cache() fp8_recipe = None if FP8_CAPABLE: fp8_recipe = recipe.DelayedScaling( fp8_format=recipe.Format.HYBRID, amax_history_len=16, amax_compute_algo=”max”, ) print(“n>> FP8 recipe:”, fp8_recipe) We install NVIDIA Transformer Engine and initialize the PyTorch environment required for GPU-accelerated execution. We inspect the active GPU, compute capability, and memory capacity to determine whether fused TE kernels and FP8 tensor cores are available. We also validate the core fused modules and configure a delayed-scaling FP8 recipe while preserving an automatic PyTorch fallback for unsupported hardware. Copy CodeCopiedUse a different Browser VOCAB, D_MODEL, N_HEADS, N_LAYERS, FFN, SEQ = 96, 768, 12, 4, 3072, 256 class MiniGPT_TE(nn.Module): “””Causal LM where every block is a single fused te.TransformerLayer.””” def __init__(self): super().__init__() self.emb = nn.Embedding(VOCAB, D_MODEL) self.pos = nn.Embedding(SEQ, D_MODEL) self.blocks = nn.ModuleList([ te.TransformerLayer( hidden_size=D_MODEL, ffn_hidden_size=FFN, num_attention_heads=N_HEADS, self_attn_mask_type=”causal”, layer_number=i + 1, params_dtype=torch.bfloat16, hidden_dropout=0.0, attention_dropout=0.0, ) for i in range(N_LAYERS) ]) self.ln_f = nn.LayerNorm(D_MODEL) self.head = nn.Linear(D_MODEL, VOCAB, bias=False) def forward(self, idx): B, T = idx.shape h = self.emb(idx) + self.pos(torch.arange(T, device=idx.device)) h = h.to(torch.bfloat16) for blk in self.blocks: h = blk(h) h = self.ln_f(h.float()) return self.head(h) class Block_PT(nn.Module): “””Plain-PyTorch transformer block, mirrors te.TransformerLayer.””” def __init__(self): super().__init__() self.ln1 = nn.LayerNorm(D_MODEL) self.attn = nn.MultiheadAttention(D_MODEL, N_HEADS, batch_first=True) self.ln2 = nn.LayerNorm(D_MODEL) self.mlp = nn.Sequential(nn.Linear(D_MODEL, FFN), nn.GELU(), nn.Linear(FFN, D_MODEL)) def forward(self, x, mask): a, _ = self.attn(self.ln1(x), self.ln1(x), self.ln1(x), attn_mask=mask, need_weights=False) x = x + a return x + self.mlp(self.ln2(x)) class MiniGPT_PT(nn.Module): def __init__(self): super().__init__() self.emb = nn.Embedding(VOCAB, D_MODEL) self.pos = nn.Embedding(SEQ, D_MODEL) self.blocks = nn.ModuleList([Block_PT() for _ in range(N_LAYERS)]) self.ln_f = nn.LayerNorm(D_MODEL) self.head = nn.Linear(D_MODEL, VOCAB, bias=False) def forward(self, idx): B, T = idx.shape mask = torch.triu(torch.full((T, T), float(“-inf”), device=idx.device), diagonal=1) h = self.emb(idx) + self.pos(torch.arange(T, device=idx.device)) for blk in self.blocks: h = blk(h, mask) return self.head(self.ln_f(h)) model = (MiniGPT_TE() if TE_CAPABLE else MiniGPT_PT()).to(DEVICE) n_params = sum(p.numel() for p in model.parameters()) print(f”n>> Model: {‘TE fused’ if TE_CAPABLE else ‘pure PyTorch’} | ” f”{n_params\/1e6:.1f}M params | {N_LAYERS} layers x {D_MODEL}d”) We define a compact causal language model using fused te.TransformerLayer blocks for Transformer Engine execution. We also implement an equivalent pure-PyTorch transformer architecture with multi-head attention, layer normalization, residual connections, and feed-forward networks. We select the appropriate model dynamically according to GPU support and report the final parameter count and architectural dimensions. Copy CodeCopiedUse a different Browser def make_batch(bsz=16): phase = torch.randint(0, VOCAB, (bsz, 1)) stride = torch.randint(1, 7, (bsz, 1)) steps = torch.arange(SEQ + 1).unsqueeze(0) seq = (phase + stride * steps) % VOCAB return seq[:, :-1].to(DEVICE), seq[:, 1:].to(DEVICE) opt = torch.optim.AdamW(model.parameters(), lr=3e-4) def run_step(x, y, use_fp8): if TE_CAPABLE and use_fp8: with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe): logits = model(x) else: logits = model(x) loss = F.cross_entropy(logits.float().reshape(-1, VOCAB), y.reshape(-1)) opt.zero_grad(set_to_none=True) loss.backward() opt.step() return loss.item() print(f”n>> Training 60 steps ({‘FP8’ if FP8_CAPABLE else ‘BF16\/FP32’})…”) t0 = time.time() for step in range(1, 61): x, y = make_batch() loss = run_step(x, y, use_fp8=FP8_CAPABLE) if step % 10 == 0: print(f” step {step:3d} | loss {loss:.4f} | ” f”{(time.time()-t0)\/step*1000:.0f} ms\/step”) print(f”>> Final loss: {loss:.4f} (random guess would be ~{math.log(VOCAB):.2f})”) We create deterministic arithmetic-pattern sequences that allow the model to learn predictable token transitions across the vocabulary. We configure the AdamW optimizer and implement a training step that conditionally wraps the forward pass in te.fp8_autocast when FP8 execution is supported. We train the model for multiple iterations, monitor the loss and step latency, and compare the final loss against the random-guess baseline. Copy CodeCopiedUse a different Browser def bench(use_fp8, iters=30, warmup=10): x, y = make_batch(bsz=32) for _ in range(warmup): run_step(x, y, use_fp8) torch.cuda.synchronize() torch.cuda.reset_peak_memory_stats() t = time.time() for _ in range(iters): run_step(x, y, use_fp8) torch.cuda.synchronize() ms = (time.time() – t) \/ iters * 1000 mem = torch.cuda.max_memory_allocated() \/ 1e9 return ms, mem print(“n>> Benchmark (batch 32, seq 256, fwd+bwd+optim):”) ms_hi, mem_hi = bench(use_fp8=False) print(f” {‘BF16’ if TE_CAPABLE else ‘FP32’}: {ms_hi:7.1f} ms\/step | ” f”peak mem {mem_hi:.2f} GB”) if<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"pmpro_default_level":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_pvb_checkbox_block_on_post":false,"footnotes":""},"categories":[52,5,7,1],"tags":[],"class_list":["post-108383","post","type-post","status-publish","format-standard","hentry","category-ai-club","category-committee","category-news","category-uncategorized","pmpro-has-access"],"acf":[],"yoast_head":"\nAccelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum<\/title>\n<meta name=\"description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/youzum.net\/it\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\" \/>\n<meta property=\"og:locale\" content=\"it_IT\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum\" \/>\n<meta property=\"og:description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta property=\"og:url\" content=\"https:\/\/youzum.net\/it\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\" \/>\n<meta property=\"og:site_name\" content=\"YouZum\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DroneAssociationTH\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-01T19:59:41+00:00\" \/>\n<meta name=\"author\" content=\"admin NU\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Scritto da\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin NU\" \/>\n\t<meta name=\"twitter:label2\" content=\"Tempo di lettura stimato\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minuti\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\"},\"author\":{\"name\":\"admin NU\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\"},\"headline\":\"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking\",\"datePublished\":\"2026-08-01T19:59:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\"},\"wordCount\":668,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"articleSection\":[\"AI\",\"Committee\",\"News\",\"Uncategorized\"],\"inLanguage\":\"it-IT\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\",\"url\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\",\"name\":\"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum\",\"isPartOf\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#website\"},\"datePublished\":\"2026-08-01T19:59:41+00:00\",\"description\":\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\",\"breadcrumb\":{\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#breadcrumb\"},\"inLanguage\":\"it-IT\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/youzum.net\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/yousum.gpucore.co\/#website\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"name\":\"YouSum\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/yousum.gpucore.co\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"it-IT\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\",\"name\":\"Drone Association Thailand\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"it-IT\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"width\":300,\"height\":300,\"caption\":\"Drone Association Thailand\"},\"image\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DroneAssociationTH\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\",\"name\":\"admin NU\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"it-IT\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"caption\":\"admin NU\"},\"url\":\"https:\/\/youzum.net\/it\/members\/adminnu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/youzum.net\/it\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/","og_locale":"it_IT","og_type":"article","og_title":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum","og_description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","og_url":"https:\/\/youzum.net\/it\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/","og_site_name":"YouZum","article_publisher":"https:\/\/www.facebook.com\/DroneAssociationTH\/","article_published_time":"2026-08-01T19:59:41+00:00","author":"admin NU","twitter_card":"summary_large_image","twitter_misc":{"Scritto da":"admin NU","Tempo di lettura stimato":"10 minuti"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#article","isPartOf":{"@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/"},"author":{"name":"admin NU","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c"},"headline":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking","datePublished":"2026-08-01T19:59:41+00:00","mainEntityOfPage":{"@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/"},"wordCount":668,"commentCount":0,"publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"articleSection":["AI","Committee","News","Uncategorized"],"inLanguage":"it-IT","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/","url":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/","name":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking - YouZum","isPartOf":{"@id":"https:\/\/yousum.gpucore.co\/#website"},"datePublished":"2026-08-01T19:59:41+00:00","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","breadcrumb":{"@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#breadcrumb"},"inLanguage":"it-IT","potentialAction":[{"@type":"ReadAction","target":["https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/youzum.net\/accelerating-transformer-training-with-nvidia-transformer-engine-fused-kernels-bf16-fp8-and-gpu-benchmarking\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/youzum.net\/"},{"@type":"ListItem","position":2,"name":"Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking"}]},{"@type":"WebSite","@id":"https:\/\/yousum.gpucore.co\/#website","url":"https:\/\/yousum.gpucore.co\/","name":"YouSum","description":"","publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/yousum.gpucore.co\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"it-IT"},{"@type":"Organization","@id":"https:\/\/yousum.gpucore.co\/#organization","name":"Drone Association Thailand","url":"https:\/\/yousum.gpucore.co\/","logo":{"@type":"ImageObject","inLanguage":"it-IT","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","width":300,"height":300,"caption":"Drone Association Thailand"},"image":{"@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DroneAssociationTH\/"]},{"@type":"Person","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c","name":"admin NU","image":{"@type":"ImageObject","inLanguage":"it-IT","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","caption":"admin NU"},"url":"https:\/\/youzum.net\/it\/members\/adminnu\/"}]}},"rttpg_featured_image_url":null,"rttpg_author":{"display_name":"admin NU","author_link":"https:\/\/youzum.net\/it\/members\/adminnu\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/youzum.net\/it\/category\/ai-club\/\" rel=\"category tag\">AI<\/a> <a href=\"https:\/\/youzum.net\/it\/category\/committee\/\" rel=\"category tag\">Committee<\/a> <a href=\"https:\/\/youzum.net\/it\/category\/news\/\" rel=\"category tag\">News<\/a> <a href=\"https:\/\/youzum.net\/it\/category\/uncategorized\/\" rel=\"category tag\">Uncategorized<\/a>","rttpg_excerpt":"In this tutorial, we explore how NVIDIA Transformer Engine accelerates transformer workloads by combining fused GPU kernels, BF16 computation, and hardware-aware FP8 execution. We begin by installing Transformer Engine and detecting the active GPU architecture so that we can determine whether the runtime supports TE kernels, FP8 tensor cores, or only the pure-PyTorch fallback path.…","_links":{"self":[{"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/posts\/108383","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/comments?post=108383"}],"version-history":[{"count":0,"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/posts\/108383\/revisions"}],"wp:attachment":[{"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/media?parent=108383"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/categories?post=108383"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/youzum.net\/it\/wp-json\/wp\/v2\/tags?post=108383"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}