{"id":42510,"date":"2025-10-06T06:53:30","date_gmt":"2025-10-06T06:53:30","guid":{"rendered":"https:\/\/youzum.net\/streamtensor-a-pytorch-to-accelerator-compiler-that-streams-llm-intermediates-across-fpga-dataflows\/"},"modified":"2025-10-06T06:53:30","modified_gmt":"2025-10-06T06:53:30","slug":"streamtensor-a-pytorch-to-accelerator-compiler-that-streams-llm-intermediates-across-fpga-dataflows","status":"publish","type":"post","link":"https:\/\/youzum.net\/th\/streamtensor-a-pytorch-to-accelerator-compiler-that-streams-llm-intermediates-across-fpga-dataflows\/","title":{"rendered":"StreamTensor: A PyTorch-to-Accelerator Compiler that Streams LLM Intermediates Across FPGA Dataflows"},"content":{"rendered":"

Why treat LLM inference as batched kernels to DRAM when a dataflow compiler can pipe tiles through on-chip FIFOs and stream converters?StreamTensor <\/a>is a compiler that lowers PyTorch LLM graphs (GPT-2, Llama, Qwen, Gemma) into stream-scheduled dataflow accelerators on AMD\u2019s Alveo U55C FPGA. The system introduces an iterative tensor<\/em> (\u201citensor\u201d) type to encode tile\/order of streams, enabling provably correct inter-kernel streaming and automated insertion\/sizing of DMA engines, FIFOs, and layout converters. On LLM decoding workloads, the research team reports up to 0.64\u00d7 lower latency vs. GPUs<\/strong> and up to 1.99\u00d7 higher energy efficiency<\/strong>.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2509.13694<\/figcaption><\/figure>\n<\/div>\n

What StreamTensor does<\/strong>?<\/h3>\n

StreamTensor compiles PyTorch graphs into a stream-oriented dataflow design so that intermediate tiles are largely avoids off-chip DRAM round-trips via on-chip streaming and fusion; DMAs are inserted only when required<\/strong>; they are forwarded through on-chip FIFOs to downstream kernels. The compiler\u2019s central abstraction\u2014iterative tensors (itensors)<\/strong>\u2014records iteration order, tiling, and layout, which makes inter-kernel stream compatibility explicit and drives converter generation only where needed. The framework also searches hierarchically over tiling, fusion, and resource allocation, and uses a linear program<\/strong> to size FIFOs to avoid stalls or deadlock while minimizing on-chip memory.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2509.13694<\/figcaption><\/figure>\n<\/div>\n

What\u2019s actually new<\/strong>?<\/h3>\n