{"id":103701,"date":"2026-07-12T19:14:11","date_gmt":"2026-07-12T19:14:11","guid":{"rendered":"https:\/\/youzum.net\/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-instrument-music-transcription-to-midi\/"},"modified":"2026-07-12T19:14:11","modified_gmt":"2026-07-12T19:14:11","slug":"kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-instrument-music-transcription-to-midi","status":"publish","type":"post","link":"https:\/\/youzum.net\/es\/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-instrument-music-transcription-to-midi\/","title":{"rendered":"Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI"},"content":{"rendered":"

Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor<\/a><\/strong> to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.<\/p>\n

This article explains how MuScriptor works, what the benchmarks show, and how to run it.<\/strong><\/p>\n

What is MuScriptor?<\/strong><\/h2>\n

At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.<\/p>\n

The release ships three weight variants on Hugging Face. Their sizes are small<\/code> (103M), medium<\/code> (307M, default), and large<\/code> (1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.<\/p>\n

How the Three-Stage Pipeline Works<\/strong><\/h2>\n

MuScriptor\u2019s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.<\/p>\n

    \n
  1. Pre-training uses D<sub>Synth<\/sub><\/strong>, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.<\/li>\n
  2. Fine-tuning uses D<sub>Real<\/sub><\/strong>, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor.<\/li>\n
  3. Reinforcement learning post-training uses D<sub>RL<\/sub><\/strong>, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.<\/li>\n<\/ol>\n


    \n Transcribe<\/button>
    \n <\/div>
    \n <\/div><\/p>\n

    <div class="”rollwrap”">
    \n <svg class=”roll” id=”roll” viewBox=”0 0 720 300″ preserveAspectRatio=”xMidYMid meet” role=”img” aria-label=”Piano roll transcription”><\/svg>
    \n <\/div>
    \n <div class="”legend”">
    \n <span><i style="”background:#2563EB”"><\/i>True positive (detected)<\/span>
    \n <span><i style="”background:#16A34A”"><\/i>False negative (missed)<\/span>
    \n <span><i style="”background:#DC2626″"><\/i>False positive (wrong)<\/span>
    \n <\/div><\/p>\n

    <div class="”grid”">
    \n <div class="”metrics”">
    \n <div class="”metric”">
    \n <small>Onset F1<\/small>
    \n <div class="”val”" id="”mOnset”">—<\/div>
    \n <div class="”baseline”" id="”bOnset”"><\/div>
    \n <div class="”metricbar”"><i id="”barOnset”"><\/i><\/div>
    \n <\/div>
    \n <div class="”metric”">
    \n <small>Multi F1<\/small>
    \n <div class="”val”" id="”mMulti”">—<\/div>
    \n <div class="”baseline”" id="”bMulti”"><\/div>
    \n <div class="”metricbar”"><i id="”barMulti”"><\/i><\/div>
    \n <\/div>
    \n <\/div>
    \n <div class="”stream”" id="”stream”">
    \n <div class="”h”">Event stream · model.transcribe()<\/div>
    \n <div class="”ln”" style="”opacity:.6″">Press Transcribe to stream note events…<\/div>
    \n <\/div>
    \n <\/div><\/p>\n

    <p class="”note”" id="”stageNote”"><\/p>
    \n <\/div><\/p>\n

    <div class="”foot”">
    \n <span>Note pattern is illustrative. F1 scores are real, from the MuScriptor paper (1.3B model on D<sub>Test<\/sub>).<\/span>
    \n <span>Built by <b>Marktechpost<\/b><\/span>
    \n <\/div>
    \n<\/div><\/p>\n

    <script>
    \n(function(){
    \n var SVG=”http:\/\/www.w3.org\/2000\/svg”;
    \n var roll=document.getElementById(“roll”);
    \n var W=720,H=300,PADL=64,PADR=14,PADT=14,PADB=26;
    \n var DUR=10; \/\/ seconds shown
    \n var rows=[
    \n {name:”Drums”,inst:”drums”,y:0},
    \n {name:”E-Bass”,inst:”e_bass”,y:1},
    \n {name:”Dist. E-Gtr”,inst:”distorted_e_guitar”,y:2},
    \n {name:”Ac. Piano”,inst:”acoustic_piano”,y:3},
    \n {name:”Voice”,inst:”voice”,y:4},
    \n {name:”Strings”,inst:”string_ensemble”,y:5}
    \n ];
    \n var NR=rows.length;
    \n var rowH=(H-PADT-PADB)\/NR;
    \n function tx(t){return PADL+(t\/DUR)*(W-PADL-PADR);}
    \n function ry(r){return PADT+r*rowH;}<\/p>\n

    \/\/ Illustrative ground-truth notes: [row, start, dur, rank(0=easy..1=hard)]
    \n var GT=[
    \n [0,0.3,0.18,0.1],[0,0.8,0.18,0.15],[0,1.3,0.18,0.2],[0,1.8,0.18,0.1],[0,2.4,0.18,0.25],[0,3.0,0.18,0.2],[0,3.6,0.18,0.3],[0,4.2,0.18,0.2],[0,4.9,0.18,0.4],[0,5.5,0.18,0.3],[0,6.2,0.18,0.5],[0,6.9,0.18,0.4],[0,7.6,0.18,0.6],[0,8.3,0.18,0.5],[0,9.0,0.18,0.7],
    \n [1,0.4,0.8,0.2],[1,1.6,0.8,0.25],[1,3.0,0.9,0.35],[1,4.4,0.8,0.4],[1,5.8,0.9,0.5],[1,7.2,0.9,0.6],[1,8.6,0.9,0.7],
    \n [2,1.0,1.1,0.45],[2,2.6,1.0,0.55],[2,4.2,1.2,0.6],[2,6.0,1.1,0.7],[2,7.8,1.1,0.8],
    \n [3,0.6,0.7,0.3],[3,1.7,0.6,0.35],[3,2.9,0.7,0.4],[3,4.1,0.6,0.5],[3,5.3,0.7,0.55],[3,6.6,0.6,0.65],[3,8.0,0.8,0.75],
    \n [4,2.0,1.4,0.5],[4,4.0,1.5,0.6],[4,6.3,1.4,0.72],[4,8.4,1.2,0.85],
    \n [5,1.2,1.8,0.55],[5,3.6,1.9,0.68],[5,6.0,1.9,0.8],[5,8.2,1.6,0.9]
    \n ];
    \n \/\/ Illustrative false positives available: [row,start,dur,rank]
    \n var FPS=[
    \n [0,2.1,0.16,0.9],[0,5.9,0.16,0.7],[0,7.9,0.16,0.85],
    \n [1,2.4,0.6,0.8],[1,6.6,0.7,0.9],
    \n [2,3.7,0.9,0.85],[2,8.9,0.9,0.95],
    \n [3,3.5,0.5,0.9],[3,7.3,0.6,0.8],
    \n [4,3.5,0.9,0.9],[4,7.7,0.8,0.95],
    \n [5,5.1,1.2,0.88],[5,9.2,0.9,0.97]
    \n ];<\/p>\n

    \/\/ Real numbers from the paper (1.3B, DTest). recall\/fpRate are illustrative thresholds.
    \n var STAGES={
    \n synth:{onset:34.5,multi:16.2,onBase:26.1,recall:0.42,fp:0.85,
    \n note:”Trained only on synthetic MIDI. It finds coarse pitch activity, but misses many onsets and mislabels instruments.”},
    \n real:{onset:54.4,multi:41.6,onBase:52.5,recall:0.78,fp:0.35,
    \n note:”Fine-tuning on 170k real recordings lifts every metric by roughly 20 points over synthetic-only training.”},
    \n rl:{onset:60.4,multi:48.2,onBase:60.4,recall:0.90,fp:0.12,
    \n note:”GRPO-style RL post-training on 300 verified tracks reduces false negatives and sharpens onset timing.”}
    \n };
    \n var current=”rl”, cond=true, playing=false, playhead=0, raf=null, streamed={};<\/p>\n

    function selectedRows(){
    \n if(!cond) return rows.map(function(r){return r.y;});
    \n \/\/ conditioning example: focus on the core rhythm section + piano
    \n return [0,1,2,3];
    \n }<\/p>\n

    function classify(stage){
    \n var s=STAGES[stage];
    \n var tp=[],fn=[],fp=[];
    \n var sel=selectedRows();
    \n GT.forEach(function(n){
    \n if(cond && sel.indexOf(n[0])===-1) return; \/\/ hidden by conditioning
    \n var boost=cond?0.08:0; \/\/ conditioning nudges recall up (illustrative)
    \n if(n[3] <= s.recall+boost) tp.push(n); else fn.push(n);
    \n });
    \n FPS.forEach(function(n){
    \n if(cond && sel.indexOf(n[0])===-1) return;
    \n var fpr=cond?s.fp*0.7:s.fp; \/\/ conditioning trims spurious notes
    \n if(n[3] > (1-fpr)) fp.push(n);
    \n });
    \n return {tp:tp,fn:fn,fp:fp};
    \n }<\/p>\n

    function el(tag,attrs){var e=document.createElementNS(SVG,tag);for(var k in attrs)e.setAttribute(k,attrs[k]);return e;}<\/p>\n

    function drawGrid(){
    \n while(roll.firstChild) roll.removeChild(roll.firstChild);
    \n var sel=selectedRows();
    \n for(var r=0;r<NR;r++){
    \n var active=sel.indexOf(r)!==-1;
    \n roll.appendChild(el(“rect”,{x:PADL,y:ry(r),width:W-PADL-PADR,height:rowH,
    \n fill:(r%2? “#FAFBFD”:”#FFFFFF”),opacity:active?1:0.4}));
    \n roll.appendChild(el(“line”,{x1:PADL,y1:ry(r),x2:W-PADR,y2:ry(r),stroke:”#EFF1F5″,”stroke-width”:1}));
    \n var lbl=el(“text”,{x:PADL-8,y:ry(r)+rowH\/2+4,”text-anchor”:”end”,”font-size”:11,
    \n fill:active?”#14161C”:”#B6BCC7″,”font-weight”:active?600:500,”font-family”:”Inter,sans-serif”});
    \n lbl.textContent=rows[r].name; roll.appendChild(lbl);
    \n }
    \n \/\/ time gridlines every 2s (segment markers)
    \n for(var t=0;t<=DUR;t+=2){
    \n roll.appendChild(el(“line”,{x1:tx(t),y1:PADT,x2:tx(t),y2:H-PADB,stroke:”#E7E9EE”,”stroke-width”:1,”stroke-dasharray”:”2 4″}));
    \n var tl=el(“text”,{x:tx(t),y:H-9,”text-anchor”:”middle”,”font-size”:10,fill:”#9AA1AE”,”font-family”:”Inter,sans-serif”});
    \n tl.textContent=t+”s”; roll.appendChild(tl);
    \n }
    \n roll.appendChild(el(“line”,{x1:PADL,y1:H-PADB,x2:W-PADR,y2:H-PADB,stroke:”#E7E9EE”,”stroke-width”:1}));
    \n }<\/p>\n

    var noteLayer,headLine;
    \n function render(reveal){
    \n drawGrid();
    \n var c=classify(current);
    \n noteLayer=el(“g”,{}); roll.appendChild(noteLayer);
    \n function bar(n,color,opacity){
    \n var x=tx(n[1]),w=Math.max(6,tx(n[1]+n[2])-tx(n[1]));
    \n var y=ry(n[0])+rowH*0.28,h=rowH*0.44;
    \n var vis=(!reveal)|| (n[1]<=playhead);
    \n var rect=el(“rect”,{x:x,y:y,width:w,height:h,rx:3,fill:color,opacity:vis?opacity:0});
    \n rect.style.transition=”opacity .18s”;
    \n noteLayer.appendChild(rect);
    \n }
    \n c.fn.forEach(function(n){bar(n,”#16A34A”,0.55);});
    \n c.fp.forEach(function(n){bar(n,”#DC2626″,0.8);});
    \n c.tp.forEach(function(n){bar(n,”#2563EB”,0.95);});
    \n headLine=el(“line”,{x1:tx(playhead),y1:PADT,x2:tx(playhead),y2:H-PADB,stroke:”#5B9A00″,”stroke-width”:2,opacity:reveal?0.9:0});
    \n roll.appendChild(headLine);
    \n postSize();
    \n }<\/p>\n

    function setMetrics(){
    \n var s=STAGES[current];
    \n animateNum(“mOnset”,s.onset); animateNum(“mMulti”,s.multi);
    \n document.getElementById(“bOnset”).textContent=”baseline YourMT3+ 32.5″;
    \n document.getElementById(“bMulti”).textContent=”baseline YourMT3+ 21.9″;
    \n document.getElementById(“barOnset”).style.width=s.onset+”%”;
    \n document.getElementById(“barMulti”).style.width=(s.multi\/60*100)+”%”;
    \n document.getElementById(“stageNote”).textContent=s.note;
    \n }
    \n function animateNum(id,to){
    \n var e=document.getElementById(id),from=parseFloat(e.textContent)||0,st=null,dur=550;
    \n function step(ts){if(!st)st=ts;var p=Math.min(1,(ts-st)\/dur);
    \n e.textContent=(from+(to-from)*(1-Math.pow(1-p,3))).toFixed(1);
    \n if(p<1)requestAnimationFrame(step);}
    \n requestAnimationFrame(step);
    \n }<\/p>\n

    \/\/ event stream
    \n var stream=document.getElementById(“stream”);
    \n function resetStream(){stream.innerHTML='<div class="”h”">Event stream · model.transcribe()<\/div>’;streamed={};}
    \n function pushEvent(n,kind){
    \n var pitchNames=[“C2″,”G2″,”E3″,”C4″,”A4″,”D5”];
    \n var line=document.createElement(“div”);line.className=”ln”;
    \n var p=pitchNames[n[0]]||”C4″;var inst=rows[n[0]].inst;
    \n if(kind===”start”){
    \n line.innerHTML='<span class="”st”">NoteStart<\/span> t=<span class="”pi”">’+n[1].toFixed(2)+’s<\/span> pitch=<span class="”pi”">’+p+'<\/span> inst=<span class="”in”">’+inst+'<\/span>’;
    \n }else{
    \n line.innerHTML='<span class="”en”">NoteEnd<\/span> t=<span class="”pi”">’+(n[1]+n[2]).toFixed(2)+’s<\/span> (‘+inst+’)’;
    \n }
    \n stream.appendChild(line);stream.scrollTop=stream.scrollHeight;
    \n }<\/p>\n

    function play(){
    \n if(playing)return; playing=true;
    \n var btn=document.getElementById(“playBtn”);btn.disabled=true;btn.textContent=”\u25cf Transcribing\u2026”;
    \n playhead=0; resetStream();
    \n var c=classify(current);
    \n var visible=c.tp.concat(c.fp).sort(function(a,b){return a[1]-b[1];});
    \n render(true);
    \n var t0=null,SPEED=DUR\/4200; \/\/ ms mapping
    \n function frame(ts){
    \n if(!t0)t0=ts;
    \n playhead=Math.min(DUR,(ts-t0)*SPEED);
    \n \/\/ reveal notes + emit events
    \n visible.forEach(function(n,i){
    \n var key=”s”+i;
    \n if(n[1]<=playhead && !streamed[key]){streamed[key]=1;
    \n var rects=noteLayer.querySelectorAll(“rect”);
    \n pushEvent(n,”start”);
    \n setTimeout((function(nn){return function(){pushEvent(nn,”end”);};})(n),120);
    \n }
    \n });
    \n \/\/ update opacities
    \n Array.prototype.forEach.call(noteLayer.querySelectorAll(“rect”),function(r){});
    \n render(true);
    \n if(playhead<DUR){raf=requestAnimationFrame(frame);}
    \n else{playing=false;btn.disabled=false;btn.textContent=”\"\u25b6\" Transcribe again”;
    \n headLine.setAttribute(“opacity”,”0″);postSize();}
    \n }
    \n raf=requestAnimationFrame(frame);
    \n }<\/p>\n

    \/\/ controls
    \n document.getElementById(“stages”).addEventListener(“click”,function(e){
    \n var b=e.target.closest(“button”);if(!b)return;
    \n Array.prototype.forEach.call(this.children,function(c){c.classList.remove(“on”);});
    \n b.classList.add(“on”);current=b.getAttribute(“data-s”);
    \n playhead=DUR;render(false);setMetrics();postSize();
    \n });
    \n document.getElementById(“condToggle”).addEventListener(“click”,function(){
    \n cond=!cond;this.classList.toggle(“on”,cond);
    \n playhead=DUR;render(false);postSize();
    \n });
    \n document.getElementById(“playBtn”).addEventListener(“click”,play);<\/p>\n

    \/\/ pipeline shimmer
    \n var pnodes=document.querySelectorAll(“#pipe .node”),pi=0;
    \n setInterval(function(){
    \n pnodes.forEach(function(n){n.classList.remove(“hot”);});
    \n pnodes[pi].classList.add(“hot”);pi=(pi+1)%pnodes.length;
    \n },900);<\/p>\n

    \/\/ resize to parent (WordPress embed)
    \n function postSize(){
    \n try{var h=document.body.offsetHeight+40;
    \n window.parent.postMessage({muscriptorHeight:h},”*”);}catch(e){}
    \n }
    \n window.addEventListener(“resize”,function(){render(false);postSize();});<\/p>\n

    \/\/ init
    \n playhead=DUR;render(false);setMetrics();setTimeout(postSize,60);setTimeout(postSize,400);
    \n})();
    \n<\/script>
    \n<\/body>
    \n<\/html>
    \n“><\/p>\n

    \n

    Performance<\/strong><\/h2>\n<\/p>

    For evaluation, the research team use D<sub>Test<\/sub><\/strong>, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir_eval<\/code> library. Among them, Multi F1 is strictest, since it also requires the correct instrument.<\/p>\n

    The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.<\/p>\n

    \n\n\n\n\n\n\n\n\n
    Model (D<sub>Test<\/sub>)<\/th>\nOnset F1<\/th>\nFrame F1<\/th>\nOffset F1<\/th>\nDrums F1<\/th>\nMulti F1<\/th>\n<\/tr>\n<\/thead>\n
    YourMT3+ (baseline)<\/td>\n32.5<\/td>\n45.5<\/td>\n17.8<\/td>\n41.4<\/td>\n21.9<\/td>\n<\/tr>\n
    MuScriptor \u00b7 D<sub>Synth<\/sub><\/td>\n34.5<\/td>\n48.9<\/td>\n16.1<\/td>\n21.0<\/td>\n16.2<\/td>\n<\/tr>\n
    MuScriptor \u00b7 D<sub>Synth<\/sub> + D<sub>Real<\/sub><\/td>\n54.4<\/td>\n69.3<\/td>\n42.3<\/td>\n43.3<\/td>\n41.6<\/td>\n<\/tr>\n
    MuScriptor \u00b7 D<sub>Synth<\/sub> + D<sub>Real<\/sub> + D<sub>RL<\/sub><\/td>\n60.4<\/strong><\/td>\n73.3<\/strong><\/td>\n49.0<\/strong><\/td>\n50.2<\/strong><\/td>\n48.2<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

    Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub>Real<\/sub> then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.<\/p>\n

    Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.<\/p>\n

    Getting Started<\/strong><\/h2>\n

    Installation takes one command, and inference streams note events directly.<\/p>\n

    \n
    \n
    \n
    \n
    <\/div>\n
    <\/div>\n
    <\/div>\n<\/div>\n
    Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
    # pip install muscriptor   (or: uv add muscriptor)\nfrom pathlib import Path\nfrom muscriptor import TranscriptionModel\n\n# Downloads the default \"medium\" variant (also accepts \"small\" \/ \"large\")\nmodel = TranscriptionModel.load_model()\n\n# Stream note events; optionally condition on known instruments\nfor event in model.transcribe(\"audio.wav\", instruments=[\"acoustic_piano\", \"drums\"]):\n    print(event)   # NoteStartEvent \/ NoteEndEvent \/ ProgressEvent\n\n# Or write a MIDI file directly\nPath(\"out.mid\").write_bytes(model.transcribe_to_midi(\"audio.wav\"))<\/code><\/pre>\n<\/div>\n<\/div>\n
    For the released models, keep cfg_coef<\/code> at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve<\/code> launches a browser web UI with a live piano roll.<\/p>\n
    Use Cases with Examples<\/strong><\/h2>\n
    Because the output is standard MIDI, many workflows open up:<\/p>\n