{"id":117262,"date":"2026-09-12T01:44:25","date_gmt":"2026-09-12T01:44:25","guid":{"rendered":"https:\/\/youzum.net\/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-and-a-ci-gate-for-skills\/"},"modified":"2026-09-12T01:44:25","modified_gmt":"2026-09-12T01:44:25","slug":"anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-and-a-ci-gate-for-skills","status":"publish","type":"post","link":"https:\/\/youzum.net\/zh\/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-and-a-ci-gate-for-skills\/","title":{"rendered":"Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills"},"content":{"rendered":"

Anthropic has published a new plugin evals workflow for Claude Code<\/a>. The claude plugin eval<\/code> command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.<\/p>\n

Deployable:<\/strong> Yes. It runs on Claude Code v2.1.269 or later against any directory with a plugin.json<\/code> \u6216\u8005 .claude-plugin\/plugin.json<\/code> manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account.<\/p>\n

What a case looks like<\/strong><\/h2>\n

An eval suite lives in an evals\/<\/code> directory inside the plugin. Each case is a subdirectory holding a prompt.md<\/code> and a graders\/<\/code> folder. The prompt body goes to Claude exactly as written, and @path<\/code> mentions are not expanded. Frontmatter on prompt.md<\/code> can set max_turns<\/code> (default 10), timeout_seconds<\/code> (default 300), model<\/code>, tags<\/code>, and allowed_tools<\/code>.<\/p>\n

Graders are markdown files whose frontmatter sets a type<\/code>, an optional weight<\/code>, and an optional arm<\/code>. There are 6 types. Four cost nothing because they are computed from the transcript and the files on disk: regex<\/code>, tool_used<\/code>, tool_order<\/code>, and file_exists<\/code>. Two call a judge model and add to the bill: llm<\/code>, which scores the reply against prose criteria you write, and baseline<\/code>, which compares it against a reference answer.<\/p>\n

claude plugin eval init<\/code> reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, --bare <name><\/code> writes a blank template instead.<\/p>\n

The number that matters is \u0394<\/strong><\/h2>\n

By default every case runs twice: a with-arm where the plugin is loaded and a without-arm where it is not. Their difference, \u0394, is what the plugin contributed. If a case scores 1.0 in both arms, the plugin is not why it passed. The docs example output shows a single case at WITH 1.00, W\/OUT 0.33, \u0394 +0.67 across 6 runs, costing an estimated $0.41 and taking 74 seconds. A grader marked with-only, typically tool_used: Skill<\/code>, is reported as an indicator and excluded from the score, since the without-arm has no skill to fire.<\/p>\n

Anthropic calls out the most common first finding: a \u0394 near zero with the tool_used: Skill<\/code> grader failing, which means Claude is not choosing the skill on natural phrasing. That is the defect claude plugin validate<\/code><\/a> cannot see, because it checks manifest syntax and schema rather than behavior.<\/p>\n

Results land under evals\/results\/<timestamp>\/report.html<\/code> with per-grader verdicts and judge votes. Where the account supports it, the report is also published to claude.ai unless --no-publish<\/code> is set.<\/p>\n

Cost and CI<\/strong><\/h2>\n

A suite makes roughly cases \u00d7 runs \u00d7 arms agent runs, plus 3 short judge calls per llm<\/code> \u6216\u8005 baseline<\/code> grader per run, and results vary between runs. The documented CI invocation is:<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
claude plugin eval . \n  --trust-plugin \n  --json results.json \n  --threshold 0.8 \n  --model claude-sonnet-5 \n  --judge-model claude-haiku-4-5 \n  --no-publish \n  --max-cost-usd 20<\/code><\/pre>\n<\/div>\n<\/div>\n
The runner needs a Claude Code install and credentials such as ANTHROPIC_API_KEY<\/code>. Without --trust-plugin<\/code>, an untrusted checkout is refused with exit 1 when there is no terminal. Report problems never change the exit code, and --json<\/code> suppresses progress output.<\/p>\n
Interactive explainer<\/strong><\/h2>\n
<\/div>\n
\n
Key Takeaways<\/strong><\/h2>\n