llm-as-a-verifier-dsh

LLM-as-a-Verifier for dsh — Best-of-N (Bo5) conversation mode

English | 中文文档

Give DeepSeek V4 Flash test-time scaling: V4 Flash + Bo5 self-verification reaches Fable-5-level scores — 88% on Terminal-Bench 2.1, frontier-model accuracy at a fraction of the cost (≈11× cheaper).

An independent dsh-native implementation of the test-time selection method from LLM-as-a-Verifier (arXiv:2607.05391, MIT). Method by the paper's authors; this implementation by Aispin.

The paper's idea, in one minute

Cheap models can generate great answers — they just can't recognize which one is great. LLM-as-a-Verifier closes that gap:

1. Sample N candidates from a cheap model (DeepSeek V4 Flash): slightly different attempts at the same task.
2. Grade with a fine-grained verifier — the same cheap model, asked to grade pairs of candidates on an A–T letter scale. The score is not the sampled letter: it is the expectation over the grade token's logprob distribution, Σ p(token)·φ(letter) — the model's full belief, not one dice throw.
3. Both orderings per pair cancel the verifier's position bias; repeated evaluations alternate slots.
4. Select the best — the paper's core result: V4 Flash sampling 5 candidates + self-verification matches Fable-5-level frontier scores on Terminal-Bench 2.1 (88.0%) at ~1/11 the cost.

This plugin packages that pipeline as a dsh plugin with a conversational twist: every assistant turn becomes Best-of-N automatically — you see one answer, the model produced five.

One plugin, three faces

Face	Entry	Use
Tool	verify tool	On demand — "use the verify tool to compare A/B/C", the agent calls it
Service	ctx.verifier.verify({ task, candidates })	For code — orchestration lines, other plugins
Mode	Best-of-N conversation mode	Invisible — Bo-N sessions sample every turn N ways, verify, replay only the winner

Install

From npm (the recommended path — resolves every dependency through your profile):

dsh plugin --profile <your-profile> add @aispin/plugin-verifier

Or plain npm:

npm install @aispin/plugin-verifier

Zero-config: the verifier inherits dsh's configured provider state (credentials + settings seams) — if you've configured DeepSeek on the Models page, it just works. Try it locally:

git clone https://github.com/aispin-dev/llm-as-a-Verifier-dsh.git
dsh plugin --profile <your-profile> add /path/to/llm-as-a-Verifier-dsh

Best-of-N: three-state switch (hot)

① settings global (Web UI panel) → ② session preset ("Bo-N mode") → ③ profile config default → off

The Web settings panel offers the tiers with transparent cost cards:

Tier	Model calls	Tokens	Latency
Off	1	1×	1×
Fast · Bo-3	~9	2–3×	~7–15s
Precise · Bo-5	~16	3–5×	~12–30s
Custom	2–8 ways	linear	linear

Every turn's footer meters the real spend: ⚡ Best-of-N · 5 选 1 → 候选 #2 · 20.0/20 · 24.3s · 10.8K tok

What's inside (implementation parity with the paper)

• Fine-grained reward: expected grade over the top-20 logprob distribution, A=20…T=1 grouped band scale, grading at temperature 1.0 (the natural belief distribution — never collapsed)
• PPT pivot tournament (the paper's O(N·k) selection): random Hamiltonian ring (each candidate lands exactly once per slot — bias cancels inside the ring) → top-k pivots → only non-pivot×pivot pairs graded. Live-verified: Bo-5 grading calls 20 → 11 (−45%)
• Prefix-cache prompt layout (paper v0.2.0, −3.4× uncached tokens): criteria at the prompt tail; role + scale + task + candidates form the shared prefix
• Capability-adaptive grading: logprobs endpoints (DeepSeek official) get expected-grade scoring; logprob-less endpoints (MiniMax, various gateway providers) auto-degrade to letter-sampling grading with double evaluation — any OpenAI-compatible endpoint works (autoDegrade: false for strict mode)
• Fail-open discipline: any breakdown degrades to a normal answer with an explanatory footer — never a dead turn

License

MIT © 2026 Aispin. The method is from LLM-as-a-Verifier (arXiv:2607.05391, MIT). Not affiliated with the paper's authors or DeepSeek.

---

中文文档

给 DeepSeek V4 Flash 测试时扩展能力：V4 Flash + Bo5 自验证达到 Fable 5 级评分——Terminal-Bench 2.1 上 88%，以前沿模型级别的准确率、约 1/11 的成本完成任务。

LLM-as-a-Verifier（arXiv:2607.05391, MIT）测试时选择方法的 dsh 原生独立实现。方法归论文作者，实现归 Aispin。

论文的思想，一分钟讲清

便宜模型能生成好答案——只是认不出哪个是好答案。LLM-as-a-Verifier 补上这一环：

1. 采样 N 个候选（DeepSeek V4 Flash）：同一任务的多个略有差异的尝试。
2. 细粒度验证器评分——同一个便宜模型，对候选两两成对按 A–T 字母量表打分。分数不是采样出的那个字母，而是 grade token 对数概率分布上的期望值 Σ p(token)·φ(letter)——模型的完整信念，不是掷一次骰子。
3. 每对双向各评一次抵消验证器的位置偏置；重复评估交替 A/B 槽位。
4. 选出最佳——论文核心结论：V4 Flash 采样 5 条候选 + 自验证择优，在 Terminal-Bench 2.1 上达到 Fable 5 级前沿评分（88.0%），成本约 1/11。

本插件把这套管线做成 dsh 插件，并加上对话形态：每个回答自动变成 Best-of-N——你看到一条答案，模型实际做了五条。

一个插件，三张面孔

面孔	入口	用法
工具面	verify 工具	有感——对话里说"用 verify 工具比较 A/B/C"，模型主动调用
服务面	ctx.verifier.verify({ task, candidates })	代码消费（编排线、其他插件）
模式面	Best-of-N 对话模式	无感——选中模式的会话，每轮后台 N 路采样 + 择优，只把胜者呈现给用户

安装

npm 安装（推荐——依赖经你的 profile 完整解析）：

dsh plugin --profile <your-profile> add @aispin/plugin-verifier

或直接 npm：

npm install @aispin/plugin-verifier

零配置：验证器继承 dsh 已配置的 provider 状态（credentials + settings seam）——在 Models 页面配过 DeepSeek 即可直接用。本地试用：

git clone https://github.com/aispin-dev/llm-as-a-Verifier-dsh.git
dsh plugin --profile <your-profile> add /path/to/llm-as-a-Verifier-dsh

Best-of-N：三态开关（热生效）

① settings 全局（Web 设置面板）→ ② session preset（"Bo-N 模式"）→ ③ profile config 默认 → 关

Web 设置面板的档位卡直接标注消耗透明：

档位	模型调用	token	延迟
关闭	1 次	1×	1×
快速经济 · Bo-3	~9 次	2–3×	~7–15s
精准 · Bo-5	~16 次	3–5×	~12–30s
自定义	2–8 路	线性	线性

每轮回答尾部 footer 显示实际开销：⚡ Best-of-N · 5 选 1 → 候选 #2 · 20.0/20 · 24.3s · 10.8K tok

实现要点（与论文对齐）

• 细粒度奖励：top-20 logprob 分布上的期望分（A=20…T=1 分组带量表），评分温度 1.0（读自然信念分布，绝不坍缩）
• PPT 概率枢轴锦标赛（论文 O(N·k) 选择算法）：随机哈密顿环（每候选恰好在 A/B 槽各一次——环内天然消位置偏置）→ top-k 枢轴 → 只补非枢轴×枢轴对。实测 Bo-5 评分调用 20 → 11（−45%）
• 前缀缓存布局（论文 v0.2.0，未缓存 token −3.4×）：criteria 置于 prompt 尾部，角色+量表+任务+候选构成跨调用共享前缀
• 能力自适应评分：有 logprobs 的端点（DeepSeek 官方）用期望分；没有的（MiniMax、部分网关）自动降级采样评分（每对双评补偿方差）——任何 OpenAI 兼容端点都能当评审（autoDegrade: false 切严格模式）
• Fail-open 纪律：任何断裂降级为普通回答并在 footer 说明原因，绝不杀死对话轮

许可

MIT © 2026 Aispin。方法来自 LLM-as-a-Verifier（arXiv:2607.05391, MIT）。与论文作者及 DeepSeek 无隶属关系。