Why DeepSeek Harness Benchmark Scores Differ So Wildly
DeepSeek Harness benchmark scores swing 30+ points across setups — a Terminal-Bench 2.1 and SWE-bench gap explained, with sources, for DeepSeek V4 Pro.
DeepSeek Harness benchmark scores vary by dozens of points not because a number was faked, but because an agent's tool-calling policy is highly sensitive to which harness runs it. On Terminal-Bench 2.1, DeepSeek V4 Pro (0813) scored 87.9 under the official DeepSeek Harness setup — and 54.68 when reran by a third party through a different harness (Terminus 2), a 33-point swing on the same model, same benchmark. The evidence trail behind that gap points to prompt and tool-schema sensitivity, not fraud, and it's part of a much bigger, industry-wide problem with how agent benchmarks get reported.
This launched two weeks ago as a "did DeepSeek lie" fight. It has since become something more useful: a case study in why an agent benchmark score without its harness disclosed is close to a number without units.
The 33-point gap: same model, two harnesses
The clearest documented instance of the discrepancy is Terminal-Bench 2.1:
| Score | Harness | Status |
|---|---|---|
| 87.9 | Official DeepSeek Harness | Vendor-reported |
| 54.68 | Vals AI, rerun using the Terminus 2 harness | Third-party |
| "mid-fifties" | An independent, stricter rerun | Third-party, cited secondhand |
The 87.9-vs-54.68 comparison and the "87.9, 78.7, mid-fifties" framing both come from a widely shared analysis by @shmidtqq (amplified by @ST4RHaze), and from a longer Chinese-language writeup translated and shared by @ZhihuFrontier, titled (translated) "Did DeepSeek V4 Pro overfit to its own harness? The real issue might be interface sensitivity." Both are community reports, not an official DeepSeek statement — but they're the most detailed public account of the gap, and they agree on the shape of it.
It's not "home-turf" bias — the same pattern shows up inside dsh itself
The tempting read of an 87.9-vs-54.68 gap is that DeepSeek tuned V4 Pro to game its own harness and fall apart everywhere else. The Zhihu analysis pushes back on that with a more interesting data point: the same spread shows up inside DeepSeek Harness itself, across its own built-in agent modes, on the same benchmark:
- Minimal mode: 99 / 96
- Standard mode: 91
- PTC (Code) mode: 92
If the model were simply locked to "its own framework," these three numbers — all running inside the same harness — should look similar. They don't. That rules out the simplest cheating story and points somewhere more specific: the starting prompt and the tool schema, not the harness brand.
The analysis backs this with what it calls an anchoring experiment: start a session using Minimal mode's prompt and its two-tool set, then — after the very first tool call — restore the full 25-tool set and let the rest of the run proceed normally. The score barely moves: still 98/99. In other words, what the model does in the first turn matters far more than what tools are nominally available for the rest of the session.
What Minimal mode actually is
DeepSeek Harness ships a built-in minimal agent preset: a system prompt fixed to You are a helpful software engineer assistant., with only two tools mounted — bash and str_replace_editor — and none of the extra prompt sections or model-side plugins that Standard or PTC mode carry (see the DeepSeek Harness GitHub repository for the full agent-mode reference). The Zhihu analysis argues this isn't a benchmark trick bolted on after the fact — it's a close match to the fixed prompt/tool distribution DeepSeek's RL post-training likely used. Under that read, V4 Pro's agent training produced a model with a higher ceiling but weaker cross-framework robustness: excellent when the runtime looks like what it was trained against, more fragile when it doesn't.
Not unique to DeepSeek: an industry-wide benchmark problem
The sharper framing from @shmidtqq's thread cites a UC Berkeley stress test (April) of the eight most commonly cited agent benchmarks: seven of them could reportedly be gamed to a 100% score without the agent actually solving a single underlying task. On SWE-bench Verified specifically, the same research reportedly found a 10-line change that let one test configuration pass all 500 cases — without genuinely fixing the bugs those cases were meant to catch.
That's the context for the blunter conclusion circulating this week: an agent benchmark score, without the harness behind it disclosed, is a number without units. The same model can look brilliant or mediocre purely as a function of prompt structure, tool schema, and verifier leniency — none of which show up in a single headline percentage.
A five-harness comparison, for scale
The most concrete third-party cross-harness data point so far comes from @composio, which ran the same DeepSeek V4 Pro (0813) model through five different harnesses — Claude Code, DeepSeek Harness 0.1, Hermes, Pi Agent, and OpenCode — on 30 hard agentic tasks. The one-line takeaway: Pi Agent solved the most tasks, while DeepSeek's own harness won on cost. We break down the full methodology and per-harness numbers in Which Harness Runs DeepSeek V4 Best? — the point here is simpler: change only the harness, keep the model fixed, and the ranking moves. That's the same phenomenon as the Terminal-Bench gap, just measured deliberately instead of stumbled into.
The other criticism: you can't rerun DeepSeek's own numbers
There's a separate, more skeptical complaint worth stating plainly, and it doesn't get resolved by the interface-sensitivity explanation above: DeepSeek's own agentic benchmark harness — the specific prompt/tool configuration behind its headline vendor-reported numbers — has not been published. Third-party coverage of the V4 Pro 0813 launch (flagged in our own research notes as unverified, not independently confirmed by us) has claimed that DeepSeek's SWE-bench Verified figure could not be reproduced by outside evaluators running their own harnesses, and that a third-party evaluator's own rerun landed at a different number entirely. We're deliberately not repeating the specific figures here, because the sourcing behind them is too thin to stand behind — but the underlying complaint (closed harness, no reproduction path) is legitimate on its own, independent of whether any particular third-party number is accurate.
How to actually read an agent benchmark score
Given all of the above, a practical checklist for the next "Model X scores Y% on benchmark Z" headline:
- Is the harness named and versioned? "DeepSeek Harness" isn't specific enough — Minimal, Standard, and PTC mode produced an 8-point spread on the same benchmark above.
- Is the harness's code public and rerunnable? If not, treat the number as vendor-reported and move on — don't average it with third-party numbers as if they measured the same thing.
- What's the verifier? A benchmark's pass/fail judge can be as consequential as the model itself; loose verifiers inflate scores independent of real task completion.
- Has anyone reported cross-harness variance for this specific model? If the answer is no, the single number you're looking at is likely one point in a much wider unreported range.
For choosing a harness for your own work, the only number that actually matters is the one you produce on your own workload. Start from a working install — see the DeepSeek Harness quickstart — and if you want a plugin that ships its own reproducible benchmark rather than a marketing claim, dsh-plugin-bridge is a useful example: it documents a 26-round benchmark for its own cross-preset session migration feature, right alongside the code.
dsh plugin --profile web add github:Totoro-qaq/dsh-plugin-bridge
Browse more benchmarking-adjacent and dev tooling in Dev & Plugin Tools, or the full plugin catalog on FindHarness.
FAQ
Did DeepSeek fake its DeepSeek Harness benchmark numbers?
No credible evidence supports that. The most detailed public analysis found the same wide score spread inside DeepSeek Harness's own agent modes (Minimal vs. Standard vs. PTC), which rules out simple home-turf gaming and points instead to prompt/tool-schema sensitivity in how V4 Pro's agent training generalizes.
Why does Minimal mode score higher than Standard mode?
Minimal mode's fixed system prompt and two-tool set (bash + str_replace_editor) is argued to closely match the prompt/tool distribution used during DeepSeek's agent RL post-training — so the model performs closer to its training conditions. Standard and PTC mode add more tools and prompt structure, which appears to reduce peak benchmark performance even though it may help in other ways.
Can I compare a DeepSeek Harness score to a Claude Code or OpenCode score directly?
Only if both ran the identical task set, the identical verifier, and you know each harness's exact prompt/tool configuration. Otherwise you're comparing two different measurement instruments, not two models.
Is this a DeepSeek-specific problem?
No — a cited UC Berkeley stress test found 7 of the 8 most commonly referenced agent benchmarks could be gamed to 100% without solving the underlying tasks, and reported a 10-line change that passed all 500 SWE-bench Verified cases in one configuration. Interface/harness sensitivity is a benchmark-methodology problem across the field, not unique to this model.
Where can I find real cross-harness comparisons instead of vendor numbers?
See Which Harness Runs DeepSeek V4 Best? for a breakdown of the Composio five-harness comparison, and treat any single-harness benchmark claim — ours included — as one data point rather than a verdict.
Next steps
Read Which Harness Runs DeepSeek V4 Best? for the full cross-harness cost/speed/completion breakdown, catch up on the rest of this week's ecosystem news in DeepSeek Harness: Two Weeks In, or start from zero with the DeepSeek Harness quickstart if you'd rather just run your own workload through it and see the number that actually matters.