Which Harness Runs DeepSeek V4 Best? dsh Benchmark Data
New third-party data on the best harness for DeepSeek V4: Composio's five-way cost and speed benchmark, KV cache economics, and Reddit reports on dsh vs rivals.
There is no single best harness for DeepSeek V4 — the honest answer, based on the third-party data that has actually appeared in the two weeks since launch, is that it depends on what you're optimizing for. The clearest dataset, a five-harness benchmark from Composio, has Pi Agent solving the most tasks, DeepSeek Harness (dsh) winning on cost per completed task, and Claude Code winning on raw speed. No single deepseek coding harness sweeps every column.
This post is a data-refresh companion to Best Harness for DeepSeek V4, which laid out the selection framework — native dsh integration versus pointing a provider-agnostic harness at DeepSeek's OpenAI-compatible endpoint — before any head-to-head numbers existed. In the two weeks since, several third parties have actually run the comparison. This post rounds up what they found, with sources, and updates the decision matrix accordingly. We aren't running our own benchmarks here; everything below is attributed to where it came from.
The Composio five-way benchmark
The most substantial data point so far comes from Composio's benchmark thread, published August 19, 2026 (78,670 views, 267 bookmarks at the time of the post). Composio ran the same model — DeepSeek V4 Pro 0813 — through five different harnesses on 30 hard agent tasks: Claude Code, DeepSeek Harness 0.1, Hermes, Pi Agent, and OpenCode.
The numbers Composio published:
| Harness | Result | Metric |
|---|---|---|
| Pi Agent | Solved the most tasks of the five | Task completion (exact count not published in the thread) |
| DeepSeek Harness 0.1 | $0.028 per successful task | Lowest cost of the five |
| Claude Code | $0.074 per successful task | ~2.6x dsh's cost |
| Claude Code | 181.8s median completion time | Fastest |
| DeepSeek Harness 0.1 | 252.1s median completion time | Second-fastest |
| Hermes, OpenCode | — | Not broken out in the public thread |
These numbers are taken from Composio's public test; conditions, task selection, and full methodology are in the original thread, not something we've independently reproduced. Read it that way — as the best available head-to-head data right now, not a verified-by-us benchmark. It does, however, corroborate a pattern that shows up repeatedly in community reports below: dsh's advantage over the field isn't raw task-solving power, it's cost efficiency.
The "lightweight vs. bloated" divide
A second, less quantitative but consistently repeated theme in local-model and self-hosted circles is that harness weight — specifically, how much the system prompt costs before a single token of actual work happens — matters more than people expect.
In a widely discussed r/LocalLLaMA thread about why dsh "feels" more efficient, the top comment argued that mainstream harnesses like OpenCode, Claude Code, and Codex tend to ship with large, feature-heavy system prompts, which is a real cost on models running locally with no token budget to spare. dsh and Pi were repeatedly named as the lighter-weight options in that same thread. One commenter cited external analysis claiming Hermes — a separate agent-orchestration layer, distinct from a harness proper — injects on the order of 19,000–20,000 tokens of system prompt; that figure is a community citation, not something we've verified against Hermes's source directly. The recurring consensus across that thread: a leaner context window translates into faster local prefill, which is the practical bottleneck for anyone running against llama.cpp or similar rather than a hosted API.
Local-model reality check: two data points, both ways
Reddit is also where the closest thing to real long-run local usage data shows up, and it cuts both directions.
On the positive side, in that same r/LocalLLaMA thread, one user described running dsh against a local llama.cpp setup (a Q4_K_M quantization) continuously for 16 hours and more than 20 million tokens, completing every task in the run without failure — a genuine stability data point for local deployment, source-attributed to that single Reddit account rather than an independent test.
On the negative side, a separate r/LocalLLaMA thread from dsh's launch week has a user reporting instability specifically under WSL, switching to the third-party harness reasonix as a workaround — echoed by another reply noting similar complaints about Claude Code's WSL behavior. Neither report is a controlled test; both are single-user accounts. We're including both because the honest picture, based on what's actually been reported, is mixed rather than uniformly positive.
Why dsh looks cheap: KV cache hit rates
The cost gap in Composio's numbers lines up with a separate, recurring community observation: DeepSeek's API-level prompt caching produces unusually high cache hit rates in agentic workloads, where large chunks of context repeat across turns. Reddit threads in r/DeepSeek and a second r/DeepSeek thread both report cache hit rates in the 97–99% range during real sessions, tracking DeepSeek's own KV cache documentation. This is a DeepSeek API-level feature, not something unique to dsh's implementation — but since dsh is DeepSeek's own harness, it's positioned to exploit that caching behavior without any extra configuration. We cover the mechanics — prefill, prefix matching, and what counts toward a cache hit — in a separate deep dive: DeepSeek Harness KV Cache, Explained.
If cost tracking matters to your own workflow, the community plugin ecosystem already has tooling for it — for example dsh-token-anxiety, a plugin that surfaces per-task spend and peak/off-peak pricing windows:
dsh plugin --profile web add github:mov-eax-eax/dsh-token-anxiety
Browse more cost and usage tooling under Usage & Billing, or the full catalog at FindHarness's plugin index — currently tracking 178,538 stars on the main dsh repository as of August 21, 2026.
Decision matrix: pick by what you're optimizing for
None of this data points at one universal winner. Based on what's actually been published:
| If you want... | Pick | Why |
|---|---|---|
| Highest raw task-solve rate | Pi Agent | Solved the most of 30 tasks in Composio's test |
| Lowest cost per completed task | dsh | $0.028 vs Claude Code's $0.074 in the same test; DeepSeek's cache economics help |
| Fastest wall-clock completion | Claude Code | 181.8s median vs dsh's 252.1s in Composio's test |
| To run fully local/self-hosted | dsh or Pi | Both repeatedly named as lightweight-prompt options in local-model threads |
| The deepest plugin/ecosystem around DeepSeek specifically | dsh | Its plugin catalog (see /plugins) is purpose-built for this model family; no other harness in this comparison has an equivalent |
| To keep an existing Claude Code workflow | Claude Code, or dsh delegating to it | dsh can natively delegate tasks to Claude Code as a subagent rather than replacing it outright — see DeepSeek Harness vs. Claude Code |
Read that table plainly: dsh is not the best deepseek v4 harness on every axis, and this data doesn't claim otherwise. If your priority is solving the most tasks or finishing fastest, the numbers above point elsewhere. Where dsh's case is strongest is cost efficiency and plugin depth specifically for this model family — which is exactly what you'd expect from the first-party option, and exactly what the selection framework in the previous post predicted before any benchmark existed to confirm it.
A necessary caveat about all benchmark numbers here
Every number above came from a single public source, run once, on a fixed task set, with a specific prompt and tool configuration. That matters more for DeepSeek V4 than for most models: separate benchmark analysis this month found DeepSeek V4 Pro's agent behavior is unusually sensitive to the exact starting prompt and tool schema it's given, which is part of why the same model has scored anywhere from the high 50s to the high 80s on Terminal-Bench depending on which harness ran it. We go into that discrepancy — and why it isn't evidence of anyone cheating — in Why DeepSeek Harness Benchmark Scores Differ. Treat the Composio numbers, and everything else in this post, as directional rather than final.
FAQ
What's the best harness for DeepSeek V4 right now?
There isn't one universal answer. Based on the one public head-to-head test available (Composio, 30 tasks), Pi Agent solved the most tasks, dsh was cheapest per completed task, and Claude Code was fastest. Pick based on which of those three axes matters most for your workload.
Is dsh the best harness for DeepSeek overall, or just cost-efficient?
The public data specifically shows a cost advantage ($0.028 vs. $0.074 per successful task against Claude Code), not a solve-rate or speed advantage. dsh also has DeepSeek's deepest first-party plugin ecosystem, which is a separate reason to pick it beyond the benchmark numbers.
Why does the same DeepSeek V4 harness benchmark score differently everywhere?
Because DeepSeek V4 Pro's agent behavior appears to be sensitive to the exact system prompt and tool schema it's given — not because any party is misrepresenting results. See Why DeepSeek Harness Benchmark Scores Differ for the fuller explanation.
Does a leaner system prompt actually make a measurable difference?
Community reports consistently associate lighter system prompts (dsh, Pi) with better local-model performance versus heavier ones (OpenCode, Claude Code, Hermes), largely because of prefill cost — but this is drawn from Reddit discussion and anecdotal usage, not a controlled benchmark, so treat it as directional.
Can I use dsh alongside Claude Code instead of choosing one?
Yes — dsh can delegate tasks to Claude Code as a subagent rather than requiring you to pick one exclusively. See DeepSeek Harness vs. Claude Code for how that works.
Next steps
Start with the selection framework in Best Harness for DeepSeek V4 if you haven't picked a harness yet, then read Why DeepSeek Harness Benchmark Scores Differ before trusting any single number too far. If cost is your main driver, DeepSeek Harness KV Cache, Explained covers the mechanism behind dsh's price advantage, and Usage & Billing plugins round up the community tooling for tracking it yourself. For the general two-week retrospective on what's changed since launch, see DeepSeek Harness: Two Weeks In.