dsh-jev-plugin

dsh-jev-plugin

Ask Jev, a System One decision model, typed questions from DeepSeek Harness and get structured answers with probabilities.

Jev returns judgements, not prose: every question is evaluated in parallel against the same block of text, so a request carrying 50 questions costs about the same and takes about as long as one carrying 1. That makes it a good fit for the judgements a coding agent has to repeat over many items — log lines, tool outputs, failing tests, search results, candidate files, review comments — without pulling them all into the model's context or spawning a subagent to read them.

Measured against the two things the harness does today, on the same task and the same data: 13x faster and 13.6x cheaper than a model at identical accuracy, and 105x faster and ~120x cheaper than a subagent (median subagent: 9 requests, 48 s, $0.035). It loses to grep whenever the criterion is mechanical, so do not use it for that. The full evidence — including the use cases that were measured and then rejected — is in docs/MEASUREMENTS.md. What this plugin deliberately refuses to do, and the measurement behind each refusal, is in docs/DECISIONS.md.

What it registers

Surface	What it does
The jev tool	Sends state + questions and returns the typed answers, probabilities and confidence.
Settings → Plugins → Jev	Per-user provider choice, model override, custom endpoint, API key, and a Test connection button.

Install

The package carries its own bundle patch, so installing it is one command. dsh plugin forwards to pnpm, so a local path, a git spec and a registry name all work:

dsh plugin --profile web add /path/to/dsh-jev-plugin         # a local checkout
dsh plugin --profile web add github:RaulLazaro/dsh-jev-plugin  # straight from GitHub
dsh plugin --profile web add dsh-jev-plugin              # from the registry, once published

That adds the dependency, appends dsh-jev-plugin to the profile's bundle list, and composes the row from cordis.patch.yml. Confirm the composition without booting anything:

dsh --profile web --dump-config | grep -A4 'dsh-jev-plugin'
# - id: jev
#   name: dsh-jev-plugin
#   config:
#     enabled: true
#     provider: typesafe

Then restart the harness to mount it — a running process does not pick up a new bundle row. Nothing else is required: the API key is entered in the UI, not in the composition.

To remove it again: dsh plugin --profile web remove dsh-jev-plugin.

Configure

Open Settings → Plugins → Jev.

1. Provider — where Jev is served from (see below).
2. Model override — optional; empty means the provider's default.
3. Base URL — only for the Custom provider.
4. API key — stored in the DSH credentials store, never in settings.yaml. The field shows whether a key is already configured.
5. Test connection — performs one real round trip and reports the resolved provider, model, latency, input tokens and the answers.

Providers

Provider	Endpoint	Default model	Credential / env fallback
TypeSafe (direct)	https://api.typesafe.ai/v1/systemone	jev-latest	TYPESAFE_API_KEY
Vercel AI Gateway	https://ai-gateway.vercel.sh/typesafe/v1/systemone	typesafe-ai/jev	AI_GATEWAY_API_KEY
Custom	your own URL	jev-latest	JEV_API_KEY

Each provider has its own key, so switching provider switches which credential is used. If you already exported the matching environment variable, no key needs to be saved.

Evaluation is not served through the OpenAI- or Anthropic-compatible endpoints. Both providers above expose a TypeSafe-compatible POST …/v1/systemone; a custom endpoint must implement the same contract.

Using the tool

{
  "state": "### ITEM 1\nTypeError: x is undefined\n\n### ITEM 2\nAll 42 tests passed in 1.2s",
  "questions": {
    "item1_failed": { "type": "noul",   "instructions": "Does ITEM 1 report a failure?" },
    "item2_failed": { "type": "noul",   "instructions": "Does ITEM 2 report a failure?" },
    "worst": {
      "type": "choice",
      "instructions": "Which item is the failure?",
      "criteria": { "item1": "a code exception", "item2": "a passing run" }
    }
  }
}

Question type	Answer
noul	noul — the probability of yes, 0–1
choice	choice, probabilities, confidence
score	score (an ordered rubric, may fall between rungs), probabilities, confidence

Ask atomic questions. Reference each item by a stable label (### ITEM 7) and ask one question per item; a question about a whole document invites a guess. Instructions and criteria are written in English; the state may be in any language.

Read the probabilities, not just the pick. A flat distribution means it is guessing; confidence between roughly 0.4 and 0.6 on a noul carries little information. Measure confidence against your own workload before gating an action on it — it is a statistic derived from the distribution, not a guarantee.

When not to use it

• Writing, summarising, translating or reformatting text — it cannot generate.
• Exact string or arithmetic work — it is not a regex engine or a calculator.
• Anything needing extended multi-step reasoning — decompose into atomic questions and combine the answers in code instead.

Configuration reference

The jev settings namespace accepts:

Field	Default	Meaning
enabled	true	Register the jev tool.
provider	typesafe	typesafe, vercel-gateway or custom.
model	(provider default)	Model id override.
baseUrl	—	Required by the custom provider.
timeoutMs	60000	Per-attempt request deadline.
maxStateChars	40000	Rejects oversized state with a message naming the limit.
maxQuestions	200	Rejects oversized question maps.
dataDir	<DSH_HOME>/dsh-jev	Where the judgment ledger lives.
dailyCallLimit	500	Judgments per local day; 0 disables the cap.
dailyTokenLimit	5000000	Input tokens per local day; 0 disables the cap.

Provider, model and limit fields can also be set directly in ~/.dsh/settings.yaml; the API key cannot, by design.

Spend, and the ledger behind it

A Jev call is not free and, without a record, it is invisible. Measured on real traffic: 11,377 input tokens for a single judgment, in a turn that can present 27 oversized results. Input is billed at $0.042 per million tokens and output is free, so cumulative input tokens is the cost.

Every call — including a failed one — is appended to a JSONL ledger under dataDir, and the Settings card shows the running total for the local day:

Today · 42 judgments · 318,904 input tokens · $0.0134

Two caps read that ledger before each request. They exist because the failure mode of a metered feature is not an error, it is a surprise:

• dailyCallLimit (default 500)
• dailyTokenLimit (default 5,000,000 input tokens)

Exceeding either refuses the call before anything is spent, with a message naming the number reached and the setting to raise. A cap of 0 means no cap. The ledger never blocks a judgment: if its directory cannot be prepared, it degrades to memory-only for that run rather than failing the call.

Nothing here thresholds on a probability. Jev's numbers are measured to be uncalibrated — accuracy by confidence band came out 33% / 25% / 67% / 57%, including two wrong answers at 0.85 and 0.96 confidence — so they are safe to sort by and not to compare against a fixed number. See docs/DECISIONS.md for what that rules out.

Operational notes

• The upstream answers 429 under load ("the upstream provider is currently experiencing high demand"). The plugin retries with a 1s → 25s backoff and honours Retry-After; expect occasional slow calls and do not treat a retry as a failure.
• A rejected credential (401/403) is reported with a pointer to Settings → Jev rather than a raw status code.
• The Settings bridge is same-origin and loopback-only.

Verify it works

After the restart, in order:

1. The card is there. Settings → Plugins → Jev should appear with a key set / no key badge.

2. The key round-trips. Pick the provider, paste the key, Save key, then Test connection. You get the resolved provider, model, latency, input tokens and a couple of answers from a real request.

3. The tool is registered. Ask the session to call jev on something small, e.g.:

   {
     "state": "### ITEM 1\nTypeError: x is undefined\n\n### ITEM 2\nAll 42 tests passed",
     "questions": {
       "item1_failed": { "type": "noul", "instructions": "Does ITEM 1 report a failure?" },
       "worst": { "type": "choice", "instructions": "Which item failed?",
                  "criteria": { "item1": "an exception", "item2": "a passing run" } }
     }
   }
   

   Expected: item1_failed: 0.9x (yes), item2 near 0.0x, and worst: item1.

4. A misconfiguration fails usefully. Switch to Custom with an empty base URL: the tool must answer with a message naming Settings → Jev, not a raw status code.

5. The bridge stays closed. curl -X POST http://your-host:3080/api/dsh-jev-settings/describe from another machine must answer 403 loopback requests only.

Development

npm test          # node --test

The suite covers endpoint resolution, question validation messages, answer formatting, key precedence and the retry policy, plus the tool's own wiring: that it reads the credentials service through the lazy accessor on every call rather than capturing it once at registration. lib/index.js imports nothing but @deepseek-ai/schemastery, declared as a peer dependency along with @deepseek-ai/cordis — the host supplies both.

Licence

MIT.