dsh-vision-bridge

@zzdream67/dsh-vision-bridge

Let text-only models read images, without switching models.

中文文档

In DeepSeek Harness, pasting an image at a text-only model is refused outright:

Model "xxx" does not support image input.

This plugin intercepts the llm/stream waterfall and substitutes each image block with a vision model's transcription before the request reaches the provider. You paste a picture; the text-only model receives words it can read.

you paste an image + a question
   ↓
the plugin captions it with YOUR configured vision model
   ↓
the text-only model receives: [image "shot.png" transcribed: a Python TypeError…] + your question
   ↓
it answers from that text

---

Installation

Prerequisite: the dsh command

Every command below starts with dsh. If it is not on your PATH:

dsh : The term 'dsh' is not recognized as the name of a cmdlet, function, script file, or operable program.

Then either install DSH globally:

npm install -g @deepseek-ai/dsh

Or leave it uninstalled and prefix every dsh in this README with npx -y @deepseek-ai/dsh:

npx -y @deepseek-ai/dsh plugin --profile web add @zzdream67/dsh-vision-bridge

The npx form downloads DSH on demand and works on a machine that has only Node.js.

Prerequisite: pnpm

dsh plugin forwards to pnpm internally. Many environments lack it:

'pnpm' is not recognized as an internal or external command
dsh: pnpm failed in profile directory ...

Install it:

npm install -g pnpm

corepack enable pnpm often fails on Windows because it needs to write into C:\Program Files\nodejs\ (EPERM). The npm install -g route is more reliable.

From npm (recommended)

dsh plugin --profile web add @zzdream67/dsh-vision-bridge

Installs prebuilt output; no build authorization needed.

From GitHub

dsh plugin --profile web add github:zzdream67/dsh-vision-bridge

A git install fetches source and builds it on install. pnpm ≥10 refuses to run a git dependency's build script without explicit permission, so the first add fails with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED.

That error prints the exact key to allow, ending with the commit it resolved:

allowBuilds:
  @zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>: true

Copy that key verbatim into the profile's pnpm-workspace.yaml — quoting it, because it contains @ and :. The file already exists, so append rather than replace:

allowBuilds:
  '@zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>': true

Then re-run add. The key includes a commit hash, so it changes whenever the branch moves. Pin a commit (github:zzdream67/dsh-vision-bridge#<sha>) to keep it stable, or install from npm and skip this entirely.

From a tarball

npm pack                                    # in this repo
dsh plugin --profile web add ./zzdream67-dsh-vision-bridge-0.1.0.tgz

Uninstall

dsh plugin --profile web remove @zzdream67/dsh-vision-bridge

The dependency and its config layer are both removed. If manageDeclarations was on, the plugin withdraws the declarations it wrote as it unloads.

Withdrawal restores the exact prior value. Each declaration records what the row's input looked like beforehand, because after removing image a leftover ['text'] is indistinguishable from a field that was never there. A rule-based revert would delete a line you wrote by hand; the recorded value prevents that.

The record is a note about a row that existed when the note was written, and the file belongs to you. If you delete the model, delete the whole route, or hand-edit the row before withdrawal, the plugin declines rather than reconstructs:

You did this	Withdrawal does
Deleted the model	Skips it; a deleted model is never recreated from a stale note
Deleted the whole route	Skips it, and still cleans up every other route
Hand-edited input into something malformed	Leaves it for you to fix
Nothing; the row is as the plugin left it	Restores the exact prior value

One missing target does not abandon the rest of the withdrawal. Recreating a row you deliberately removed would be worse than leaving a declaration on a row that no longer exists, so missing rows are skipped.

The inline-comment alignment is the one thing that does not come back; see Development for the details.

---

Configuration

The plugin ships a browser-side bundle (dist/client.js) that registers its own section into the Web settings page's settings.section slot. Every field is editable there, applies live, and needs no restart.

Three buttons in that page are worth knowing about:

• Test and enable sends one 16×16 PNG to the selected vision model. The declaration is written only if the endpoint accepts the image, and is withdrawn again on failure. Capability is tested rather than assumed, because an OpenAI-compatible /v1/models reply carries no modality information.
• Enable directly skips the test and writes the declaration. Use it when you already know the model can see.
• Reload re-reads the configuration from disk. The desktop client has no reload gesture, so this is the only way to pick up a change made in the Models page, in another window, or by hand. It asks first if you have unsaved edits.

Host-side schema registration alone does not render a form. The section exists because this plugin authors that browser bundle by hand; the build preset for DSH's client format is not published, but the format itself is plain and stable.

Why a declaration is required first

The admission gate (dsh-host-apiproxy) checks the model's declaration before the image enters the session:

if (modelInfo.inputModalities !== undefined && !modelInfo.inputModalities.includes('image'))
  return err(request, { details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' } })

A text-only model must declare input: [text, image], or the image never arrives and this plugin's conversion never runs.

manageDeclarations (default: on) writes that declaration for each model in bridge and withdraws it when you turn the switch off, disable the plugin, or unload it. With the plugin loaded and the model bridged, the route genuinely does accept image input, because conversion happens before the provider is called.

input: [text, image] is an admission label. Every place the host reads it is guarded by "does this request contain an image", so a text-only request behaves identically whether or not the declaration is present. It does not affect token accounting, context window, sampling, or model behaviour.

Declaring a model without bridging it moves the failure later: the host admits the image and the provider then rejects it. Declarations therefore follow the bridge list rather than being applied broadly.

Fields

Field	Default	Meaning
enabled	true	Master switch. When off the plugin stays loaded but intervenes in nothing and withdraws its declarations, like uninstalling without touching node_modules
visionProvider	''	Vision route id. Empty leaves the plugin inert (see Behavior details)
visionModel	''	Vision model id in that route. Empty leaves the plugin inert (see Behavior details)
prompt	see below	Caption instruction sent with each image
bridge	[]	Text-only models to bridge
manageDeclarations	true	Let the plugin write/withdraw modality declarations
cacheSize	64	Cached transcriptions; 0 disables caching
timeoutMs	120000	Per-image caption timeout
verbose	false	One log line per conversion (never logs transcription text)

Or in cordis.patch.yml:

- id: zz-vision-bridge
  name: '@zzdream67/dsh-vision-bridge'
  config:
    visionProvider: my-local-route
    visionModel: my-vision-model
    bridge:
      - provider: my-text-route
        model: my-text-model
    manageDeclarations: true

A model not listed in bridge is untouched and keeps the host's exact behavior.

Models under the built-in deepseek-official route cannot be bridged: its modality is hardcoded in the plugin that provides it, and its adapter refuses image content. To bridge DeepSeek, add a custom model provider with a route using openai-completions at https://api.deepseek.com/v1, then bridge the models under that route.

The default prompt

It asks for transcription, not interpretation: transcribe all visible text faithfully, describe layout and structure, never speculate or fill gaps, and state plainly what is illegible. The reasoning belongs to the model consuming the transcription; a vision model that editorializes costs the caller information it can never recover.

---

Behavior details

Situation	Behavior
No vision backend configured	The plugin is inert: it loads, does nothing, and warns once when an image arrives
Target model not in bridge	Passes through with zero impact
The request is the plugin's own caption call	Skipped (otherwise it would recurse forever)
No image in the request	Fast path, skipped
Same image appears repeatedly	Resolved sequentially, hits the cache, captioned once
Caption fails or times out	Injects a failure notice telling the model not to pretend it saw the image
The request object (always deep-frozen)	Not mutated. A fresh request carrying the rewritten messages is issued through ctx.llm.stream()

Caption calls go through ctx.llm.stream(), so the vision route's credentials, retry policy, attribution headers, and observability all still apply. The plugin issues no HTTP requests of its own.

---

Before you install

The model receives words, not pixels.

Works	Does not work
Reading errors, code, logs in a screenshot	Comparing two images precisely
Describing UI layout and button positions	Picking colors, measuring pixels
Transcribing tables and chart trends	Fine spatial or geometric reasoning
Reading handwriting and printed text	Judgments resting on subtle visual detail

Every substitution is labeled as a transcription, so the model does not treat it as having seen the image itself.

You need your own vision model. The plugin ships no vision service. It uses a vision model you configure: a local LM Studio / Ollama instance, or a remote service you hold a key for. A local model costs nothing and sends nothing off your machine; a remote one bills your own account.

The plugin writes to settings.yaml. manageDeclarations writes input: [text, image] into the llm-pi-ai section for each bridged model and withdraws it when you turn the switch off, disable the plugin, or unload it. The switch is on by default, because the declaration is what makes the feature work. What is preserved and what is not is detailed in Development. Back up settings.yaml before enabling it the first time.

---

Development

npm install
npm run build      # tsc -> dist/
npm test           # node --test, 68 cases
npm run typecheck

src/rewrite.ts is pure and I/O-free, so the rewriting logic is fully testable without a live model. test/uninstall.test.mjs asserts the declare/withdraw round trip is byte-identical for an absent field, a hand-written [ text ], unusual modality lists, and untouched sibling rows.

Why llm/stream instead of a new route

The common approach registers a twin route (e.g. xxx (vision)) with hardcoded modalities. This plugin does not:

• It registers no route and no adapter, so the host's model registry keeps telling the truth and no extra entry appears in the model picker.
• It never competes in the registry, so none of the defensive machinery against "another plugin rebuilt the registry" is needed.
• It leaves nothing behind: ctx.on() is an effect and is cleaned up automatically, and declaration writes are reverted explicitly.

Deep rewriting

An image block is legal at any depth, including nested inside a tool-result. The built-in read_image tool records an image in its result, so a session that once used it carries nested image blocks on every later turn.

A top-level-only rewrite leaves those in place, and the text-only adapter then fails on every subsequent turn, not just the one that uploaded an image. This plugin recurses through all depths.

Prompt-injection guard

Text recovered from an image is attacker-controlled, exactly like a web page: a screenshot can say "ignore all previous instructions".

Every transcription carries an explicit notice that image text is untrusted evidence and must not be executed as an instruction. The notice precedes the transcription and the text is fenced, so content inside the fence cannot pass itself off as surrounding narration.

Writing declarations to settings.yaml

manageDeclarations writes input: [text, image] into the llm-pi-ai settings section and withdraws it when you turn the switch off, disable the plugin, or unload it. Measured against the real dsh-settings-file, which patches the YAML AST minimally:

Preserved:

• Header, section, and inline comment text
• Other routes and other namespaces
• Existing inline style (e.g. input: [ text, image ])
• Blank lines and overall structure

One real side effect:

- displayName: LM Studio Local     # hand-aligned comment
+ displayName: LM Studio Local # hand-aligned comment

Hand-aligned padding before an inline comment collapses to a single space and does not come back on revert.

The declaration restates text explicitly:

input: [ text, image ]

pi-ai maps this array straight onto inputModalities. Writing [image] alone would declare a model that accepts images but not text, so the write restates text.

Verified facts

All of the following are measured against the real DSH runtime:

• The llm/stream listener fires, but the request arrives deep-frozen: dsh-agent-loop.buildRequest wraps it in deepFreeze, so options.messages = x throws TypeError. Replacing the argument slot does not help either, because cordis' inner callback closes over the original object; neither arguments[0] = … nor next(replacement) reaches the adapter. The only working seam is to skip next() and issue a fresh ctx.llm.stream({ ...options, messages: rewritten }), marked as the plugin's own request to prevent re-entry.
• The adapter receives no image block at all, including images nested inside tool-result.
• A nested ctx.llm.stream() does re-enter the listener; a WeakSet guards against that.
• Declaring [text, image] makes the admission gate admit the image.
• An image block carries an attachment reference, not inline bytes. pi-ai resolves it via attachments.readImage(block.attachment), and a block holding { data, mediaType } is silently dropped from the payload.
• Every terminal outcome, including an adapter refusal, arrives as { type: 'finish', reason: { kind: 'error' | 'aborted', failure } }. There is no type: 'error' chunk and no finishReason field.
• Cross-namespace settings writes carry no ownership restriction, but path addressing does not support array indices, so a row edit must write the whole array back.
• The real dsh-settings-file preserves comments on disk, with the one formatting side effect described above.

---

License

MIT © ZZ Dream (zzdream67)

Acknowledgements

The public source of the following projects informed several design considerations in this plugin:

• dsh-vision-router (MIT)
• @anionex/dsh-vision-toolkit
• Anionex/agent-vision-toolkit

This plugin is an independent implementation taking a different architectural path (llm/stream waterfall interception, registering neither route nor adapter).