dsh-vision-bridge

@dsh-extension/dsh-vision-bridge

On-demand vision for text-only DeepSeek Harness (DSH) sessions.

中文文档 · npm

A DSH plugin that gives a text-only DeepSeek session on-demand multimodal capability: the session stays on its text model for every turn, and only when the model actually needs to look at pixels — a screenshot, an uploaded image, a diagram, a chart — does it call the vision_describe tool, which sends only the image(s) + a focused question to an OpenAI-compatible vision model.

• No long context ever reaches the vision model — a 300k-token conversation history is never sent; each vision call is just image + question, keeping cost minimal.
• Session log and UI keep the original images — only the model input is rewritten to text markers.
• Bring your own vision endpoint — any OpenAI-compatible /v1/chat/completions service (OpenAI, DeepSeek, Gemini proxy, local vLLM/One-API, …).

❤️ Sponsors

Want to appear here? — sponsor this project with an API donation.

🎉 Thanks to xiaoyaoapi for donating their API to this project! xiaoyaoapi is an OpenAI-compatible AI API aggregation gateway for developers, built on New API with a unified admin dashboard. It offers unified key management, transparent usage tracking, and multi-channel access to mainstream large models under a single endpoint — letting developers integrate leading LLM services at lower cost and with greater convenience, ready to use as the vision endpoint of this plugin.

How it works

User / tool produces an image ──► image stays in the session and UI
                                   │
                                   ▼  (model input layer)
                              image is rewritten to a text marker
                          (marker carries the attachment id and hints
                           the model to call vision_describe)
                                   │
                                   ▼
   text model calls vision_describe(attachmentIds / paths, question)
                                   │
                                   ▼
   vision model (receives only image + question) → text answer
                                → returned as a normal tool result

• The agent/pre-step hook records every image attachment that appears in the session (user uploads and tool-produced screenshots, including ones nested inside tool-result), building an attachment index that vision_describe uses to resolve bytes by id.
• session.deriveMessages() is wrapped so that no text-model request ever contains image blocks (the native DeepSeek adapter rejects them); images are replaced by text markers. The session event log and UI keep showing the original images.
• DSH's built-in llm-pi-ai builds the OpenAI multimodal request; the plugin only maintains a single vision-bridge provider route and does not re-implement a protocol adapter.

Image admission

DSH Web runs an image-capability check before a message enters the agent, based on the current DeepSeek model. This plugin keeps an admission bypass so images can enter the session first; the marker rewrite then guarantees no text-model request carries image blocks. When the plugin is disabled or uninstalled, the native admission check is restored.

Check GET /_dsh/vision-bridge/settings for the live value.admissionBypass and dependency-service status.

Installation

Install from the npm registry (not a local checkout) — one command:

# if you already have the `dsh` CLI on PATH:
dsh plugin --profile web add @dsh-extension/dsh-vision-bridge

# or, if you have been using npx all along:
npx @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile web add @dsh-extension/dsh-vision-bridge

The --profile flag targets the profile you boot (web is the browser UI profile). Omit it or adapt it if your profile has a different name.

After a new client bundle is added, restart dsh web once so the UI picks it up.

Configuration

Configure it in Settings → Vision Bridge (DSH Web), or edit ~/.dsh/vision-bridge.json:

{
  "enabled": true,
  "baseUrl": "https://api.openai.com/v1",
  "apiKey": "sk-xxxx",
  "apiKeyEnv": "",
  "model": "gpt-5.6-terra"
}

• baseUrl accepts an API root, a .../v1 base, or a full .../chat/completions URL (the plugin normalizes it).
• apiKey and apiKeyEnv are mutually exclusive. A directly entered key is synced to the DSH credential store and referenced as DSH_VISION_BRIDGE_API_KEY.
• The plugin maintains exactly one vision-bridge route inside DSH's llm-pi-ai.providers and never touches other providers.
• enabled: false disables the whole chain: no tool registration, no image rewriting, no admission bypass (native behavior restored).

Precedence (highest wins): Settings page (with schema defaults) → environment variables → config file.

Environment overrides: DSH_VISION_BRIDGE_BASE_URL, DSH_VISION_BRIDGE_API_KEY, DSH_VISION_BRIDGE_API_KEY_ENV, DSH_VISION_BRIDGE_MODEL, DSH_VISION_BRIDGE_ENABLED.

vision_describe tool

• Arguments
  • attachmentIds: image attachment ids from the current conversation (shaped like sha256:...), one or several;
  • paths: absolute local image file paths (png/jpeg/webp/gif) — use either or both, 1–4 images in total;
  • question: required — a focused, specific question about the image(s).
• Behavior: resolves the images → sends image(s) + question to the vision model via the vision-bridge route → returns the text answer as a tool result.
• Multi-image comparison is supported: put several images in the same user message.
• Attachment ids must come from the current conversation (user uploads or tool output); paths go through DSH's sandbox-aware file service.

Verify

npm test

The suite covers: image-marker rewriting (including nested tool-result), both id- and path-based resolution, event-log attachment indexing, full-chain shutdown when disabled, and text-only sessions staying untouched.

Development

From a local checkout:

dsh plugin inject /path/to/dsh-vision-bridge

Search keywords

deepseek · deepseek-harness · dsh · plugin · vision · multimodal · vision-language-model · VLM · image understanding · screenshot · OCR · image analysis · OpenAI-compatible · text-only-llm · on-demand vision · LLM agent

License

MIT