IntelligentEyes

Universal Agent Vision Gateway for text-only agents. One stable tool, vision_analyze, turns any text-only model into an agent that can answer questions about images — the pixels are resolved out-of-band by a vision model you choose, and the answer comes back as structured, untrusted-by-default evidence.

中文文档

Why

Text-only agents hit two hard walls when a user attaches an image:

The admission gate. Hosts reject image input before it ever reaches the session when the selected model declares inputModalities: ['text'].
The wire. Even when images get in, OpenAI-compatible backends of text models reject inlined image parts (unknown variant 'image_url') — the request dies.

IntelligentEyes routes around both walls without touching host source code:

a vision bridge adapter registers its own provider route that declares image input, so images are admitted into the session;
the adapter serializes every image block as an id-carrying placeholder ([image attachment id=sha256:… 700x1483 image/png] → call vision_analyze…), so the text model learns the attachment id instead of the pixels;
the model calls vision_analyze with that id; the plugin resolves the attachment, runs the vision pipeline against the VLM you configured, and returns intelligenteyes.vision.v1 evidence the model can ground its answer in.

Architecture

flowchart LR
    U["👤 User attaches an image<br/>(DSH web chat)"] --> G["Admission gate<br/>(host checks model modalities)"]
    G -- "bridge route declares<br/>text + image" --> S[("Session log<br/>image attachment + id")]
    B["VisionBridgeAdapter<br/>provider: deepseek-vision-bridge"] -- "image block →<br/>id-carrying placeholder text" --> M["🤖 Text-only DeepSeek model<br/>(deepseek-v4-flash …)"]
    M -- "tool call with attachment id" --> V["vision_analyze"]
    V --> P["VisionService pipeline<br/>resolve → prepare/tiles → OCR? → VLM → normalize → cache"]
    P -- "HTTPS" --> VL["👁 Vision model of your choice<br/>(mimo-v2.5, gpt-4.1-mini, qwen2.5-vl, …)"]
    VL --> E["intelligenteyes.vision.v1 evidence<br/>observations · answer · uncertainties"]
    E --> M
    M --> A["✅ Answer grounded in visual evidence"]

The same vision_analyze surface is exposed through five interfaces — DSH plugin (native tool), MCP server (stdio + Streamable HTTP), HTTP API, CLI, and TypeScript SDK — so any agent runtime can use it.

Evidence contract

Every analysis returns one stable, versioned shape:

{
  "schemaVersion": "intelligenteyes.vision.v1",
  "images": [{ "id": "img_1", "sha256": "…", "width": 700, "height": 1483 }],
  "task": { "instruction": "Extract the exact error", "mode": "ui", "detail": "high" },
  "summary": "An application error dialog…",
  "observations": [
    { "id": "obs_1", "kind": "text", "text": "ModuleNotFoundError: No module named torch",
      "imageId": "img_1", "confidence": "high", "source": "vlm" }
  ],
  "uncertainties": [],
  "answer": "The exact error is …",
  "backend": { "id": "openai-compatible", "model": "mimo-v2.5" },
  "cache": { "hit": false },
  "trust": "untrusted_visual_content"
}

Visual content is never trusted: text inside images is treated as data, model-emitted confidences are normalized onto a fixed enum, and the whole evidence is stamped untrusted_visual_content.

Install as a DSH plugin (web UI image chat)

One command into any profile:

dsh plugin --profile web add @intelligenteyes/dsh   # from npm
dsh plugin --profile web add ./intelligenteyes-dsh-1.0.0.tgz  # from a tarball

A fresh install boots safely on the deterministic mock backend — the plugin never takes the host down with an unusable config. Then open /intelligenteyes-config-ui in the DSH web UI:

Field	Meaning
视觉后端 (vision backend)	`OpenAI Compatible` / `vLLM` / `Ollama` / `Mock`
识图模型名称	any vision model id, e.g. `mimo-v2.5`, `gpt-4.1-mini`, `qwen2.5-vl:7b`
API Base URL / API Key	endpoint + key of the vision backend (key stored as a secret, never echoed)
OCR 增强	optional local Tesseract assist (lazy — install `tesseract.js` only if you enable it)
主模型桥接 (bridge)	dropdown: Auto (inherit your first OpenAI-compatible pi-ai provider), a named provider (`opencodego`, `deepseek-official`, …), or a custom endpoint

Select a model from the DeepSeek（可识图） group once (or set agent-default-model to it) and attach images in chat — the model unlocks and calls vision_analyze on its own; results are hot-reloaded without restarting the instance.

Preset note: with the full standard tool catalog the tool is visible immediately. Minimal/bootstrap presets that hide tools behind a discovery tool (e.g. dev_tool_search) should add vision_analyze to their resident set for reliable image turns; the placeholder text carries the unlock instruction either way.

Other interfaces

# CLI
intelligenteyes analyze photo.png --prompt "What is in this image?" --mode describe
intelligenteyes doctor          # environment self-check

# Environment for a real backend
INTELLIGENTEYES_BACKEND=openai-compatible \
INTELLIGENTEYES_OPENAI_BASE_URL=https://api.example.com/v1 \
INTELLIGENTEYES_OPENAI_MODEL=mimo-v2.5 \
INTELLIGENTEYES_OPENAI_API_KEY=… intelligenteyes analyze ui.png --mode ui

// SDK
import { createServiceFromEnv } from '@intelligenteyes/sdk'
const evidence = await createServiceFromEnv().analyze({
  images: [{ type: 'path', path: 'screen.png' }],
  instruction: 'Find the exact failure and its location',
  mode: 'ui',
})

HTTP: POST /v1/vision/analyze plus /health, /ready
MCP: stdio by default; intelligenteyes-mcp --http for Streamable HTTP

Image refs support sandboxed paths, hardened URLs (SSRF-checked), host attachment ids, MCP resource URIs, and selected PDF pages. Formats: PNG, JPEG, WebP, first GIF frame, BMP, TIFF, PDF.

Security model

Untrusted visual content — text in images is data, never instructions (prompt-injection test suite included)
Path sandbox — file refs resolve only inside configured roots
SSRF guards — URL refs are checked before fetch
Secret handling — API keys stored via the host credentials seam, never rendered back
Schema-strict evidence — model output is normalized and validated; malformed fields never crash a turn

Development

pnpm install && pnpm build
pnpm test          # 46 tests: unit · integration · security · e2e
pnpm verify        # build + typecheck + lint + tests + benchmark + doctor + pack check
pnpm dsh:smoke     # live smoke against a deepseek-harness checkout (DSH_REPO=…)

Workspace packages: schema, core, four VLM adapters (openai-compatible, vllm, ollama, mock), two OCR adapters (none, tesseract), sdk, cli, http, mcp, dsh — each independently buildable and publishable.

Verification matrix

Host	Platform	Install	Result
source checkout `0.1.0-rc.5`	Windows	`link:`	✅ end-to-end image turn
fresh `DSH_HOME`, `0.1.0-rc.5`	Windows	tarball, one command	✅ install → config → image turn
published `@deepseek-ai/dsh@0.1.0-rc.6`	Ubuntu 24.04 (WSL2)	tarball, one command	✅ install → config → image turn

License

MIT

intelligenteyes

Установка

README