dsh-plugin-chrome

dsh-plugin-chrome

A DeepSeek Harness browser visualization plugin: opens a real, visible Chrome window per session, lets the agent drive the browser through the chrome_* tool suite, and streams the live view into a Chrome tab in the Web GUI — take over manually at any time.

中文文档

Features

• A visible window per session: every DSH session gets its own Chrome window (real window, not headless). Watch every agent action as it happens; the window uses an isolated user-data-dir, so it never mixes with your daily browser.
• Live view: the Chrome tab in the Web GUI streams the window through Chrome screencast (smooth while pages are active). A screenshot heartbeat keeps idle pages from freezing (about one frame every 3 seconds).
• Complete agent tool suite (16 tools): chrome_open / chrome_status / chrome_close / chrome_navigate / chrome_tabs / chrome_snapshot / chrome_screenshot / chrome_click / chrome_click_at / chrome_fill / chrome_type / chrome_press_key / chrome_hover / chrome_scroll / chrome_evaluate / chrome_wait.
• Accessibility-tree snapshots: chrome_snapshot returns a compact a11y tree with stable element uids; clicks and fills target uids directly — far lighter than DOM dumps and robust against fragile selectors.
• Dual-channel screenshots: chrome_screenshot sends the image into the model context (as an image block) AND saves it to the session's screenshot history shown in the panel.
• Security-minded: CDP never exposes a fixed port; the Web API enforces same-origin checks and a sessionId whitelist; browser data is isolated per session.
• Resource governance: idle windows auto-close (default 10 min, configurable), chrome_close closes explicitly, and plugin unload / host shutdown closes every window it opened.

Install

Prerequisites: DeepSeek Harness (DSH) installed, and Chrome or Edge on the machine.

# Option 1: install from GitHub (recommended)
npx -p @deepseek-ai/dsh dsh plugin --profile web add github:jiaererw/dsh-plugin-chrome

# Option 2: local path (development)
npx -p @deepseek-ai/dsh dsh plugin --profile web add D:/harness/dsh-plugin-chrome

Restart DSH after installing — a Chrome tab appears at the top of every conversation.

Usage

For the agent (tools)

The agent gets the chrome_* suite automatically. Just ask it:

Open Chrome, go to https://example.com, take a screenshot, then click the "Login" button and fill in the username.

The agent will: chrome_open → chrome_navigate → chrome_screenshot (sees the image) → chrome_snapshot (gets uids) → chrome_click / chrome_fill.

For you (visualization)

1. Open the Chrome tab at the top of the conversation:
   • Live view: continuously shows the window. Native screencast frames flow while the page changes; a heartbeat fallback force-captures idle pages so the picture never freezes.
   • Tab management: switch or close tabs from the side list, in sync with the real window.
   • Manual takeover: click around in the Chrome window yourself at any time — the agent sees your changes on its next tool call.
2. Screenshot history: every chrome_screenshot is stored in the panel; click a thumbnail to enlarge.

Window lifecycle & resilience

• Lazy start: Chrome launches only on the first chrome_* call (or the panel's Open button).
• Orphan adoption: if DSH died and left a Chrome behind (profile locked), the plugin reconnects through DevToolsActivePort and takes the window over instead of failing (the same autoConnect idea as chrome-devtools-mcp).
• Idle reaping: a window idle past idleTimeoutMs (default 10 min) closes automatically — never while a Web UI viewer is watching.
• Auto tab recovery: every operation makes sure a usable tab exists, so a window full of chrome:// internal pages never dead-ends.

Configuration

Override the plugin row in the profile's cordis.patch.yml (config is replaced wholesale):

- id: dsh-plugin-chrome
  config:
    headless: false            # keep false — a visible window is the point
    executablePath: ''         # empty auto-detects Chrome/Edge; or set an absolute path
    idleTimeoutMs: 600000      # idle auto-close (0 disables)
    windowWidth: 1280
    windowHeight: 900
    screencastFrameSkip: 4     # live-view frame decimation (1 = smoothest)
    screencastQuality: 70      # JPEG quality 1-100
    maxSnapshotText: 60000     # max chars per snapshot
    maxTabs: 16
    autoScreenshot: false      # capture after every action
    extraArgs: ''              # extra Chrome launch flags

Data directory (browser profiles & screenshots): ~/.dsh/data/dsh-plugin-chrome/sessions/<sessionId>/ (override with dataRoot).

FAQ

• The Chrome tab shows nothing: check the window is running (status dot at the top); the first launch takes a few seconds. Idle pages update roughly every 3 seconds via the heartbeat; activity raises the frame rate automatically.
• Agent says "unknown uid": the page changed — have it re-run chrome_snapshot.
• I closed the window myself: the panel shows "window closed"; any next chrome_* call or the Open button relaunches it.
• Login state: each session uses an isolated profile, so logins don't carry over from your daily browser — that's by design. To log in somewhere, let the agent complete the login (it persists for the session).
• Chrome stays open after DSH is killed: the orphan window is adopted on the next session call (or close it by hand); a clean DSH shutdown closes its windows.

Development

npm install
npm run typecheck   # host + client programs
npm test            # vitest unit tests
npm run test:e2e    # real-Chrome end-to-end smoke (pops a visible window)
npm run build       # lib/index.js (host) + lib/client.js (client bundle)
npm run watch       # continuous build; client changes hot-reload, host changes need a DSH restart

Architecture: the host half (cordis plugin) drives the local Chrome through puppeteer-core, registers the chrome_* tools and the /dsh-chrome/* HTTP/WS API; the client half (browser bundle) registers the Chrome tab on conversation.view and consumes the API and the frame stream. The picture = native Chrome screencast (active pages) + screenshot heartbeat (idle fallback). The control layer borrows proven designs from chrome-devtools-mcp (CDP control, a11y snapshots with uid lookup, wait discipline, autoConnect adoption) and mcp-chrome (screenshot compression, CDP coordinate input, session refcounting).

License

MIT