dsh-sidebar-browser-control

dsh-sidebar-browser-control

Let the agent drive DSH's own built-in browser — the Browser tab in the right-hand Sidebar — instead of launching a second Chromium.

中文说明 →

you › open the docs page in the built-in browser and tell me what the API section says
agent › browser_control { action: "open", url: "https://example.com/docs" }
        browser_page   { action: "text" }

Why not a CDP/Playwright plugin

There is already a DSH plugin that drives browsers over CDP (dsh-browser-control). This one is the in-app counterpart, and the differences are the point:

	This plugin	CDP / Playwright wrapper
What it drives	the Browser tab DSH already ships	a separate browser process
Where you see it	in the Sidebar, next to your session	in another window
Extra runtime	none	a browser + driver
Logins	the workspace partition the tab already uses	whatever you set up
Reach	that tab only	any page, headless, screenshots

Use this one when you want the agent to look at something with you, in the place you are already working. Use a CDP wrapper when you need headless automation, downloads or screenshots.

What it can do

Tool	Actions
browser_tabs	list open built-in-browser tabs: id, live title, URL, owning session, panel state, whether page control is available, plus capture diagnostics
browser_control	open · navigate · back · forward · reload · close · focus · expand · collapse · fullscreen · split · float · dock · external
browser_page	text · links · elements · eval · click · type · fill · key · login · scroll · wait · images · image · screenshot (page actions: Desktop only)
/browser	open <url> · tabs · back · forward · reload · close · focus · expand · collapse · fullscreen · split · float · dock · external · read · images · image <N> · screenshot

The tools and the slash command share one implementation, so /browser open … costs no model round trip and behaves exactly like the tool.

Operating the page, not just reading it

• elements reports what a person sees: type, name, label, placeholder, autocomplete, value, checked, plus a selector for click.
• fill takes several fields at once and matches them by label or placeholder, not only by CSS selector — [{ label: "用户名", value: "alice" }, { label: "密码", value: "…" }, { label: "记住我", value: true }] — and can submit. Passwords are masked in every result.
• login is the whole flow in one call: find the account and password fields (also inside a same-origin iframe), fill, submit, wait, then report which fields were found, whether the password field disappeared, the address it landed on and an excerpt of the page. It never echoes the password back.
• key presses Enter/Tab/Escape; Enter submits the surrounding form.
• wait can wait for a selector, a text, or url_contains.

Seeing the page

Two ways to give the model actual pixels:

• images lists the pictures in the viewport; image { index } attaches one of them, so the model can look at a photo, chart, QR code or captcha. HTTP(S) sources are fetched by the Host (the page's own fetch is bound by CORS); data:, blob: and <canvas> sources are decoded inside the page.
• screenshot captures the Browser pane itself and attaches it. It deliberately avoids every native capture API — webview.capturePage() crashed the renderer (see below) — and instead reads the pane's geometry from the page, then captures the DSH window with PrintWindow through PowerShell and crops to that rectangle. Because it renders the window's own pixels, it also works while other applications cover the window. A minimized window has no rendered surface, so the tool refuses with that reason — pass restore: true to let it bring the window back first. This path is Windows-only today; elsewhere the tool says so instead of failing quietly.

Install

The plugin is a normal DSH bundle, so any spec the plugin manager accepts works.

From npm (published as dsh-sidebar-browser-control). Ask the agent, no restart needed:

plugin_manager { action: install_bundle, target: "dsh-sidebar-browser-control" }

or with the CLI:

dsh plugin --profile desktop add dsh-sidebar-browser-control

From GitHub, if you prefer tracking main:

plugin_manager { action: install_bundle, target: "github:XUFENGCHINA/dsh-sidebar-browser-control" }

From a checkout, for development:

dsh plugin --profile desktop add link:/abs/path/to/checkout

Remove it with plugin_manager { action: "remove_bundle", target: "dsh-sidebar-browser-control" }.

After editing client.js in a checkout, ask the Host to re-scan the bundle (set_bundle off, then on) — the client artifact is served from a snapshot.

Use

Just ask, in any session whose Sidebar is on screen:

• "open https://news.ycombinator.com in the built-in browser and summarise the top 5"
• "in the built-in browser, click the Sign in link and tell me what the form asks for"
• "search Bing in the built-in browser for DeepSeek Harness and give me the first three results"
• /browser open https://example.com — straight from the composer, no model call

Two behaviours worth knowing while driving it:

• A link that opens a new window becomes a new sidebar tab — DSH turns the popup request into a tab of its own. browser_page click reports it as opened_tab/opened_url; continue with that id, because the original tab stays where it was.
• A page that fails to load says so — DNS, network, certificate and bot-check failures leave Chromium's error page in the tab, and page actions report that explicitly instead of returning empty content that reads as "the page is empty".

How it works

The browser lives in the Web renderer, tools live in the Host, and DSH exposes no supported Host → Client call to third-party plugins. So the plugin is two halves joined by four loopback routes (a long-poll carries Host → Client commands, with no timer in either direction), and all layout/navigation goes through ctx.sidebarRight so the shipped address bar, history and restore logic stay authoritative.

Full protocol, design decisions and the DSH internals it depends on: docs/architecture.md.

Security

browser_page eval runs arbitrary JavaScript inside the browser tab, with that page's cookies — if you are logged in there, the agent can act as you in that tab. Tool calls are logged with their arguments, but treat these tools like letting someone read your screen. The four routes admit only the DSH application origin (no cross-site CSRF) and only this plugin's own tools can enqueue a command. Details: SECURITY.md.

Compatibility

Verified end to end on DSH Desktop 0.2.0-rc.2 / Electron 44 / Windows x64: list, open, navigate, back/forward/reload, read text, eval, list elements, click, type + submit, float/dock, close, expand/collapse. The Web profile gets tab and layout control (its sandboxed iframe cannot be read).

The plugin talks to public services (ctx.sidebarRight, ctx.webServer, ctx.tools, ctx.commands) plus a few DOM markers owned by the shipped Sidebar packages. If a marker changes, page-level actions fail with an explicit message while tab and layout control keep working; the table of those dependencies is in docs/architecture.md.

Known limitations

• No screenshots through Electron. webview.capturePage() natively crashed the Electron renderer on the verified deployment, so it is not used anywhere. browser_page screenshot goes through PrintWindow instead: the window must not be minimized, and only Windows is implemented (macOS/Linux report that). Evidence and survivable alternatives: docs/architecture.md.
• Page-level actions are Desktop only.
• Tab ids are unique per session; an ambiguous id is reported and accepted as <sessionId>#<tabId>.
• Actions are selector-based — no coordinate clicks, uploads or downloads — and they target the session currently on screen (ctx.sidebarRight semantics).

Development

No build step and no dependencies: client.js is the shipped artifact, written by hand in the loader's classic-script format.

npm test        # offline protocol test: stub context + real HTTP + a fake client half
npm run verify  # syntax + protocol test + packaged-tarball check

npm test needs no DSH running, so it works in CI and on a clean checkout. Guidelines and the list of invariants that keep this plugin loadable: CONTRIBUTING.md. Releasing: RELEASING.md.

License

MIT