ben-dsh-crawl4ai

ben-dsh-crawl4ai

A Crawl4AI-backed WebFetchProvider for the DeepSeek Harness web capability seam (ctx.web).

Crawl4AI drives a real browser, so a page that a plain HTTP fetch sees as an empty <div id="app"> arrives here as rendered Markdown — including the code blocks, tables, and lazily-loaded text that only exist after JavaScript runs.

This is an implementation package: it registers a provider into ctx.web and does not register a model-facing tool. @deepseek-ai/dsh-tool-web owns web_fetch — its schema, its prompt guidance, and its result card — so installing this package makes that existing tool render pages through Crawl4AI instead of a plain fetch.

Provider id	ben-crawl4ai
Registers	ctx.web.registerFetchProvider(...)
Configures	on the Plugins page, on this package's ben-crawl4ai row
Requires	a reachable Crawl4AI server (0.9.x tested)

English | 中文

Install

dsh plugin --profile <name> add github:you/ben-dsh-crawl4ai   # or an absolute path

The package declares dsh.bundle, so the install also activates it: the bundle patch inserts the provider row and selects it on the web row.

Developing in place (link: installs need one extra step)

dsh plugin add <absolute path> records the dependency as link:<path>, which makes node_modules/ben-dsh-crawl4ai inside the profile a junction back to your working copy. Node resolves a module's import specifiers from its real path, so the walk up for @deepseek-ai/… starts in your working copy and never reaches the profile's resolution root — and a published plugin, which pnpm materializes inside the profile, does not have this problem.

One command reproduces that reachability:

node scripts/link-harness-deps.mjs

It junctions each package named in peerDependencies from the harness's own install into this directory's node_modules, preferring the profile tier so this plugin shares the harness's module instances rather than copies. Re-run it after changing peerDependencies. It defaults to $DSH_PROFILE_DIR/../node_modules, which the harness sets for its child processes; pass --root <node_modules> otherwise.

Configuration

Everything lives on Plugins → this package → the ben-crawl4ai row → Configure. The page stages what you type and writes only on Save, through the shell's own settings surface (ctx.configForms), so an edit reaches the next web_fetch without a restart.

Field	Meaning
Crawl4AI URL	Server root, e.g. http://crawl4ai.internal:11235. Unset or not http(s) makes the provider report unavailable, and web_fetch says so instead of failing every call.
Auth header name	e.g. Authorization or X-API-Key. Blank sends no authenticating header.
Auth header value	Sent as that header's value. Rendered masked; blank sends no authenticating header — the name and the value take effect together.

Both header fields are independent, so a deployment can use any header name or none at all.

The values are stored in the active profile's patch ($DSH_HOME/profiles/<profile>/cordis.patch.yml), in plain text. This is a fetch back end reading its own token, not a per-user credential, and the masked control keeps it off the screen rather than out of the file. Two further fields stay off the page because they are deployment concerns, edited in YAML instead:

Key	Default	Meaning
timeoutMs	60000	Resource backstop for one crawl. Keep it below the model-facing tool budget (tool-web's timeoutMs) so a slow server surfaces as a provider failure.
maxContentChars	200000	Cap on the Markdown returned for one page; a longer page sets truncated.

A patch row replaces the targeted row's whole config, so restate every key you want to keep.

Provider selection is required

The bundle patch sets web.fetchProvider: ben-crawl4ai, and that is not politeness: @deepseek-ai/dsh-web-fetch-http is mounted alongside this package and its available() is unconditionally true, so two fetch providers are usable at once. The seam auto-selects only when exactly one is, and otherwise fails every call with WEB_PROVIDER_AMBIGUOUS.

A patch replaces the whole config of the row it targets, and a profile patch is applied after every bundle layer — so if you also pin searchProvider on the web row, restate fetchProvider: ben-crawl4ai there too.

The server contract

One fetch is POST <base>/crawl with {"urls": ["<url>"]} and the configured header. The response is Crawl4AI's { success, results: [ … ] }, and the first result supplies the final URL (redirected_url over url), the target's own status_code, and its markdown object, from which fit_markdown is preferred and an empty fit_markdown falls through to raw_markdown (the server returns fit_markdown: "" unless a fit filter was requested).

A server that does not serve /crawl (an older build, or a proxy that exposes only part of the API) answers 404/405 and is retried once against /md, which returns the same content without the target's status or final URL.

A non-2xx target page is a result, not an error — the status is part of the fetched resource, so it is returned with whatever body Crawl4AI extracted. A throw is reserved for failing to retrieve the page at all.

Errors

Failures are WebErrors the tool layer renders in a tool result.

Situation	Code
Caller cancelled	WEB_ABORTED
timeoutMs elapsed	WEB_FETCH_TIMEOUT
Requested URL is not absolute http(s)	WEB_INVALID_URL
No Crawl4AI URL configured	WEB_PROVIDER_ERROR (and available() is false)
Server refused the request (401/403)	WEB_PROVIDER_ERROR, naming the auth header
Server unreachable, 5xx, or a non-JSON body	WEB_PROVIDER_ERROR
Server produced no result or no content	WEB_PROVIDER_ERROR

Tests

node --test test/crawl4ai.test.mjs test/client.test.mjs   # offline

crawl4ai.test.mjs covers the HTTP contract and mapping against a stubbed fetch. client.test.mjs evaluates the browser bundle with the module loader stubbed, so a stray @deepseek-ai/* import, an unknown primitive name, or a slot key that does not match the inserted row fails the suite instead of breaking the Web UI.

Two suites need a live server and skip without one:

$env:CRAWL4AI_URL='https://ca.example:1443'
$env:CRAWL4AI_AUTH_HEADER_NAME='Authorization'
$env:CRAWL4AI_AUTH_HEADER_VALUE='Bearer …'
node --test test/seam.test.mjs          # registers into the REAL ctx.web and fetches
node --test test/crawl4ai.test.mjs      # includes a live crawl

seam.test.mjs is the end-to-end one: it mounts @deepseek-ai/dsh-web, applies this plugin, and dispatches a fetch through the seam, covering selection by id, the unavailable gate, and that a volatile row edit reaches the very next call.

Known limitations

• Markdown only. The seam's body kinds are html and text; this provider always returns text, so dsh-tool-web's HTML-to-Markdown step is skipped rather than applied twice.
• No crawl tuning. crawler_config is not exposed: scan depth, caching, filters, and screenshots are Crawl4AI features this seam's request shape has nowhere to put.
• The Crawl4AI server holds the page cache. cache is left at its default, so a repeat fetch may answer from the server's cache.
• /md loses the target's status. On the fallback path a successful conversion is reported as HTTP 200 by construction.

License

MIT