dsh-guide-dog

Guide Dog for DSH, powered by MiniMax

English | 简体中文

A dynamic Cordis plugin that gives DeepSeek Harness multimodal superpowers through the mmx CLI (MiniMax):

• Eyes for DeepSeek — MiniMax VLM (guide_dog_vision / guide_dog_inspect) describes images, so a model with no native vision input (e.g. DeepSeek) can still review frontend designs, figures, screenshots, and generated images.
• Hands for generation — images (image-01), video (MiniMax-H3 / Hailuo), speech (MiniMax TTS), music (music-3.0), text (MiniMax-M3), and web search.
• Web UI preview & playback — every generated file is served same-origin at /guide-dog/media/<file> and rendered inline in the conversation tool cards (<img>, <audio controls>, <video controls>), plus a Guide Dog settings page with auth status and a speak tester.
• Skill integration — guide_dog_speak reuses your existing audio-conversation and speech-mmx skill pipelines (text transform, CJK auto-detect, per-language voices, host playback), and falls back to raw mmx speech synthesize when the skill scripts are absent.
• Automatic invocation — a mounted system-prompt section (guide-dog-vision, order 110) tells the agent to auto-invoke the inspection tools for any job needing visual checks, especially when the active model cannot see images.
• Call mode (Phase 2, shipped) — hands-free, real-time voice conversation in the web UI: VAD / push-to-talk turn-taking, streaming sentence-level TTS with barge-in, consensus-first protection for write commands, progress announcements, and a unified floating dual-pill UI at the composer with zh/en i18n (details in "Phase 2 — call mode" below).
• Accessibility mode (Phase 3, planned) — an a11y config block is reserved (auto-narration, vision-cloud, summary-first); accessibility features are next on the roadmap and will be tested and rolled out after the call-mode shakeout.

Featured

Guide Dog is featured in dsh-recommend, a community-curated plugin directory for DSH. It passed the project's certification review and carries the gold certified badge at the top of this README; the score badge updates automatically on every registry sync.

Files

File	Purpose
plugin-host.js	Host half — source of record (tools, RPC, media route, prompt section, voice mode)
plugin-client.js	Client half — source of record (tool cards, settings page, voice cluster)
bundle/	Static web-profile bundle generated from the two halves (deploy/convert_bundle.py)
deploy/	convert_bundle.py (source → bundle) and publish.py (bundle → ~/.dsh/dsh-guide-dog + web profile registration)
package.json	Root manifest declaring the dsh.bundle (makes the repo installable via dsh plugin add)
README.md	This file
README.zh-CN.md	Simplified-Chinese version of this README

Install

Install straight from this repository with the standard DSH plugin command — the root package.json declares the dsh.bundle manifest, so the profile's pnpm layer resolves the package itself (no npm publish required):

dsh plugin --profile web add github:AtropinolTT/dsh-guide-dog

Deploy (static web-profile bundle — current)

1. Edit the source of record: plugin-host.js / plugin-client.js.
2. python3 deploy/convert_bundle.py — regenerate bundle/lib/.
3. python3 deploy/publish.py — copy to ~/.dsh/dsh-guide-dog, idempotently register in ~/.dsh/profiles/web (dependency link + bundles entry + node_modules symlink), remove the superseded autoload bundle.
4. Restart DSH (dsh web) — bundles are parsed at startup.

No dynamic plugin, no approval cards, no per-session instances: after a DSH restart the tools and voice UI come back with the profile itself. Full details and pitfalls in the "Restart recovery" section below.

plugin-source.js is a legacy dynamic-era artifact (both halves concatenated); kept for reference, not used by the current deploy flow.

Tools

Tool	Args	Returns
guide_dog_speak	text*, voice (auto), speed, language, playOnHost	{ok, url, voice, bytes} mp3
guide_dog_image	prompt*, aspectRatio, n, width, height, seed, promptOptimizer, watermark	{ok, urls[], files[]}
guide_dog_video	prompt*, model (MiniMax-H3 default), image, subjectImage, duration, ratio	{ok, url, taskId} mp4 (polls until done)
guide_dog_vision	image*, prompt	{ok, answer} VLM description
guide_dog_inspect	image*, focus (general/frontend/figure/screenshot/ocr), prompt	{ok, answer, focus} structured review
guide_dog_voices	language	{ok, voices[]}
guide_dog_music	prompt*, lyrics, instrumental, vocals, genre, mood, model	{ok, url} mp3
guide_dog_text	message*, system, model, maxTokens, temperature	{ok, text}
guide_dog_search	q*	{ok, results[]} (max 10)

* required

Auto-invoke contract (visual checks)

While the plugin runs, a system-prompt section instructs the agent:

• For visual checks (frontend design review, figure/plot/chart generation, screenshots, UI mockups, generated-image QA) it MUST call guide_dog_inspect (structured) or guide_dog_vision (general) on the produced image file before finalizing — never claim to have seen an image it has not inspected.
• Generated media is served to the user at /guide-dog/media/<file>; the agent must include the returned url fields so the user can preview.
• Speech requests route to guide_dog_speak.

Example visual-check flow on DeepSeek:

1. (agent) create figure/screenshot file, e.g. chart.png
2. (agent) guide_dog_inspect { image: "chart.png", focus: "figure" }
          → structured review of axes/labels/readability/encoding
3. (agent) iterate the figure, re-inspect, then finalize with the url
4. (user)   previews chart.png in the web UI card

Media store & serving

• Media lives in ~/.dsh/guide-dog/.guide-dog/media — the global store under GLOBAL_ROOT = ~/.dsh/guide-dog (one instance for the whole web profile since 2026-08-16; no longer the per-workspace sandbox root — see "Restart recovery" below).
• Served by a same-origin prefix route /guide-dog/media with:
  • extension allowlist (jpg/jpeg/png/gif/webp/mp3/wav/m4a/ogg/mp4/webm),
  • basename-only lookup + traversal guard,
  • Accept-Ranges: bytes with real byte-range responses (video seeking),
  • 404/405/413/416 as appropriate.
• .index.json keeps metadata (prompt, voice, ts, kind) for the settings gallery (guide-dog/list-media RPC). A corrupt index is rebuilt from the directory.
• Files persist across plugin restarts; stopping/removing the plugin only removes the runtime registrations, never the files.

Skill integration (audio-conversation / speech-mmx)

guide_dog_speak honors the exact pipeline of your two skills:

1. ~/.agents/skills/audio-conversation/scripts/transform.py — markdown/code/URL stripping (falls back to a built-in JS transform when absent).
2. CJK auto-detect → per-language voice defaults (English_Trustworthy_Man / Chinese (Mandarin)_Gentle_Youth), same as the skill env contract. Explicit voice overrides; language boosts accents.
3. ~/.agents/skills/speech-mmx/scripts/mmx_tts.py speak --input … --out … (falls back to mmx speech synthesize).
4. Browser playback via the returned mp3 URL. With playOnHost: true the host speakers play it too — one file at a time (previous playback is terminated first), mirroring the skill's latest-only rule.

Env vars of the skills that still apply when set in the dsh process environment: AUDIO_CONVERSATION_VOICE(_EN/_ZH), AUDIO_CONVERSATION_SPEED, AUDIO_CONVERSATION_DIR, AUDIO_CONVERSATION_NO_PLAY, AUDIO_CONVERSATION_KEEP_FILES, TTS_GEN. Turn files keep the turn-NNN.mp3 naming convention.

Settings page

Settings → Guide Dog (id guide-dog):

• Auth — mmx auth status result with the key masked (sk-c…xxxx); never logged in full.
• Voice mode — global default on/off radios (per-session override lives on the small speaker button at the input's bottom-left).
• Voice input — STT engine select (whisper / sherpa / minimax), recognition language (auto/zh/en), input device select (defaults to the system default), and auto-send-after-recognition checkbox.
• STT — faster-whisper availability + version/python, and the whisper model select (base/small).
• Speak tester — text + voice selector (from guide-dog/voices), plays the mp3 in the browser.

Phase 1 — voice mode & voice input

Feature list

• Voice mode (host event-driven) — a host session/event listener watches assistant/message events, extracts the reply text (event.data.content blocks with type === 'text'), checks whether voice mode is effective for that session (session override else global default), and enqueues the TTS result ({url, key}) or error into a per-session voiceQueue. The client polls the queue every second and plays it with a module-level Audio object, or shows a bottom-right toast + beep for 6s.
• Voice cluster — conversation.input.left entry guide-dog-voice (order 30) at the input box's bottom-left, themed with DSH tokens (--dsw-alias-*), inheriting the app font:
  • small speaker icon — click toggles the per-session voice-mode override (guide-dog/set-config with voiceMode.sessions); hover tooltip shows "Voice mode: on/off · Global default: on/off".
  • language dropdown — recognition language detection (auto/zh/en).
  • mic icon — record → transcribe → insert (feather-style SVG; recording state pulses red with a second counter).
• Session-scoped playback — playback runs on a module-level Audio object, so switching sessions never replays or interrupts it: the current clip plays to the end unless a new playback task (a fresh queue entry from any session) overrides it.
• Mic voice input — the mic in the cluster: MediaRecorder with 1s timeslices, live second counter, maxSeconds auto-stop, language from the dropdown, and transcribe via guide-dog/transcribe. Recognized text is inserted into the input box with inputActions.setDraft(text) (auto-send via inputActions.submit() when configured). Error states: mic_denied, no_device, empty_speech, stt_failed, stt_timeout, engine_unavailable, insert_failed (never silent).
• Recorder page — sandboxed clients that cannot record in-page get a 🎙 Open recorder page link to the standalone /guide-dog/recorder page (GET serves a self-contained HTML recorder; POST /guide-dog/transcribe-upload accepts raw audio/webm, 20 MB cap, and runs the same transcribeImpl).
• Settings controls — the Phase 1 config blocks above, backed by guide-dog/get-config / guide-dog/set-config / guide-dog/status.

config.json schema

Lives at ~/.dsh/guide-dog/.guide-dog/config.json (auto-created from defaults; all keys optional, deep-merged over the defaults):

{
  "voiceMode": { "default": false, "sessions": { "<sessionId>": true } },
  "voiceInput": {
    "autoSend": false,
    "engine": "whisper",
    "language": "auto",
    "maxSeconds": 60,
    "whisper": { "python": "python3", "model": "small" }
  },
  "tts": {
    "voiceEn": "English_expressive_narrator",
    "voiceZh": "Chinese (Mandarin)_Gentle_Youth",
    "speed": 0.95,
    "format": "mp3"
  }
}

• voiceMode.sessions maps a session id to a boolean override; default is the fallback. The speaker button at the input's bottom-left toggles the current session's override.
• voiceInput.engine: whisper (only engine implemented; sherpa/minimax are reserved — selecting them returns engine_unavailable).
• voiceInput.maxSeconds forces the mic recording to stop.

STT engine (faster-whisper)

The whisper engine shells out to a bundled Python script (.guide-dog/scripts/whisper_transcribe.py) using faster-whisper:

pip install faster-whisper        # needs Python 3.8+; installs torch cpu wheels
python3 -c "import faster_whisper; print(faster_whisper.__version__)"

The host probes availability at startup and writes the result to .guide-dog/status.json (whisperAvailable, whisperVersion, whisperPython), shown in the Settings → STT row. Model choices: base (fast) / small (accurate); first run downloads the model weights.

Verification

node --check plugin-host.js && node --check plugin-client.js          # syntax
curl -s http://127.0.0.1:3080/guide-dog/recorder | head -5             # recorder page serves HTML
curl -s -X POST http://127.0.0.1:3080/guide-dog/api/guide-dog/status \
  -H 'content-type: application/json' -d '{}' | head -5               # status RPC (compat layer)
cat ~/.dsh/guide-dog/.guide-dog/status.json                            # whisper probe result

Manual checks (after deploy): click the speaker button (voice mode on, turns green) → send a message → the assistant reply is spoken automatically; switch sessions mid-playback → the clip continues to the end and is NOT replayed; use the mic button → recognized text appears in the input box; Settings → Guide Dog shows the Voice mode / Voice input / STT blocks.

Phase 2 — call mode

Feature list

• WebSocket-free dual channel — uplink is a whole-clip POST /guide-dog/call-transcribe (webm/opus, ≤20MB, reuses the Phase 1 transcribeImpl and the local whisper pipeline) → {ok, text, language, durationMs}; downlink is GET /guide-dog/tts-stream over a chunked HTTP stream (the host spawns mmx speech synthesize --stream --format pcm --sample-rate 24000 per sentence and pipes stdout incrementally into res.write; the client reads the stream with fetch().body.getReader() → PCM→WAV → seamless Web Audio scheduling). No new WebSocket protocol surface on the transport layer; browser and CLI reuse the same pipeline.
• Automatic VAD + push-to-talk (PTT) — default VAD (call.mode='vad'): MediaRecorder (audio/webm;codecs=opus, 250ms timeslice, continuous recording) + a parallel AnalyserNode energy detector (RMS ≥ vad.threshold; silence for vad.silenceMs ends an utterance; vad.minSpeechMs minimum speech; vad.maxSegmentSeconds per-segment cap) — speak-pause-speak automatically becomes two turns; the panel can switch to ptt push-to-talk (hold the mic to talk, release to send; VAD parameters do not participate in endpointing, only in interruption monitoring).
• Consensus-first (core interaction paradigm) — active only when call/a11y is on; typing mode keeps the Phase 1 behavior: a prompt soft constraint (guide_dog_call_consensus systemPrompt variable, conversational wording: understand intent first, ask when unclear, explain before writing/modifying and wait for the user's go-ahead) plus a mechanical hard guarantee (tools/pre-execute waterfall interception: write/edit and destructive-bash heuristics rm/mv/cp/truncate/dd/overwriting-redirect/git push etc. → unconfirmed returns {kind:'deny', reason: 'needs_voice_confirmation'} and the model asks by voice; user confirmation keywords hit → released for this turn; before every execution the host TTS-broadcasts a one-sentence summary built from the tool args (not through the model), then opens a consensus.summaryWindowMs interruption window; speech inside the window aborts the execution — the tool has physically not started). Interceptor failure → deny and announce "consensus check failed" (better to block wrongly than to allow wrongly, spec §6.8).
• Progress announcements (minimalist principle, RC10) — only useful information is announced: agent/status (running → "processing"), tools/result (tool name → phrase: write/edit → "modifying files", web_search → "searching the web", guide_dog_image/video/music/speak → "generating media", bash only for destructive commands (the same DESTRUCTIVE_BASH_RE as consensus) → "running a command"; read/grep/glob/ skill/non-destructive bash/unknown tools stay silent), agent/error ("processing failed: "); same-phrase 4s cooldown dedupe (multi-step same-kind operations announced once); >120s without any event during a call → heartbeat "still processing, please wait". Announcements and reply playback share one queue: announcements first (queue head), replies yield; announcements go through the streaming channel (the same WebAudio PCM chain as replies) → constructive serialization on a single player, one after another — overlap is impossible.
• Streaming TTS — reply text is split per sentence (stream.sentenceSplit charset 。！？.!?\n; stream.maxSentenceChars force-truncates over-long sentences) and synthesized sentence by sentence; each sentence gets a fresh one-time token via guide-dog/tts-token (single-use, 5-minute validity, bound to sessionId); pre-synthesis between sentences (while the current sentence plays, the client requests the next sentence's stream ahead of time and appends decoded frames seamlessly on the playback-time chain — the next sentence continues before the previous one finishes; long replies are read in full with ≤400ms gaps). Measured ~600ms to first byte for short Chinese sentences, satisfying the "first audio <1.5s" criterion. Only the turn's final message is played (RC13): intermediate assistant messages (with tool-call blocks) are not enqueued — playing near-identical text per step was the root cause of "the same content repeated"; intermediate steps are covered by progress announcements. A terminal tool turn (the last message still has tool calls) is covered by the turn/end fallback that plays the buffered text — never silent.
• Barge-in — VAD detects user speech during playback (≥ vad.interruptMinMs 300ms to avoid false triggers) → the browser immediately stops playback and clears the unplayed buffer → the stop is a 10ms fade-out (RC13) — src.stop() hard cuts click at sentence boundaries. The first transcript segment after the interruption goes straight to the current turn via the interrupt RPC (agent.steer, RC11) instead of queueing as a new turn → abort the current tts-stream fetch → the new speech naturally becomes the next turn (Pipecat InterruptionFrame semantics).
• Voice commands — call transcriptions that hit the command table (pause, resume, repeat/say-again, slower/faster, look-at-screen [Phase 3 stub]) execute locally and are not submitted to the conversation (stop/continue are consensus confirmation words, not in the command table — they pass through to the agent unchanged); guide-dog/call-command RPC provides host-side commands such as clear-queue.
• Dual-channel mutual exclusion (RC13) — text already spoken on the host speakers via guide_dog_speak(playOnHost=true) is not replayed through the voice-mode/call queue channels (consumed on use, the same text blocked once) — eliminates the "host + browser double sound".
• Fault tolerance — a stream interruption auto-reconnects once (at most one retry per (sid,text) within 5s, no retry on 429; a fresh token per sentence; failure toast "playback interrupted"); STT failure does not submit + beep + toast; TTS failure still lands the text + failure beep + panel error state (never silent); consensus-interceptor failure denies conservatively and announces the reason. Session ownership for call transcription/interruption/polling is captured once when the call starts (RC13) — multi-session switching no longer cross-talks.

RC14 fixes (2026-08-17): announcement content selection + queue tail-truncation + progress dedupe + double-play pinpointing

• Announcement sanitization (sanitizeSpeechText, F1) — before enqueueing, reply text goes through a markdown/URL/emoji strip: [title](url) keeps the title, drops the URL; bare URLs (https?://, www.) are removed entirely; leading list/quote markers (-/+/*/>) and leading ordered-list markers (1. 1、 1)) are stripped; markdown markers such as **bold** and backticks are stripped; emoji ranges (U+1F000-U+1FAFF etc.) are stripped. Call mode reads only human language — no URLs/**/-/📢 metacharacters, so URL fragments like "thepaper/newsD/weather.com" are no longer read out in pieces.
• Smart sentence splitting (splitSentences, F2) — in the default separators '。！？.!?\n', . follows a smart rule: split only when . is followed by whitespace + an uppercase letter/digit/CJK ('Hello. Next' → 2 sentences; '8.17 的上海' stays 1; dots inside URLs are never split). The Chinese separator set gains ；; so …； no longer cuts one Chinese reply into two halves.
• Queue cap 40 with tail truncation (VOICE_QUEUE_MAX, F3) — raised 10→40, drop from the tail to keep content: on overflow while (q.length > VOICE_QUEUE_MAX) q.pop() (first-in content wins; the old splice(0, …) head-removal strategy cut the main content first while keeping URL fragments). announce/hb progress still use pop() (unshifted to the queue head, progress first).
• 30s progress-phrase dedupe window (F4) — announce's progressDedupe cooldown extended 4s→30s: web_search results ~4.3s apart no longer announce "searching the web" three times. The progressDedupe function body is untouched; repro-progress.js semantics preserved.
• Dual-channel mutual exclusion by sanitized text (F5) — wasHostSpoken / markHostSpoken uniformly use sanitizeSpeechText-processed text as the key: all three wasHostSpoken call sites (downlink, turn-end flush, voice-mode) match on the sanitized key; speakImpl registers both the raw and the sanitized key after a successful playOnHost (double key) so any downstream channel dedupes correctly. Known edge: when transform.py rewrites the text the two keys may differ slightly (acceptable edge).
• Diagnostic instrumentation (F6, one-shot retest to pinpoint "reading twice") — zero behavior change, logs only:
  • host ([gd-host], visible in the DSH terminal): enqueue from=downlink|turnend|voice-mode|consensus|announce|heartbeat n=... qlen=..., shift key=... remain=..., skip host-spoken sid=... text=..., QUEUE-DUP text=....
  • client ([gd], browser DevTools): playStreamEntry ... times=... (accumulates per entry.key || entry.text on each play), PLAY-SUMMARY key=N | ... (when the queue empties, summarizes all current counts then clears).
  • Retest calibration (basis for the RC15 direction): QUEUE-DUP → host double-enqueue; PLAY-SUMMARY key=2 → client double-play; neither but still twice → tts-stream double audio write; enqueue from= same source twice with the same text → event replay.

RC15 fixes (2026-08-17): persistent player + gesture unlock + failed-entry requeue + event-replay dedupe

• Persistent voice player (playVoiceEntry, F1) — voice-mode playback changed from "new Audio() + temporary URL each time" to fetch + Blob + single-element reuse: the whole clip is fetched once into a Blob, bound via URL.createObjectURL to a single <audio> element; later entries only replace src and the playback callbacks — no more repeated Audio-object creation/destruction, which eliminates the ERR_CONTENT_LENGTH_MISMATCH retry storm (each Audio rebuild replays the pre-mismatch portion; with long audio this looked like "repeated replay + stutter").
• Gesture unlock + blocked-pending retry (F2) — under the browser autoplay policy the first play may be blocked (play() rejected): enter a "pending playback" state, bind the first user gesture (click/keydown/touchstart, capture phase, persistent listener) and continue automatically; blocked entries are no longer dropped — they replay after the gesture. stopCurrent now correctly releases busy and requeues (prevents a busy deadlock from swallowing entries).
• Failed-entry requeue RPC (voice-requeue, F3) — on playback failure (decode/network/blocked) the client calls the host voice-requeue RPC to re-enqueue the entry (requeueEntry pure function: new text inserts, duplicate text skips, tail pop truncates), max 3 retries per entry (attempts map) — no more lost content.
• Event-replay 10s text-window dedupe (replayDup, F4) — host enqueue for the "call downlink + voice mode" channels gains a 10s last-text dedupe window (lastStreamText/lastVoiceText maps): the same text enqueued again within 10s is skipped ([gd-host] skip replay text= / [gd-host] skip voice-dup text= instrumentation). Root cause: event replay on voice mode + call downlink enqueued the same text twice — the male-voice reply repeated "7 times" because this window was missing.
• url-entry play counts (F5) — PLAY-SUMMARY summary logs cover voice-mode entries (playCounts keyed by entry.key || entry.url, summarized and cleared when the queue empties) — url-entry play counts are trackable; retest pinpointing no longer relies on guessing.
• Build marker — the client build tag was bumped to rc15-20260817 at the time (plugin-client.js source and bundle/lib/client.js in sync; visible in the DevTools console after a hard refresh; since superseded — the current tag is rc20-20260817).

config.json schema (Phase 2: call / a11y)

New keys added on top of the Phase 1 config (~/.dsh/guide-dog/.guide-dog/config.json); all optional, deep-merged over defaults (spec §4 copy):

{
  "call": {
    "mode": "vad",
    "vad": {
      "method": "energy",
      "threshold": 0.02,
      "silenceMs": 700,
      "minSpeechMs": 300,
      "maxSegmentSeconds": 60,
      "interruptMinMs": 300
    },
    "stream": {
      "format": "pcm",
      "sampleRate": 24000,
      "sentenceSplit": "。！？.!?\n",
      "maxSentenceChars": 200
    },
    "voice": "English_expressive_narrator",
    "speed": 1.0,
    "progress": true,
    "consensus": { "enabled": true, "summaryWindowMs": 3000 }
  },
  "a11y": {
    "enabled": false,
    "autoNarrate": true,
    "visionCloud": true,
    "summaryFirst": true
  }
}

• call.mode: vad (default, automatic endpointing) | ptt (push-to-talk).
• call.vad.method: energy (Phase 2 v1, RMS energy threshold; raise threshold in noisy environments) → upgrade slot silero (web-vad browser WASM) / sherpa (VAD+ASR integrated).
• call.stream.format/sampleRate: mmx speech synthesize --stream args (s16le mono PCM; 24000 is the explicit override — mmx's own default is 32000).
• call.consensus: enabled toggles consensus-first (effective only in call/a11y); summaryWindowMs is the window that waits for the user to interrupt after the summary announcement.
• a11y: Phase 3 accessibility-mode config (this stage only enabled participates in the call streaming/consensus decisions; autoNarrate/visionCloud/summaryFirst are reserved for Phase 3).

Routes (Phase 2)

Method & Path	Purpose
POST /guide-dog/call-transcribe	Uplink: whole-clip audio (the client sends a raw audio/webm body with an x-session-id header; the host base64s the whole body and hands it to whisper), hard cap ≤20MB; the host reuses the Phase 1 transcribeImpl → {ok, text, language, durationMs}
GET /guide-dog/tts-stream?token=…&sid=…&text=<sentence>	Downlink: chunked PCM audio stream (content-type: audio/pcm, cache-control: no-store); requires a one-time token issued by guide-dog/tts-token — missing/wrong token → 403, in-flight stream for that session → 429

RPC-style endpoints (tts-token / call-active / call-command) go through the same JSON POST compatibility layer; their physical URL is /guide-dog/api/guide-dog/<name> (double prefix, same as the Phase 1 guide-dog/status example) — see the three new rows in the RPC surface table below.

Verification

node --check bundle/lib/index.js && node --check bundle/lib/client.js       # bundle syntax ×2
curl -s -X POST http://127.0.0.1:3080/guide-dog/call-transcribe \
  -H 'content-type: application/json' -d '{}' | head -5                    # uplink route reachable (empty audio → error JSON)
curl -s -o /dev/null -w '%{http_code}\n' \
  'http://127.0.0.1:3080/guide-dog/tts-stream?token=bad&sid=x&text=hi'      # invalid token → 403

Manual acceptance checklist (full criteria in specs/2026-08-14-guide-dog-v2-design.md §6.9; verify item by item after deploying and restarting DSH):

1. VAD: speak-pause-speak becomes two separate turns; silence detection does not cut wrongly (threshold adjustable).
2. Turn loop: voice → transcription → submit → agent execution (incl. tool calls) → reply read aloud; an end-to-end "use voice to have the agent generate an image / search" completes.
3. Barge-in: speaking during playback stops it; the next turn works.
4. Progress announcements: at least one stage announcement while the agent runs a tool.
5. Stream safety: non-allowlisted Origin and missing/wrong tokens rejected; recovery after a dropped-stream reconnect.
6. Full streaming: long replies read in full; "repeat / pause / slower" commands work; first-audio latency ≤1.5s, playback gaps ≤400ms (measured).
7. Consensus-first: voice "change X in README to Y" → not executed immediately → voice confirmation → after confirmation a short summary is heard before every write; speaking during the summary aborts that execution and the user's speech becomes a new turn; an unconfirmed write/edit is blocked (check the tools/pre-execute interception path).
8. Ambiguous intent (e.g. "change that file" without context) → the agent asks the key question by voice instead of guessing; the user asking "why change it?" → the agent explains by voice.
9. PTT: hold to talk / release to send; the mode switch takes effect in VAD mode.

RPC surface (Client → Host)

Method	Args	Returns
guide-dog/speak	{text, voice?, speed?, language?, playOnHost?}	{ok, url, file, voice, bytes}
guide-dog/list-media	{limit?}	[{name, kind, prompt, voice, ts, bytes, url}]
guide-dog/auth-status	—	{ok, method, source, keyMasked}
guide-dog/voices	{language?}	{ok, voices[]}
guide-dog/get-config	—	{ok, config} (merged defaults)
guide-dog/set-config	{patch}	{ok} / {ok:false, error}
guide-dog/status	—	{ok, status} (whisper probe + probeAt)
guide-dog/transcribe	{audioB64, mime, sessionId?, language?}	{ok, text, language, durationMs} / {ok:false, error}
guide-dog/beep	—	{ok, dataUri} (WAV beep data URI)
guide-dog/voice-queue	{sessionId}	{ok, entry} — pops one entry (play/error) or null
guide-dog/tts-token	{sessionId}	{ok, token} — one-time stream token (5 min, single-use, bound to session)
guide-dog/call-active	{sessionId, kind ('session'|'speaking'), active}	{ok} — session persistence vs instantaneous speaking flag (C4)
guide-dog/call-command	{sessionId, cmd}	{ok} — host-side call commands (clear-queue …)

Security notes

• Media dir inside the workspace root → no sandbox widening required.
• The route serves only plugin-owned media with allowlisted extensions.
• The MiniMax API key stays in mmx's own config (~/.mmx/config.json); the plugin never reads or forwards it.
• Host playback uses the raw subprocess service (players must outlive the sandbox's --die-with-parent bwrap profile); each new playback terminates the previous one.

Troubleshooting

• mmx not found / auth missing — tool returns {ok:false, error}; the settings page shows the auth problem. Fix: npm install -g mmx-cli and mmx auth login --api-key sk-… (or export MINIMAX_API_KEY=…).
• Sandbox denial — the tool error reports denied: true; keep media inside the workspace (the plugin already does).
• MiniMax-H3 returns "TokenPlan or Credit does not yet support the MiniMax-H3 model family" — the account's MiniMax plan does not include the H3 model family. Use model: "MiniMax-Hailuo-2.3" (legacy V1) or upgrade the plan. The plugin surfaces the API error verbatim (the message may appear in Chinese), so this is visible in the tool result.
• Video never finishes — the poll loop honors the call's abort signal and times out after 15 minutes; re-run with a shorter duration or different model.
• Cards show generic JSON — the client half did not load; check that the bundle client route /plugins/dsh-guide-dog/client.js returns 200 after a DSH restart, and refresh the page.
• Stop / update — everything (tools, route, prompt section, cards, settings entry) is disposed automatically; media files remain.

mmx output-shape notes (verified against mmx 1.0.19)

• --quiet changes per-command JSON shapes: speech voices prints a flat array of voice-id strings, text chat prints only the reply content (so the plugin runs text chat without --quiet), while auth status / search query keep their objects.
• video generate --async always prints {taskId} (raw stdout write).
• H3 (V2) task results carry content.url; the plugin downloads it with curl. Legacy V1 tasks return file_id, downloaded via mmx video download --file-id.
• File-writing commands (image generate --out-dir, music generate --out, speech synthesize --out, video download --out) may print nothing parseable; the plugin treats exit 0 as success and verifies the file via fs.stat.

Restart recovery (static web-profile bundle)

Since 2026-08-16 Guide Dog ships as a static bundle mounted in the web profile — one global host half + one client half, exactly like the published dsh-better-sidebar. No dynamic plugin, no per-session gdog-* instances, no approval cards: after a DSH restart the tools and the voice UI come back with the profile itself.

The source of record stays the two dynamic-plugin halves (plugin-host.js / plugin-client.js); deploy/convert_bundle.py regenerates bundle/lib/ from them:

• host half (bundle/lib/index.js, ESM name/apply): a tiny compatibility layer replaces the dynamic sandbox's harness — tool definitions are registered via the global tools registry (visible to every session), and the former harness.handle RPCs (guide-dog/*) become JSON POST routes under /guide-dog/api/. The per-workspace sandbox root is replaced by the global store ~/.dsh/guide-dog/ (config, media, scripts).
• client half (bundle/lib/client.js): a window.__ModuleLoader__.load({id, factory}) CJS factory like the published bundles; require('react') from the platform seed, self-managed <style> tag instead of the sandbox styles, and host.call becomes same-origin fetch against the JSON routes.

Deploy once after any plugin change:

1. python3 deploy/convert_bundle.py — regenerate bundle/lib/.
2. python3 deploy/publish.py — copies the bundle to ~/.dsh/dsh-guide-dog (outside workspaces), idempotently registers it in the web profile (~/.dsh/profiles/web: dependency link + bundles entry + node_modules symlink) and removes the superseded dsh-guide-dog-autoload bundle (which otherwise keeps deploying per-session dynamic instances).
3. Restart DSH (dsh web) — bundles are parsed at startup, so a restart is required after any change.

Legacy history: the earlier auto-deployer (autoload/) — a host bundle that watched agent/created and define+runed a fresh gdog-* dynamic plugin per session — is retained in the repo and still published to ~/.dsh/guide-dog-deploy / ~/.dsh/guide-dog-autoload for rollback, but nothing consumes it once removed from the profile.

Profile pitfall (observed 2026-08-15): dsh web is an alias for --profile web — the GUI runs the web profile. Registering a bundle only in another profile (e.g. cc-tui) silently does nothing for the GUI; deploy/publish.py always targets ~/.dsh/profiles/web.

Service-scope pitfall (observed 2026-08-16 — root cause #3): the dynamicCordisRunner and agents services are registered on agent-scoped contexts, not on the global/profile context a bundle's apply(ctx) runs in. ctx.get('dynamicCordisRunner') there returns undefined, so an early if (!runner || !agents) return in apply bailed out before the agent/created listener was even registered — the bundle loaded fine (verified via dsh web --dump-default-config) yet never deployed. The fix resolves both services through the event payload's agent.ctx (Agent exposes readonly ctx: Context; probe-verified that both services are visible there), with a global-ctx fallback for hosts that register them globally. Debugging aid: a temporary dynamic probe plugin (inject: ['dynamicCordisRunner', 'agents']) sees both services in its (agent-scoped) apply ctx — that asymmetry is the signature of this pitfall.

The bundle shape mirrors the published dsh-better-sidebar precedent: dsh.bundle.patch → cordis.patch.yml with a single insert row, named exports (export const name + export function apply(ctx)), no default export, host-only (no dsh.client block needed).

Phase 2 backlog (deferred from the V4-Pro final review)

• M9 — mic onstop closure holds a stale inputActions when switching sessions mid-recording; re-check recorder ownership before transcribing.
• M10 — the media route buffers the entire file in memory to satisfy range requests; stream only the requested byte range (matters once Phase 2 streaming TTS/playback lands).
• M11 — setVoiceOverride rebuilds the whole voiceMode.sessions map from possibly-stale config, so concurrent session toggles can clobber each other; move to per-key merge (host-side patch) or refresh cfg before write.