Passer au contenu principal
Z

dsh-omni

ztr66/dsh-omni

Un seul plugin pour toutes les modalités : explorer, transcrire, analyser et composer des contenus image, vidéo et audio.

Installer

dsh plugin --profile web add github:ztr66/dsh-omni

README

dsh-omni

One plugin for every modality: watch video, hear audio, generate images and clips — all through tools a model can call.

The DeepSeek Harness message model carries only text and images, so "let the model watch this video" cannot mean handing it a video file. Everything here converts: a video becomes a grid of timestamped frames, audio becomes a transcript plus measurable features. The model reads those and reasons about them.

What it does

Watching a video: you choose the density

video_scan turns a time range into one image of cells, each with a burned-in timecode, and returns a timeline mapping cell number to second.

video_scan({ input, fps: 2, cellWidth: 240 })            // whole film, find the structure
video_scan({ input, from: 24, to: 30, fps: 5, cellWidth: 480 })  // read one stretch
video_frames({ input, at: [26.5], width: 1280 })         // one exact frame

The sampling density is a parameter, not a policy. A first pass over an unfamiliar film wants 2 fps at 240 px — cheap, shows where things change. Reading on-screen text needs 5–10 fps at 480 px or more on a short range. Encoding "two passes" as a fixed strategy would make both cases worse.

Measuring music

audio_analyze returns numbers, not adjectives:

{
  "tempo":    { "bpm": 117.45, "beats": 22, "regularity": 0.977, "detected": true },
  "tonality": { "root": "C", "mode": "major", "estimatedKey": "C major" },
  "timbre":   { "centroidHz": 1562.5, "zeroCrossRate": 0.0779 },
  "dynamics": { "rmsDb": -21.64, "peakDb": -0.92, "crestDb": 20.73 },
  "sections": [ { "at": 0, "level": "loud", "energy": 0.9 } ]
}

tempo.detected: false is a real answer, not a failure — ambient beds and speech have no periodic beat, and inventing a number there would be misleading. There is no instrument classifier; that needs a trained model this plugin does not ship.

Speech, both ways

audio_transcribe tries the Host's recogniser first, then a local faster-whisper model, and reports which route ran. A local run needs no network and no key. audio_speak synthesises speech through edge-tts and reports the measured pace in characters per second, so a narration whose rhythm is wrong is visible before the film is assembled.

Drawing vs paying

video_generate is the most expensive and least controllable tool here, so it refuses to be convenient:

  • Scenes code can draw are refused outright — a title card, a chart, a geometric shape, a transition. The check is by kind, not by how much of the film they represent: a title card is a title card whether it is 2% or 80% of the running time.
  • Anything that passes needs consent — either an explicit user request with the user's own words, or an approval prompt through the Host. A deployment with no approval service is refused rather than trusted, because an unattended process should not spend money.
  • You build the film with video_compose and generate only the shots that genuinely need a model (a real wave, a real face, a specific material).

Tools

ToolWhat it does
omni_env_checkReport ffmpeg, Python libraries, Whisper models, local generation services, and the interpreter split
omni_probeRead real media metadata: codec, resolution, frame rate, sample rate, channel layout
video_scanVideo range → grid of timestamped cells + timeline
video_framesExact frames at full resolution
video_composeStills and clips + audio → MP4 (H.264 High, yuv420p, AAC 48 kHz)
video_extract_audioVideo → audio track
video_generateText → clip, behind a capability gate and a consent gate
audio_transcribeSpeech → timestamped text (Host or local)
audio_analyzeLoudness, tempo, tonality, timbre, energy profile
audio_speakText → MP3, with measured pace
audio_cutCut segments or join files for audio
image_generateText → image across eight provider adapters

Plus a omni-media skill that loads the decision tree into the model when a media task arrives.

Install

dsh plugin --profile <your profile> add dsh-omni

Requires DSH >=0.2.0-rc.1 and Node 20+.

Dependencies

The plugin is small; the tools are not self-contained. omni_env_check reports what is missing and how to install it.

ComponentNeeded forInstall
ffmpeg / ffprobeeverything involving a containerwinget install Gyan.FFmpeg
librosa + soundfileaudio analysispip install librosa soundfile
faster-whisperlocal transcriptionpip install faster-whisper
edge-ttsspeech synthesispip install edge-tts

Generation needs a provider and a key, configured in 设置 → 全模态. Nothing about generation works without one, and the key is stored in the Host credential store rather than this plugin's config.

Image-to-video

video_generate accepts firstFrame, lastFrame or images. The adapter picks the mode from what you supply, because the three modes have disjoint field sets and a stray field is a rejected request:

SuppliedAgnes modeEffect
nothingtexttext-to-video
firstFrame / lastFramekeyframethe frames anchor the shot's start and end
imagesreferenceassets inform content and style, not exact framing

The image must be a public URL. The provider fetches it from its own network while the task runs, so a local path, a file:// URI, localhost or a private address will never resolve for their crawler — and the failure surfaces as a generic task error naming neither the URL nor the reason. The plugin checks the shape of the URL before spending a request and says which of those it is.

A signed URL that expires before the queue drains fails the same way; keep the link alive until the task completes.

Getting a local file onto a public URL

Since the provider fetches the image itself, a file on your disk needs a host first. Configure one under Settings → 图片图床 and the plugin uploads local paths automatically — you pass the path, it returns the URL, and the request proceeds.

FieldWhat it is
Host七牛云 Kodo. Choose 不配置 to disable and pass URLs yourself
Region华东 z0 / 华北 z1 / 华南 z2 / 北美 na0 / 东南亚 as0
AccessKey / SecretKeyQiniu credentials, from the Qiniu console's key page
BucketThe bucket name, which must already exist
DomainThe bucket's bound domain, e.g. abc123.hd-bkt.clouddn.com

Two things catch people out:

  • A trial domain serves plain HTTP only. Its certificate is issued for the CDN's own hostnames, so an https request fails TLS before it reaches the file. The upload probes once and returns the scheme that actually loads, which is why a working link may read http://.
  • The secret key field is write-only. It is masked in the UI; leaving it blank when saving keeps the stored value rather than clearing it.

Qiniu trial domains expire after 30 days and are rate-limited. For anything long-lived, bind your own domain to the bucket.

Provider adapters

Image — OpenAI Images, GPT-Image, Gemini, SiliconFlow, Volcengine Ark, Zhipu CogView, DashScope Wanx, Mona Studio, plus local A1111/ComfyUI/SwarmUI. Fields differ per provider (size, quality, response shape); the differences live in descriptors, so the caller sees one shape.

Video — Seedance (Volcengine Ark), Agnes Video, Zhipu CogVideoX.

Two environment traps this plugin works around

The Python split. On a machine with both a DSH runtime and a system Python, packages divide between them: the runtime may have librosa while the system one has edge-tts. A single script that both analyses and synthesises fails at import time, so each operation runs in the interpreter that can actually do it, and omni_env_check reports the split as pythonConflict.

Windows stdio encoding. Python defaults stdio to the console code page there, so a json.dumps(..., ensure_ascii=False) payload comes back as mojibake even though the data is correct. Every Python child runs with PYTHONIOENCODING=utf-8.

Known limits

  • Video and audio are never handed to the model directly; the Host has no such modality. Everything is converted first.
  • No instrument classification.
  • The local generation backends are wired by configuration but were never verified on a machine without one running. If one fails, report the error rather than assuming the plugin is broken.
  • Transcription of lyrics from a dense mix needs a vocal-isolation step first.

Licence

MIT. See LICENSE.

Plugins associés