Плагины

Нативная координация GPU для локальных ИИ-компаньонов в AstrBot/DSH: LLM, IndexTTS и ComfyUI используют GPU по очереди, без прокси.

Бюджеты размышления на запрос, бэкенд компактификации и вкладка настроек для локальной линейки Qwen3.8 (llama.cpp или NInfer).

Ставит перед stdio MCP-серверами три инструмента маршрутизатора (list_capabilities, describe_tool, call_tool), чтобы загружать схемы по запросу, а не в каждом запросе. Результаты проходят детерминированную последовательность обработки, исходник сначала архивируется (retrieve_spill читает его обратно). Можно задавать стоимость групп в токенах и скрывать встроенные группы для агента. Подключается маршрутизатор Python на стандартной библиотеке; требуется Python 3.9+.

Плагин DeepSeek Harness (DSH) для Docker-панели llama.cpp `llamapad`.

One-step local model onboarding plus a light harness for DSH. On load it probes loopback and private-network inference endpoints (llama.cpp, Ollama, vLLM, LM Studio), reads each model's context window and output budget from the backend, and registers the provider through settings, so a local model becomes selectable without hand-writing config. It then trims only that provider: masks the tool list through tools.restrict, drops named system-prompt sections and memory contexts, swaps the injected instruction block for a short one while preserving its first 80 and last 96 characters, and truncates tool-schema descriptions without touching structure or required fields. Cloud providers are returned as the same assembly object, untouched. The injected block is re-read from disk on every assembly, so editing the file takes effect on the next request with no reload. The panel in the plugin detail page is a live operations view: it shows whether the takeover is active and by which hook, how many characters the last assembly was trimmed from and to, and which providers are online (probed concurrently, with a real one-shot completion reporting latency and reply text per provider). If a local backend is not running, the panel offers a one-click launch: it probes the port first (never starting a duplicate), then starts the matching server (ollama serve, or a custom command you supply) detached and polls until the endpoint answers, clamping the wait to 3-600 seconds, reaping the child it just spawned if it never answers, and naming the paths it searched when it cannot find the binary. A stop button kills backends this plugin started (SIGTERM, then SIGKILL) and refuses to touch a server somebody else launched. When a model has just been downloaded and no backend exists yet, the panel can also scan the machine for .gguf files with hard caps (300 models, 20000 directory entries, Downloads only one level deep), start llama-server on the chosen one (skipping the .mmproj. vision projection and reusing a server that already answers), register the provider under the model id the backend reports, and run one real completion to prove it works. Before spawning anything it does a fit check that counts only the part of the file the GPU cannot hold: hostBytes = fileSize - VRAM, compared against max(total RAM - 2GB, free RAM). A file far larger than RAM is refused outright; a heavy host-side remainder is refused unless you confirm; the same remainder under mmap is only a warning, because the weights land in reclaimable page cache. A load-mode selector (auto / mmap / none / mlock / mmap+mlock) maps to llama.cpp --load-mode, and the default is whatever the engine picks. From 0.4.2 everything above is derived from the platform instead of assumed: home and data directories come from os.homedir() and the OS conventions (LOCALAPPDATA on Windows, ~/Library/Application Support on macOS, XDG_DATA_HOME elsewhere); backend binaries are looked up in the official install locations of each OS and can be pinned with OLLAMA_BIN, LLAMA_SERVER_BIN, VLLM_BIN or LMSTUDIO_BIN; model discovery walks DSH_LOCAL_MODELS_DIR, ~/models, ~/llama.cpp/models, ~/.cache/llama.cpp and the LM Studio data directory, with the WorkBuddy layout demoted to one candidate among them. VRAM is asked from nvidia-smi, then rocm-smi, then sysctl hw.memsize on Apple Silicon (flagged unified, so the same bytes are not charged twice), and any flip to a unified-memory budget is handled explicitly. Before building a command line the plugin runs llama-server --help once and keeps only the flags that build actually has, so older engines get --n-gpu-layers 99 and no KV quantization while newer ones get auto plus q8_0; anything it cannot probe is passed through rather than dropped. LM Studio is started with lms server start --port N rather than being lumped in with the engines that need a model argument, and vLLM gets its own --model example instead of llama.cpp syntax. Windows, macOS and Linux are all exercised in CI, including a windows-latest and a macos-latest job, because a cross-platform claim that has only ever run on one OS is not a claim. Two of those suites exist specifically to catch this machine's blind spots: one replays the whole suite with process.platform rewritten to each OS, and one deletes every advantage this development box has — no nvidia-smi, no rocm-smi, no sysctl, an empty PATH, a home directory with no models — and requires graceful degradation rather than a crash. That last one caught a real class of bug: several assertions hardcoded 'lms.exe', which cannot pass where the executable has no suffix. Measured on the reference machine: a 16.9GB 27.2B MoE with 14.5GB of VRAM holds a 285-691MB working set and sustains 76 tok/s after a 1000-2000 token warm-up, so file size alone is not a fit signal. From 0.4.3 the scanned models are shown as a table rather than a dropdown (one row per gguf, with its size and any fit warning), so you can see what is on disk before choosing. From 0.4.4 the delete button on the provider table removes the provider entry, not the weights file on disk. That distinction matters: what people actually want to clear out is the dead entry - one pointing at a port that never answered, one registered twice for the same port, one whose model file was deleted long ago - not a 15.8GB gguf. Deleting a config is undone by importing again; deleting a gguf is not. The removal goes through settings.mutate with an unset op on \['providers', id], because update merges layers: writing null there leaves a null provider key that the loader reads as a broken config rather than an absent entry, and only mutate/unset is implemented as Reflect.deleteProperty. The result is read back afterwards, because unset on a key that comes from a lower layer (the profile cordis.patch.yml or a home patch) does not delete it - it restores the inherited value, so the call succeeds while the entry stays. When the entry is still there the panel says so and points at the file to edit instead of reporting success, and a DSH whose settings service predates mutate() is told plainly rather than silently ignored. Deleting takes two clicks and clearing the confirmation is as easy as clicking another row; on success the online and baseURL caches are cleared and the tool surface is recomputed, since the provider list changed. The gguf itself is never touched by this plugin - the scanned-model table is display and selection only.VRAM reporting is driver-dependent. From 0.4.5 model discovery also reads Ollama's own storage: its blobs are named sha256-… with no extension, so extension matching could never list them. Weights are now accepted by extension OR by GGUF magic header (only dot-free files of at least 8MB, capped at 600 header reads per scan; obviously-not-weights extensions like .json/.md/.safetensors/.part are never opened). Ollama models are resolved from their manifests: the model layer is picked by mediaType application/vnd.ollama.image.model (never adapter/mmproj layers), the human-readable name is restored from the manifest path (…/qwen3/8b becomes qwen3:8b, …/latest stays bare), tags sharing one blob merge into a single row, and the metadata kv supplies name, quantization and context length. Adopting an Ollama model does not start a duplicate llama-server over the same weights: it confirms ollama serve answers on 11434 (launching it if not) and registers the provider under the tag the endpoint itself reports — the blob's sha256 name would never match anything, since 11434 lists qwen3:14b, not sha256-a8cc…. Two real-machine-only traps fixed on the way: existsSync('<file>/') returns true on Windows, so manifest-vs-directory branching now goes by parseable content instead of trailing-slash semantics; and the quantization label table had been copied from a stale ggml enum generation, labelling ggml_type 15 as Q5_K_S while ollama show reports Q4_K_M. Same-blob dedup is by (dev,ino), keeping whichever entry has a name. Verified on the reference machine: the scan goes from 1 row to 4 — qwen3:14b (8.64G) and qwen3:8b (4.87G) now appear with Q4_K_M badges — in 4ms; npm run verify exits 0 (launcher 57, portable 35, panel-host green including the linux/darwin simulated-platform runs); the new panel-host suite builds a real miniature Ollama store in tmpdir and points OLLAMA_MODELS at it, so results no longer depend on whether the test machine happens to have Ollama installed. From 0.4.6 the panel shows each model its real name instead of its file name, because some download tooling stores every weight under one generic file name, leaving the real name only inside the GGUF header. The scan now reads each file GGUF KV section - not a fixed-size head buffer, because tokenizer.ggml.tokens (hundreds of thousands of strings on a large MoE model) and the chat template push general.file_type past any window; and GGUF string arrays are interleaved (u64 length plus payload per element), so treating them as a length table walks the cursor off the rails and produces astronomically large reads. Reading and skipping are accounted separately: key names and types are always read, string values over 256KB and string arrays are only walked past without copying, non-string arrays over 64 elements are skipped outright, and a cursor past the file size stops the read, because an out-of-range position means the cursor is wrong rather than that a limit should be raised. The displayed name goes Ollama tag, then general.basename plus size_label, then general.name, then the file name, with the full string kept for the hover text since it usually carries internal build numbers; mmproj files are now identified by general.type as well as by name. A short read (truncated file) no longer feeds half a value into the parser. On the reference machine, reading one file header parses 44 keys in about 16ms, and a whole scan takes about 31ms. From 0.4.7 the smoke test no longer fails the first time and passes the second. The old code gave both request attempts one shared AbortController and one shared 30-second timer, so the total budget covered both rounds: a cold load of a large model burned it in round one, the abort left the signal permanently aborted, and the second round (the Ollama-style reasoning_effort: none retry) could not even be sent - which is exactly why the first click returned This operation was aborted and the second succeeded. Now each attempt gets its own controller and timer, a timeout retries once automatically (retryOnTimeout, on by default) because a first-request timeout almost always means the weights are still being read from disk, and non-timeout errors (refused connection, HTTP error) are not retried at all. Timeouts report themselves: the message says the wait exceeded smokeTimeout, whether a retry already happened, and points at the new smokeTimeout setting (seconds, default 60, clamped to 5-600) - port responding is not the same as tokens coming out, so it is a separate knob from launchTimeout. Unrelated: an example naming one specific model was dropped from the panel copy, since not every machine has that model downloaded, and the explanation now describes the mechanism instead.

Добавляет в список выбора модели собственного Hugging Face Space с GPU, запускает Space при отправке сообщения, показывает ход запуска и ставит его на паузу при простое или достижении лимита времени. Нужны аккаунт Hugging Face, GPU Space с API, совместимым с OpenAI, токен Hugging Face в vault DSH под именем HF_TOKEN и платный тариф на GPU-оборудование. Hugging Face выставляет счёт за время работы GPU.

Обнаружение долгих задач на host для локальных моделей с малым контекстом: длинный вывод делится на parts skeleton (одна часть на ответ, затем сборка), многошаговая работа — PLAN.md ledger и notes/ (сначала план, шаг на ответ). Эвристика host-side, не полагается на оценку моделью своих пределов.

Плагин DeepSeek Harness (dsh): показывает состояние локальных LLM-эндпоинтов в правой боковой панели Web в реальном времени. Отображает слоты llama.cpp (занят/TTFT/завис, `/metrics`) и MODELS Ollama (загружены или пусты). Определяет движок по форме ответа и устанавливается как `dsh.bundle`.

Плагин DSH добавляет параметры сэмплирования модели (top_p, top_k, min_p, repeat_penalty, presence_penalty, frequency_penalty) в каждый запрос chat-completions к локальному шлюзу llama.cpp/llama-server.

Панель управления локальными LLM-серверами в Web-представлении DSH: реестр служб с проверкой состояния и остановкой, именованные профили параметров llama.cpp/SGLang/vLLM с проверкой KV/VRAM перед сохранением, обнаружение GPU и тесты tok/s.