dsh-continual-evolve

dsh-continual-evolve

中文 | English

Continual self-evolution for DeepSeek Harness: a versioned, auditable, rollback-safe harness state layer — prompt notes, memories, skills, subagent specs — refined from session trajectories.

The model proposes, the code guarantees. Every mechanical safety property — schema validation, atomic writes, snapshots, versioning, audit trail, acceptance decisions — is enforced in code, never by prompt discipline.

Why

Agents accumulate reusable experience (repeated failures, durable facts, reusable procedures) and forget it next session. This plugin turns that experience into first-class state:

• Three scopes with merge semantics (global < project < local): local per-session staging, project per-workspace cross-session store, global cross-project — plus mechanical promotion guards so only portable, substantial, non-duplicate knowledge reaches global
• Typed one-fact memories: every memory entry carries a recall type (user | feedback | project | reference); pitfalls (feedback) must include Why + How to apply
• Dedicated background memory agent: eligible successful turns feed a bounded ZCode-style loop that searches the frozen memory manifest and proposes memory-only edits through a closed tool set; it cannot call agents, MCP, the network, or write source files. The generic review/planner/fate path is not run by this listener
• Memory recall, projection, and receipts: evolve_recall reads back full memory content by query/kind/scope/type; every memory apply also materializes a readable MEMORY.md index plus one fact file per entry; each extraction lands a unified audit receipt (no-op/applied/declined with duration and turn stats) and only applied outcomes notify the session
• Deterministic rollback: inverse edits generated from applied results — no LLM re-guessing
• Benchmark loop: candidate refinements are evaluated against frozen cases by a separate scorer before acceptance (rubric encrypted at rest)
• Store hygiene: /evolve consolidate turns write-time conflict hints and zero-use staleness into one approved, fully reversible batch of archives — with merge, near-duplicate content folds into the surviving original

How it works

1. Sediment — the model creates entries via evolve_add, or the automatic Memory Agent consumes incremental snapshots after successful turns. Generic review/planner remains manual unless separately invoked.
2. Capability-aware auxiliary calls — the memory loop, review, planner, wrapup, and fate resolve exact provider/model metadata through src/llm-text.ts, use the lowest advertised enabled reasoning effort (falling back to a closing effort only when no enabled level exists), and forward the host session id for provider routing; models without reasoning metadata use their provider default.
3. Guard — code-enforced validation: edit schema, blast-radius/scope coherence, and the promotion policy (project-scoped markers, thin content, near-duplicate detection, credential screening keep the global store clean — secrets are rejected at every write sink, including mount materialization). Global creates that near-duplicate an existing entry are rejected at write time (≥0.8 similarity); moderate overlaps carry a conflictHint for later consolidation.
4. Approve — global and project writes require explicit human approval; the dialog shows the bounded structured edit diff and conflict warnings, while malformed/lost responses remain retryable rather than counting as rejection.
5. Apply & inject — memory batches preflight every persistent approval, recheck abort before writes, and compensate earlier scope writes if a later batch fails; every successful scope still passes through snapshot + audit. Prompt notes and delegation specs inject into the system prompt (capped, relevance-ranked, contradicted entries demoted, zero tokens when empty); memory CONTENT injects at session start under a hard character budget, frozen per session, with the when_to_save guide riding along; skills still appear as a relevance-ordered capped directory index.
6. Validate & roll back — benchmarks score candidates against frozen cases; rejected candidates roll back deterministically and are captured as draft regression cases (auto_regression benchmark).

Install

# from npm (installs and activates — ships its own bundle patch)
dsh plugin add dsh-continual-evolve

# or from source (first GitHub installs require approving the allowBuilds step)
dsh plugin add ZK-Andy/dsh-continual-evolve

Restart the DSH profile you use (dsh web or the desktop host) after installing or updating.

Usage

Commands (in-session):

Command	Effect
/evolve	short help + current local store (/evolve help all lists every subcommand)
/evolve list · history · rollback <id>	inspect and revert (add project for this project's store, global for the cross-project store)
/evolve plan [msg]	run the LLM planner against the store
/evolve wrapup	assess this session's local entries: promote / archive / keep
/evolve archive · unarchive <id>	hide from injection (data kept, restorable)
/evolve consolidate [apply] [merge]	report (or apply) one batch archive of conflict-hinted + stale zero-use global entries; merge folds near-duplicate content into the survivors
/evolve failures	aggregated failure classes (gate + benchmark)
/evolve log [tail N] [session <id>]	plugin log
/evolve export · import <path>	backup / restore a store
/evolve mount · unmount <skillId>	hot-mount an executable skill as a live plugin
/evolve goal [objective · done · block]	round-driven auto-review goal
/evolve benchmark …	case lifecycle, runs, acceptance
/evolve pause · resume · status	pause/resume the auto-review gate (manual tools keep working), gate state
/evolve usage	per-entry injection counts + exact provider-reported tokens for direct memory/review/planner/wrapup/fate calls (benchmark host subagents excluded)

Model tools: evolve_list / add / update / delete / rollback / recall (evolve_delete takes id or a batch ids array — one refinement, one approval; evolve_recall filters by query, kinds, scopes, memory types, and limit, and returns full content with version, source, and staleness). Memory reads and writes live in the conversation — the model carries the injected memory index plus evolve_recall/evolve_add — so there are no /evolve remember · forget · recall counterparts: ask in chat.

For third-party consumers: every applied evolution (gate or manual) appends a structured evolve_complete event to reviews.jsonl (src/evolve-event.ts defines the shape) alongside the human-readable audit records.

/evolve usage also reads evolve/token-usage.jsonl: exact provider-reported input/cache/output/total tokens for the plugin's direct memory-agent, review, planner, manual-wrapup, and automatic-fate calls. The report covers a retained tail rather than lifetime usage, distinguishes missing provider samples, and explicitly excludes host benchmark subagents, their agent-loop calls, and per-entry injection attribution.

Injection shape: prompt notes and delegation specs inject with content (≤6/kind × 180 chars, relevance-ranked). Memories inject as CONTENT at session start: the evolve:memory-index section (order 400) carries the when_to_save guide plus every memory body the budget allows, ranked project > feedback > user > reference, with overflow degraded to [memory:type:id] title hooks and the remainder counted (hard cap memoryIndex.maxChars, default 6000 chars; evolve_recall reads anything not shown). The section is computed once per session and reused byte-for-byte, so the system prompt stays stable and prompt-cache reads keep hitting — a memory written mid-session appears in the next session. Skills still appear as a relevance-ordered directory index (capped at 15 lines with a fold counter); memories no longer do. Every memory apply also refreshes a readable MEMORY.md index plus one fact file per entry in the store directory. With memoryIndex.guide off and an empty store, injection is zero tokens.

Configuration

Key	Default	Meaning
baseDir	resolved DSH home	root for the evolve/ stores
autoReview	false	initial Memory Agent default when no evolve/runtime.json exists yet; the listener is always registered, so this is not a registration gate
memoryMinUserWords	3	ZCode-style minimum lexical words in one direct user text part; uses CJK-aware segmentation
sessionCloseDrainMs	15000	session-close bounded drain for in-flight extraction in ms (0 aborts immediately)
reviewIntervalTurns	6	legacy local-fate cadence fallback; successful-turn review no longer waits for this interval
maxReviewInputChars	40000	trajectory slice handed to the gate
reviewBudgetTokens	4096	output budget for the gate call
notifyOnAutoReview	true	visible follow-up notice after an applied gate run
requireGlobalApproval	true	global and project edits ask for explicit approval
recordLanguage	auto	authoring and dialog language: auto follows the DSH client preference, then the session's own user text, then en; zh/en pin it
localFate	false	optional local-entry promote/archive fate assessment; unreachable while the listener runs memory-only, so it only affects direct/full callers
fateIntervalTurns	follows reviewIntervalTurns	minimum turns between fate assessments
goalBlockedWrapupTurns	3	consecutive blocked-goal gate runs trigger one fate assessment (0 disables)
promotionBlockPatterns	POSIX paths, session ids, ~/.dsh	content matching these is project-scoped and never promoted to global
promotionMinChars	100	whole promotions below this length stay local
injectionDirectoryLines	15	entry-directory lines per build before folding into a counter
sectionOrder	118	system-prompt section order for the guidance + entries sections
memoryIndex.enabled	true	register the evolve:memory-index content section
memoryIndex.guide	true	include the when_to_save guide (with the store empty and this off, the section renders nothing)
memoryIndex.order	400	section order (upstream named slots start at PLAN_POLICY=500; the plugin's other sections sit at 118/119)
memoryIndex.maxChars	6000	hard character budget for the injected memories block
skillsDir	<dshHome>/skills	where skill entries materialize as SKILL.md bundles
rubricKey	auto-generated key file	AES-256-GCM passphrase for benchmark rubrics (DSH_EVOLVE_RUBRIC_KEY overrides)
logToFile / logLevel / logMaxBytes	true / 1 / 5 MiB	plugin-owned JSONL file log with rotation
autoRollbackOnReject	true	deterministic rollback after a benchmark rejection
autoCase	true	failed evolution attempts are captured as draft regression cases (auto_regression benchmark)
reviewModel	agent's own	optional cheaper model for the dedicated memory agent and review gate ("provider/model")
plannerPrefixCache	auto	Route A session-prefix input when cache evidence exists (session always, off legacy flat text)
plannerPrefixMaxChars	12000	session-prefix budget for Route A planning inputs (chars)
historyRetain	{snapshots: 20, refinements: 500, reviews: 500, tokenUsage: 500}	storage hygiene: snapshots per store, tail lines per store history, shared reviews.jsonl tail, and direct-call token-usage.jsonl tail

Example profile patch:

- id: continual-evolve
  config:
    autoReview: true
    reviewIntervalTurns: 6

The Memory Agent listener is registered even when autoReview is false — autoReview only supplies the initial default, so an install works without editing the profile. Use /evolve resume to enable successful-turn snapshots immediately, /evolve pause to suppress new snapshots and model work, and /evolve status to inspect the configured default plus the current runtime state. The runtime switch is stored in evolve/runtime.json; manual evolve_* tools and /evolve commands are not paused. Only the Memory Agent runs automatically: the generic review/planner, prompt/skill writes, and local-fate phases are not reachable from this listener. The memory trigger follows ZCode's lightweight eligibility: direct user text must contain at least memoryMinUserWords lexical words (CJK-aware segmentation), while empty/internal/direct-memory-write snapshots are skipped; compaction does not add a separate memory-only trigger. Every extraction writes a unified audit receipt (noop/applied/declined with duration and turn/search stats) to reviews.jsonl; only applied outcomes queue a visible follow-up, and a closing session lets in-flight extraction settle up to sessionCloseDrainMs before aborting.

Development

pnpm install && pnpm build   # deps + tsc -> lib/
pnpm test                    # vitest (1097 tests)
pnpm test:coverage           # v8 coverage, thresholds enforced in CI
pnpm coverage:gaps           # locate uncovered lines per file (read-only)
pnpm lint                    # oxlint src test

Project layout:

├── src/                   # engine, tools, commands, memory agent, recall, projection, gate, fate, benchmark, injection + token usage…
├── test/                  # vitest suites (59 files)
├── lib/                   # build output (tsc)
├── docs/
│   ├── design.md          # full design doc (hardening matrix)
│   ├── FAQ.md             # real failure/fix records
│   ├── gap-analysis.md    # vs prime-agent /refine + penguin-harness
│   ├── research/pi-dsh-competitor-gap-analysis.md  # pi/dsh ecosystem competitors
│   ├── experiment-bootstrap.md
│   ├── archive/           # closed point-in-time reports
│   └── research/          # penguin report + prime-agent annotated source
├── examples/README.md     # seed benchmark cases
└── .agents/               # AI collaboration layer (AGENTS.md, skills, ADR notes)

Docs & provenance

• Design: docs/design.md · Pitfalls: docs/FAQ.md · Gap analysis: docs/gap-analysis.md · D2 experiment: docs/experiment-bootstrap.md
• Lineage: penguin-harness (concept; Apache-2.0) — report in docs/research/penguin-harness-self-evolution.md; prime-agent /refine (engineering shape; MIT) — annotated reference source in docs/research/prime-agent-refinement.ts. This package is an original implementation on the DSH plugin surface.

License

MIT