dsh-analyze

dsh-analyze

English | 中文

Local dataset analysis tools for DSH.
Ingest a CSV/JSONL once, keep large tables on disk, and let the model work through small JSON tool calls instead of dumping entire files into context.

Without dsh-analyze          With dsh-analyze
─────────────────────        ─────────────────────────────
paste whole CSV → tokens     dataset_ingest  → { datasetId }
context blows up             dataset_head    → 5 sample rows
no audit trail               stats_describe  → column stats
                             transform_agg   → grouped result
                             (full table stays on disk)

Why DSH needs a local compute engine

DSH runs agents that reason through tool calls, not notebooks. Real analysis sessions often start from a workspace file (exports, logs, CSV dumps) that is far larger than the context window — and the agent still needs schema, distributions, filters, joins, and aggregates to answer the user.

Without a dedicated engine, hosts usually fall into one of three traps:

Trap	What happens
Paste / cat into context	Multi-MB tables blow the window, truncate mid-answer, and burn tokens on data the model never needs in full.
Unbounded shell + Python	pandas in the agent shell works until it does not: no row/byte caps, no path sandbox, no stable JSON contract, hard to audit or harden.
Full SQL / warehouse	Right for live multi-tenant data; wrong for “here is a local file in this session workspace” with zero network.

dsh-analyze is the missing middle layer for DSH sessions:

1. On-disk dataset store — ingest once, keep the table under <workspace>/.dsh-analyze/, address it as a datasetId.
2. Small JSON tools — head / describe / values / sample / agg / join return tiny structured results (samples ≤ 20 rows; agg limits apply).
3. Hard, explicit limits — byte/row caps, path confinement, whitelist expressions (no arbitrary code), fail-soft plugin registration.
4. Offline by design — pure-std Rust binary; no cloud, no drivers.

The model explores like an analyst (schema → sample → stats → grouped aggregates → join) while the full table never has to live in context, and every step is a bounded tool call the host can log and constrain.

What it is (and is not)

Is	Is not
Local file dataset handles (CSV / JSONL)	A SQL database connector
Offline Rust CLI + DSH tool plugin	A Python/R scientific stack
Session-workspace sandbox with hard limits	A multi-tenant cloud service

Install

dsh plugin --profile web add dsh-analyze

Place the dsh-analyze binary on PATH, or set:

export DSH_ANALYZE_CLI_PATH=/path/to/dsh-analyze

Prebuilt binaries

Download from GitHub Releases and verify against SHA256SUMS:

Target	Artifact
x86_64-pc-windows-msvc	dsh-analyze-x86_64-pc-windows-msvc.zip
aarch64-apple-darwin	dsh-analyze-aarch64-apple-darwin.tar.gz
x86_64-apple-darwin	dsh-analyze-x86_64-apple-darwin.tar.gz
x86_64-unknown-linux-gnu	dsh-analyze-x86_64-unknown-linux-gnu.tar.gz

CLI lookup order: DSH_ANALYZE_CLI_PATH → legacy INV_ANALYSIS_CLI_PATH → dsh-analyze on PATH → plugin logs a warning and registers nothing.

Build from source

cargo build --release -p dsh-analyze

The crate is pure-std and builds offline.

Tools

Tool	Purpose
dataset_ingest	Load a workspace CSV/JSONL → datasetId + schema summary
dataset_head	Up to 20 sample rows
dataset_schema	Column names, dtypes, row count
dataset_list	All datasets in this workspace store
dataset_export	Write full table to CSV under outputs/
dataset_rm	Permanently remove a dataset
stats_describe	Null counts, min/max/mean/std/quartiles
transform_agg	Filter / mutate / project / group-by / aggregate (incl. time buckets)
dataset_values	Top value distribution for one column (nunique, top)
dataset_sample	Deterministic random or stratified sample (≤ 20 rows)
dataset_join	Equi-join two datasets → new on-disk dataset (ids + counts only)
dataset_profile	Quality profile: nulls, nunique, type stability, suspect keys
dataset_dedup	Exact or keyed duplicate detection (counts + sample groups)
dataset_outliers	IQR / z-score / MAD outliers; optional write of outlier dataset
dataset_corr	Pairwise Pearson/Spearman correlations among numeric columns (top pairs by |r|)
dataset_crosstab	Two-column contingency table: top categories, matrix, margins, top cells
dataset_series	Time-series transforms (lag/lead/diff/pct_change/rolling) → new dataset

CLI mirrors the same names as subcommands (values, sample, join, profile, …). See docs/interface.md for the frozen JSON contract.

Quality tools (v0.1.3)

• dataset_profile: one-call “can I use this table?” summary.
• dataset_dedup: omit keyColumns for whole-row exact dupes; pass keys for export-style collisions.
• dataset_outliers: method: iqr|zscore|mad; outLabel writes a new dataset — never dumps full outlier lists.
• stats_describe: also returns q01 / q10 / q90 / q99 on numeric columns.

Relation tools (v0.1.4)

• dataset_corr: 2–16 numeric-capable columns; method: pearson|spearman; pairs sorted by |r| (cap 50).
• dataset_crosstab: rows × cols frequency table with top categories and top cells only.
• dataset_join: how now includes anti | semi (left rows only; null keys never match).

Series tools (v0.1.5)

• dataset_series: one order-by column (nulls last); one-call lag / lead / diff / pct_change / rolling (mean|sum|min|max).
• Always writes a new dataset; response is ids + schema only — inspect with dataset_head / stats_describe.

transform_agg extras (v0.1.2)

• mutate: derived columns with a whitelist expression ([A-Za-z_][A-Za-z0-9_]*, numbers, + - * /, parentheses). No function calls. Invalid expr → analysis/invalid-args.
• Time groupBy: {"dateColumn":"created_at","period":"day|week|month"} → column created_at_day etc. Unparseable non-null values go to the null bucket and increment parseFailCount (never silently dropped).

Limits & data root

<workspace>/.dsh-analyze/{datasets,outputs,tmp}/

Env	Default	Meaning
DSH_ANALYZE_CLI_PATH	—	CLI executable path
DSH_ANALYZE_WORKSPACE	process.cwd()	Session workspace root
DSH_ANALYZE_MAX_ROWS	5,000,000	Ingest row cap
DSH_ANALYZE_MAX_INGEST_BYTES	256 MiB	Ingest byte cap
DSH_ANALYZE_DEFAULT_ROW_LIMIT	1000	Agg default row limit
DSH_ANALYZE_DEFAULT_SAMPLE_SEED	42	Sample seed when omitted

Legacy INV_ANALYSIS_* names are still accepted (new name wins).

Results returned to the model are intentionally small (samples ≤ 20 rows). Full tables go through outLabel / dataset_export and stay on disk.

Comparison

	dsh-analyze	SQL connectors	Notebook stacks (Python/R)
Input	Local CSV/JSONL	Live database	Anything
Network	Never	Required	Often
Model token cost	Small JSON per step	Query results	Kernel output
Runtime	Single static binary	Drivers + server	Full interpreter

Security

Path confinement, credential-path deny lists, env whitelist, and hard limits — see SECURITY.md and docs/security.md.

License

MIT — see LICENSE.