CLI tool + Rust library — local content-addressed store for large binary artifacts

Qxet

OVERVIEW

QXet ("Quality AI eXchange") is content-addressed storage for large binary artifacts (model weights, datasets, checkpoints): files are chunked with FastCDC content-defined chunking, every chunk is addressed by its SHA-256, identical chunks are stored once across all artifacts, and every read verifies hashes and fails closed on any mismatch — "git semantics for large binary blobs — local-first, no network." Status is v0.1.0; remotes, the Hub protocol, and Python bindings are v0.2 and do not exist yet.

HOW IT WORKS

1Create a store with `qxet init`, which makes `.qxet/` (chunks/, manifests/, config.toml) in the current directory.
2Store a file with `qxet put <file>`: it is FastCDC-chunked, each chunk SHA-256-addressed and deduplicated, a manifest is written, and the content ID (`qxet:<sha256>`) is printed.
3List what is stored with `qxet list` (content ID, name, size, chunk count, stored bytes, dedup ratio).
4Retrieve with `qxet get <id> [-o <path>]`: chunks are reassembled, every chunk hash is re-verified, and output is renamed into place only after the whole artifact verifies.
5Audit the store with `qxet verify`, which walks all manifests and rehashes every chunk.
6Reclaim space with `qxet gc` — a dry run by default; `--apply` actually deletes unreferenced chunks.

How it works, step by step — each step as the project's own documentation states it.

CONCEPTS

Content-defined chunking at Qxet's configured parameters (FastCDC; 16 KiB minimum / 64 KiB average / 256 KiB maximum): a rolling Gear hash picks the cut points, each chunk is stored once under its SHA-256 in a 2-hex fan-out, and the manifest records the ordered chunk ids under the artifact's content ID.

h_{i} = (h_{i-1} \ll 1) + G[b_{i}]

h_i is the rolling Gear fingerprint after consuming byte b_i of the stream; G is FastCDC's fixed table of 256 64-bit words, indexed by the current byte; ≪ is a left bit shift, the bit shifted out discarded. Each step is one shift and one add, so the fingerprint over the scanning window updates in constant time per byte — the property that makes finding cut points in a multi-gigabyte artifact cheap.

The Gear rolling hash, from the FastCDC paper — the fingerprint the cut test reads.

(h_{i} \mathbin{\&} M_{i}) = 0 \iff \text{cut at } i

M_i is M_S (the strict mask) while the current chunk is shorter than the target average and M_L (the lenient mask) at or beyond it; the strict mask sets more bits, so demanding (h & M) = 0 is rarer early and more common later — FastCDC's normalized chunking, which narrows the chunk-size distribution around the average. In Qxet's configuration the average is 64 KiB; independently of the masks, cuts are suppressed before the 16 KiB minimum and forced at the 256 KiB maximum.

The normalized cut rule: the mask relaxes once the chunk passes its target size.

CONFIGURATION

Option	Default	What it does
`FastCDC min chunk size`	16 KiB	Minimum content-defined chunk size, recorded in the store's `config.toml`
`FastCDC avg chunk size`	64 KiB	Average content-defined chunk size
`FastCDC max chunk size`	256 KiB	Maximum chunk size; also the largest single in-memory chunk buffer
`Store config file`	.qxet/config.toml	Records the chunking parameters the store was built with; a store refuses to open under a build with different parameters
`Store layout`	.qxet/chunks/ab/cdef...`, `.qxet/manifests/<id>.json	Chunk bytes named by SHA-256 with 2-hex fan-out; canonical JSON manifests named by content ID
`MAX_CHUNKS_PER_ARTIFACT`	1,048,576	Most chunks one artifact may reference
`MAX_MANIFEST_BYTES`	128 MiB	Largest manifest the store will read or write
`MAX_MANIFEST_COUNT`	100,000	Most manifests one store operation will walk
`MAX_NAME_BYTES`	255	Longest artifact name, in bytes
`Toolchain pin`	stable Rust 1.99.0	Pinned in `rust-toolchain.toml`

CONTROLS

Action	Control
Initialise a store	`qxet init [dir]`
Store a file	`qxet put <file>`
Retrieve an artifact	`qxet get <id>` (`qxet:` prefix or bare hex; `-o/--out <path>` for an explicit output file)
List artifacts	`qxet list`
Verify the whole store	`qxet verify`
Garbage-collect (report only)	`qxet gc` (dry run by default; `--dry-run` says so explicitly)
Garbage-collect (delete)	`qxet gc --apply`
Select a store explicitly	`--store <path>` (global flag; otherwise the store is found by walking up from the current directory)
Format check (development)	`cargo fmt --check`
Lint (development)	`cargo clippy --workspace --all-targets -- -D warnings`
Test (development)	`cargo test --workspace` (39 tests at v0.1.0)

NOT COVERED

- No install/distribution command is documented (no `cargo install`, release binary, or package); the README covers usage and development commands only. - `hf_xet/` (pyo3 bindings) and `data/`/`xtool` are reference fragments excluded from the workspace that do not compile; nothing in v0.1 depends on them. - The sandbox clone at `~/workspace/repos/qxet` is stale (no README/Cargo workspace, only `DESIGN.md` plus the fragments); this pack is based on the GitHub state, which matches `DESIGN.md`'s v0.1 command table.

REFERENCES

FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication (Xia et al., USENIX ATC 2016)The defining paper for the chunking Qxet implements: the Gear rolling hash and normalized chunking above.
FIPS 180-4 — Secure Hash Standard (SHA-256)The hash that names every chunk and every artifact in the store; reads re-verify against it and fail closed on mismatch.