Skip to main content

The smart chunker

Retrieval quality is only as good as your chunks. A system that slices text every N characters produces chunks that start mid-sentence, split functions across boundaries, and lose track of which heading a passage belongs to. Midnight Manual doesn't do that.

The chunker understands the structure of what it reads (Markdown heading hierarchy, programming-language syntax, package membership) and splits on real boundaries. Every search hit lands on a named thing, not an arbitrary window.

Markdown: heading-aware chunks​

Markdown files are parsed with pulldown-cmark and split along the heading hierarchy. A level-2 section stays together. A long level-3 subsection is split at a natural boundary rather than mid-sentence.

Every chunk carries its heading_path: the full chain of ancestor headings from the document root down to the chunk. A hit at /docs/intro.md from inside ## Installation > ### macOS carries exactly that path, so your AI assistant knows where in the document outline the passage lives.

Code: semantic, symbol-aware chunks​

Source files are parsed with tree-sitter and split on real syntactic boundaries (functions, classes, impl blocks, modules), never mid-expression.

Every code chunk records a structured symbol_path, such as impl Widget › fn render. Search hits on code land on a named symbol, not an arbitrary window of lines. This matters for retrieval: ask "how does Widget render?" and you find the render method, not a fragment that happens to mention "render" in a comment on line 147.

Supported languages​

Compact .compactRust .rsTypeScript .ts .tsx
JavaScript .js .jsx .mjs .cjsPython .py .pyiGo .go
Solidity .solJava .javaC# .cs
Kotlin .kt .ktsSwift .swiftRuby .rb
Haskell .hsBash .sh .bashScheme .scm .ss .sld
TOML .tomlYAML .yaml .ymlHTML / XML

Grammars are organized into Cargo-feature tiers (core-grammars, markup-grammars, extended-grammars, all-grammars), so a lean build stays small.

Compact symbol awareness​

Midnight's smart-contract language gets full symbol awareness: circuits, ledger declarations, witnesses, and contracts each become their own semantically-bounded, attributable chunk. That comes from the compactp parser (a default-on feature) rather than tree-sitter, because Compact's grammar predates general tree-sitter support.

Graceful degradation for unknown languages​

When a grammar is absent or a language is unrecognized, the chunker falls back to a token-budgeted, non-overlapping line-window chunker: it grows line-by-line to approximately 90% of the token budget, then starts a new window. The file is still ingestible and searchable; it just won't have symbol paths. An absent grammar never aborts an ingest run.

Implementation details​

Token-budgeted chunks​

Chunks target a token budget (default: 1024 tokens) so they fit the embedder comfortably. Token counts come from a subword tokenizer (a vendored BGE tokenizer), not a character-count heuristic. That keeps chunks from being silently truncated by the embedding API, and the resulting embeddings represent the chunk's full content.

.gitignore-aware file discovery​

File lists are built with the ignore crate and follow a clear precedence ladder:

  1. .git/ is always excluded.
  2. Built-in skips: node_modules, target, vendor, dist, *.min.js, and similar.
  3. .gitignore / .ignore rules in the repository.
  4. Your manifest node's exclude: globs.
  5. Your manifest node's include: globs — when set, a file must match one of them to be kept. This is a whitelist that narrows what's ingested; it does not rescue a file that an exclude: glob already dropped (exclude beats include).

This means a standard Midnight project ingests cleanly without any manifest configuration: the chunker already knows to skip compiled output, lock files, and generated code.

Package membership​

Walking up from each file, the chunker attaches package membership: the name of the nearest Rust crate (Cargo.toml [package], workspace roots skipped) or npm package (package.json .name). Search results can be filtered and attributed by package, so "find the deployContract function in @midnight-ntwrk/midnight-js-contracts" actually works.

Per-file failure isolation​

A badly malformed source file falls back to line-window chunking and is flagged in the ingest report rather than aborting the whole run. Chunks that fail to embed (for any reason) land in an embed_failed state and are skipped by readers, so navigation has clean gaps, never broken links.