Skip to main content

Interface: MdChunker

Defined in: internal/md-chunker/MdChunker.ts:78

MdChunker -- splits a Markdown document into the chunks the knowledge base stores

About​

One idea per chunk, under its headings, sized for retrieval.

Implements IMdChunker: a pure transform, Markdown in, IChunk[] out, no tenant, no model call, no state. The gateway's own splitter (D23), no library:

  • One chunk per heading section (parseSections); a section under minTokens merges with its neighbour.
  • A section over maxTokens is cut at paragraph, line, then sentence borders (toUnits), the pieces overlapping by up to overlapTokens; a fenced code block is never cut.
  • Every chunk carries its heading path ("Title > Heading > Subheading"), the document's ids, tags and language, and a token estimate (characters / 4, countTokens).
  • Chunk ids are docId#order, so re-ingesting a document replaces its chunks cleanly.

It is the first step of the ingest path: KbIngestor chunks, the Embedder embeds, the VectorStore stores.

Who needs it​

KbIngestor.ingestDocument (chunk, and countTokens for the report). A host replaces it with its own IMdChunker in overrides.mdChunker.

What needs​

Nothing shared: only its config.

Usage​

When to use it​

Always, through the gateway: createGatewayDeps creates it from IAIGatewayConfig.mdChunker (optional, all defaults).

When shouldn't use it​

Documents that are not Markdown (HTML, PDF text) go through a host chunker that implements IMdChunker, or are converted to Markdown first.

Pitfalls​

The settings are global: every book is chunked alike. A config with minTokens or overlapTokens at or above maxTokens, or a non-positive maxTokens, is a 400 at construction (error 5372621037192). The token estimate is a ruler for sizing, not the provider's count.

Implements​

Methods​

chunk()​

chunk(__namedParameters): IChunk[]

Defined in: internal/md-chunker/MdChunker.ts:123

chunk -- the chunks of one document

About​

Headings first, size second, code blocks never cut.

One chunk per heading section; a section under minTokens merges with its neighbour (the earlier one when the two fit, else it is carried into the next); a section over maxTokens is cut at paragraph, line, then sentence borders, the pieces overlapping by up to overlapTokens. A fenced code block is never cut, so a chunk that holds one can be larger than maxTokens. Chunk ids are docId#order, deterministic, so the same document gives the same ids again.

Who needs it​

KbIngestor.ingestDocument, once per saved document, before the texts are embedded.

Pitfalls​

A document with no heading is one section; a heading with no text of its own gives no chunk but stays in the path of the headings below it. A whole document under minTokens stays one small chunk.

Parameters​

__namedParameters​
doc​

IKbDocument

Returns​

IChunk[]

Implementation of​

IMdChunker.chunk

countTokens()​

countTokens(__namedParameters): number

Defined in: internal/md-chunker/MdChunker.ts:170

countTokens -- the chunker's own token estimate of a text

About​

The same ruler the sizes were cut with.

Characters / 4, rounded up: the estimate maxTokens, minTokens and overlapTokens are measured in, and the tokens of every chunk.

Who needs it​

KbIngestor.ingestDocument, for the tokens of the ingest report; chunk itself.

Pitfalls​

An estimate, not the provider's count: a tokenizer counts code and non-latin text differently.

Parameters​

__namedParameters​
text​

string

Returns​

number

Implementation of​

IMdChunker.countTokens