Interface: IMdChunker
Defined in: internal/md-chunker/types.ts:39
Markdown → IChunk[]. Stateless.
Implemented by MdChunker; the type of IAIGatewayDeps.mdChunker, so a host may pass its own in overrides.
Methods
chunk()
chunk(
__namedParameters):IChunk[]
Defined in: internal/md-chunker/types.ts:62
chunk -- the chunks of one document
About
Headings first, size second, code blocks never cut.
One chunk per heading section; a section under minTokens merges with its neighbour (the earlier one when the two
fit, else it is carried into the next); a section over maxTokens is cut at paragraph, line, then sentence
borders, the pieces overlapping by up to overlapTokens. A fenced code block is never cut, so a chunk that holds
one can be larger than maxTokens. Chunk ids are docId#order, deterministic, so the same document gives the same
ids again.
Who needs it
KbIngestor.ingestDocument, once per saved document, before the texts are embedded.
Pitfalls
A document with no heading is one section; a heading with no text of its own gives no chunk but stays in the path of
the headings below it. A whole document under minTokens stays one small chunk.
Parameters
__namedParameters
doc
Returns
IChunk[]
countTokens()
countTokens(
__namedParameters):number
Defined in: internal/md-chunker/types.ts:84
countTokens -- the chunker's own token estimate of a text
About
The same ruler the sizes were cut with.
Characters / 4, rounded up: the estimate maxTokens, minTokens and overlapTokens are measured in, and the
tokens of every chunk.
Who needs it
KbIngestor.ingestDocument, for the tokens of the ingest report; chunk itself.
Pitfalls
An estimate, not the provider's count: a tokenizer counts code and non-latin text differently.
Parameters
__namedParameters
text
string
Returns
number