bamaer opened a new issue, #8415: URL: https://github.com/apache/hop/issues/8415
### What would you like to happen? ### Background Hop has no way to split a long text field into smaller passages. That is the first step of any retrieval pipeline: an embedding model has a bounded context window, and retrieval quality is better on passages than on whole documents. A typical pipeline is chunk, then embed, then write to a vector store. Splitting on a fixed character count is easy to do badly. Cutting mid-word or mid-sentence produces chunks that retrieve poorly, and a sentence that straddles a boundary becomes unfindable from either side. ### Proposal A `Text chunker` transform that splits a text field and emits one output row per chunk, carrying the fields of the input row along with each. The transform has **no dependency on an AI provider or a model**. It is plain text processing and is useful on its own wherever a large document has to be broken into passages, independently of whether the result is ever embedded. #### Chunking strategies | Strategy | Behaviour | |---|---| | Character | Split on a fixed character count, backing off to the nearest word boundary so words are not cut in half. Overlap carries the tail of each chunk into the next, so a sentence crossing a boundary stays retrievable from both sides. | | Paragraph | Split on paragraph breaks, packing whole paragraphs up to the chunk size. A paragraph longer than the chunk size falls back to character splitting. | | Structure | Parse the document into a heading tree and emit one chunk per section, prefixing each with a breadcrumb of its heading path so an isolated chunk still says where it came from. A section larger than the chunk size falls back to character splitting within that section. | #### Structure parsers The Structure strategy selects a parser from the content type: | Content type | Parsed structure | |---|---| | Auto | Infer from the content type field, or from the text itself | | Plain | Blank-line separated blocks | | Markdown | `#` heading levels | | AsciiDoc | `=` heading levels | | Pipeline | A Hop pipeline (`.hpl`): one section per transform, plus the hops between them | | Workflow | A Hop workflow (`.hwf`): one section per action, plus the hops between them | | Metadata | A Hop metadata JSON document | The three Hop-native parsers are the part worth highlighting. They make a pipeline, a workflow or a metadata file retrievable per transform, action or entry rather than as one opaque blob, which is what makes a Hop project itself searchable rather than just the data flowing through it. ### Options All options carry an `injectionKey`, so the transform is fully usable with metadata injection. | Option | Default | |---|---| | Input field | — | | Source document ID field | empty, a row counter is used | | Output chunk field | `chunk_text` | | Chunking strategy | `Character` | | Content type | `Auto` | | Content type field | empty, for a stream that mixes formats | | Chunk size | `1000` | | Chunk overlap | `200` (Character strategy only) | | Include metadata | `true` | | Chunk index field | `chunk_index` | | Chunk start position field | `chunk_start_position` | | Document ID field | `chunk_doc_id` | | Total chunks field | `total_chunks` | The metadata fields matter for anything writing to a vector store: the document id and chunk index are what let a later search merge neighbouring chunks back into a passage, or re-index a single document without touching the rest. Chunk overlap applies only to the Character strategy. Paragraph and Structure split on natural boundaries, where an overlap would duplicate whole paragraphs or sections between neighbours. ### Scope The transform, unit tests, integration tests covering each strategy and parser, and a user manual page. This is **independent of the Vector value type (#8409) and of any AI plugin** — it produces plain text rows and can land on its own. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
