ss666 commented on issue #9717: URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-4933269652
Hi, @Hisoka-X and @davidzollo, I'd like to pick this up. ### Proposed design - Plan to add a `TextChunk` transform under a new `org.apache.seatunnel.transform.chunk` package in `seatunnel-transforms-v2`. - Plan to add these options: `text_field`, `output_field`, `chunk_size`, `overlap_size`, `separators`，following Dify [general mode](https://docs.dify.ai/en/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text#general-mode). ### Question to confirm - Output shape The earlier discussion suggested extending `MultipleFieldOutputTransform`, but that is a 1-row→1-row map, so a document could only emit **its chunks as a single `ARRAY<STRING>` column**. For a typical RAG pipeline, though, I think the output should be **one chunk per row**, so that each chunk can be embedded and stored as an independent vector record downstream. That means we should instead extend `AbstractCatalogSupportFlatMapTransform` and use `flatMap` (1-row→N-rows) to emit one `STRING` chunk per row. Once confirmed above, I'll go ahead and open the PR against the latest `dev`. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
