DanielLeens commented on issue #9717: URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-4934221733
Thanks for the detailed proposal and for picking this up, @ss666 — and thanks @Hisoka-X for the earlier design guidance. Your instinct on the output shape is correct. For a RAG pipeline where each chunk needs to be independently embedded and stored as a separate vector record, one-chunk-per-row is the right model. Extending `AbstractCatalogSupportFlatMapTransform` and using `flatMap` (1-row→N-rows) is the better fit here. The earlier `MultipleFieldOutputTransform` suggestion was reasonable at the time because the initial scope discussion hadn't pinned down the output shape. But as you noted, it is a 1-row→1-row map, so it would only emit a single `ARRAY<STRING>` column — which is not what most RAG downstreams expect. A few notes on the proposed design: 1. The option set (`text_field`, `output_field`, `chunk_size`, `overlap_size`, `separators`) following Dify general mode looks reasonable for a first version. I'd suggest also thinking about what happens when `separators` is empty or not provided — a fallback chunking strategy (fixed-size with overlap) would be useful. 2. For `flatMap`-based transforms, checkpoint/state behavior is straightforward since the transform is stateless per row, which is good. Just make sure the output `CatalogTable` correctly declares the produced schema (a single `STRING` column for the chunk, plus optionally carrying over source metadata fields if needed). 3. Package name `org.apache.seatunnel.transform.chunk` is fine. The existing transform packages follow a similar flat structure (`split`, `copy`, `jsonpath`, etc.). 4. One thing worth considering: should the transform preserve the original document identifier or index in the output, so downstream stages can correlate chunks back to their source document? This is common in RAG pipelines. It could be an optional `keep_source_field` or similar. Since @iinow has explicitly stepped away from this issue and there has been no activity for several months, I've removed the old assignment and added the `feature` label. Unfortunately, GitHub does not allow assigning @ss666 to the issue at this time (this typically means the contributor has not yet been added as a collaborator). That's a minor technicality — please go ahead and open the PR, and we can sort out the assignment once it's open. Looking forward to seeing the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
