ggjh-159 commented on issue #12459: URL: https://github.com/apache/gluten/issues/12459#issuecomment-4921450372
<img width="1667" height="658" alt="Image" src="https://github.com/user-attachments/assets/6acd3d4b-f70f-4f2c-84c9-5d21be5f01a1"; /> Hi, @lgbo-ustc @zhanglistar @KevinyhZou @Hw-Tiny @Moemenmohamed24. This describes the changes in data format following the addition of `RowKind`. The `MergedRowVector` differs from the standard `RowVector` by including an extra `$row_kind` column at the end, while `ChangelogRowVector` is a new data structure defined in Velox that encapsulates the original `RowVector` along with the new `$row_kind` column. - In **Gluten-Flink**, `MergedRowVector` is used for both input and output, and the component handles the conversion to and from `RowData`. - **Velox4j** is responsible for converting between `MergedRowVector` and `ChangelogRowVector`. - On the **Velox** side, the system processes `StreamRecord` objects that encapsulate a `ChangelogRowVector`; these records are passed along the operator chain, with individual operators selectively processing them based on their specific functions. - Specifically, within the stateless `Calc` operator, the `ChangelogRowVector` must be converted into a `MergedRowVector` for processing by the Velox `FilterProject` operator (which performs filtering and projection); once the result is obtained, it is re-wrapped as a `ChangelogRowVector` for output. The `RowKind` column is implicitly added to the operator chain, with conversion logic implemented in the Input/Output Bridge and Velox4J; the execution plan remains unaware of this column's existence. Do you consider this an appropriate approach for adding `RowKind` column support to enable retraction semantics? Or are there other aspects that require further discussion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
