ggjh-159 commented on issue #12459: URL: https://github.com/apache/gluten/issues/12459#issuecomment-4955890086
> Thanks for the detailed design. and I have some questions: > > * 1. The flink rowdata convert to velox's `ChangelogRowVector`, does this means that：the input rowdata is generated by flink operators (like source), and then convert to velox data structure? as we have implemented some native operators(kafka/pulsar source) in velox side, should we use these to process the data directly without these convertions? > * 2. `ChangeRowVector` is a new data structure, can we use `StatefulRecord` that has been introduced in `Velox` by adding a new field `RowKind` instead of using this new one? However, returning to the `StatefulRecord` representation, adding a column to it might be clearer than converting a `ChangelogRowVector` into a `MergedVector`. Velox4j and Gluten still use `StatefulRecord` for data conversion, simply adding the conversion of an extra variable (the `RowKind` column). The downside of this approach, however, is that serialization and deserialization must be performed twice—once for the data and once for the `RowKind` column—whereas previously, only a single pass was required for the `MergedVector`. I believe that adding an extra column to `StatefulRecord` is a better approach than converting to `MergedVector`. @KevinyhZou What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
