ggjh-159 commented on issue #12459: URL: https://github.com/apache/gluten/issues/12459#issuecomment-4931285431
> Thanks for the detailed design. and I have some questions: > > * 1. The flink rowdata convert to velox's `ChangelogRowVector`, does this means that：the input rowdata is generated by flink operators (like source), and then convert to velox data structure? as we have implemented some native operators(kafka/pulsar source) in velox side, should we use these to process the data directly without these convertions? > * 2. `ChangeRowVector` is a new data structure, can we use `StatefulRecord` that has been introduced in `Velox` by adding a new field `RowKind` instead of using this new one? @KevinyhZou Thank you for your questions, here are the corresponding answers: i. The data conversion here does not introduce a new path; instead, it adds a hidden column—`$row_kind`—to the existing data. The C++ native source should directly support constructing and outputting a `ChangelogRowVector`. In scenarios where C2R (Columnar-to-Row) or R2C (Row-to-Columnar) conversions are not required, data should flow continuously through the C++ operator chain without being converted to Flink `RowData`. ii. Velox does not have a direct implementation of `StatefulRecord`. The existing `StatefulRecord` in velox4j corresponds to Java's `StreamRecord`, whereas the new `ChangelogRowVector` should correspond to `GenericRowData`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
