andygrove commented on issue #3762: URL: https://github.com/apache/datafusion-comet/issues/3762#issuecomment-5860286212
The current-path description here predates #4572. `CometBatchIterator` and `arrow_ffi_safe` are gone. Today the broadcast is decoded on the JVM by `Utils.decodeBatches` and re-exported to native once per partition through the Arrow C Stream, via `CometArrowStream.wrapColumnarBatchRDD`, `ColumnarBatchArrowReader` and `ScanExec`. The per-column export no longer exists, but the JVM-side decompression and decode, and the stream hop, still do, so the proposal still stands. #5538 implements it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
