viirya commented on PR #3230: URL: https://github.com/apache/datafusion-comet/pull/3230#issuecomment-3856516072
Looks like `CometNativeShuffleWriter` was added after, I am not familiar with its design. I took a quick look at the code. So it is basically a `ShuffleWriter` and executes a native plan to write inputs from an iterator of `ColumnarBatch` into shuffle file. Although it executes a native plan to write, its input is still coming from JVM even its input plan maybe a native plan too. So this idea is to avoid the JVM/JNI overhead on the input. It sounds good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
