andygrove opened a new pull request, #5114: URL: https://github.com/apache/datafusion-comet/pull/5114
## Which issue does this PR close? Part of #5112. Related to #4440. ## Rationale for this change Isolated benchmarking in #5112 shows the native columnar-to-row converter is 3.7x slower per row than the JVM `CometColumnarToRowExec` at the default 8192-row batch size, and up to 15.7x slower for small batches, because the native path carries roughly 10us of fixed JNI and FFI cost per batch. End-to-end microbenchmarks and the TPC-DS SF1000 comparison in #4440 show no measurable benefit from the native operator, and the original GC-pressure rationale no longer holds since #3367 made the converter deep-copy every row to the JVM heap. Real workloads that reach C2R with small batches (selective filters, small shuffle partitions, broadcast dimension tables) regress, so the safer default is the JVM implementation until the native path is competitive. ## What changes are included in this PR? - Change the default of `spark.comet.exec.columnarToRow.native.enabled` from `true` to `false`, and update the config doc to mention the per-batch JNI cost. - Update the `CometNativeColumnarToRowExec` scaladoc: it claimed zero-copy variable-length data and reduced GC pressure, neither of which holds since the #3308 fix. Users can still opt in with `spark.comet.exec.columnarToRow.native.enabled=true`. ## How are these changes tested? Existing tests cover both code paths explicitly: `CometNativeColumnarToRowSuite` enables the config in its own `withSQLConf` blocks, `CometFuzzTestBase` parameterizes it, and plan assertions in `CometExecSuite` and `CometPlanChecker` match both operator variants, so no test depends on the default. The rest of the suite now exercises `CometColumnarToRowExec` by default, which was the default behavior before #3299. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
