malinjawi commented on PR #13158: URL: https://github.com/apache/gluten/pull/13158#issuecomment-6019054668
@felipepessoto thanks, and agreed, the fallback is a MERGE regression. It moves the write projection to Spark, and with CDF on it also keeps a row-to-columnar hop on every written row, so this PR should be read as the minimal correctness fix, not the end state. The full native version is up as #13165, stacked on this one. It needs no Velox change: Velox already counts the rows each expression is evaluated on, and `FilterProject` exports those counts per function name once `operator_track_expression_stats` is on. Each Delta metric becomes a non-deterministic pass-through function named after the metric, the counts come back through the operator stats, and the write projection stays on Velox with exact counters, CASE WHEN branches included. Both PRs are green on the Delta UT gate. We measured the two against each other on a 10M-row Delta target with CDF on, 120 cores, seven runs per case. Native is 12% faster on a mixed MERGE with a 1% source, 9% faster on a no-op MERGE with a 1% source, and within noise at a 10% source where the join dominates. Rows, CDF output and all persisted counters matched a Spark oracle on every run. Details are in the #13165 description. Happy to go either way: merge this one first as the safe step and rebase #13165 on top, or review #13165 directly. One thing to know: keeping the projection native re-exposes a pre-existing struct field-order bug in MERGE schema evolution that this fallback happens to hide. I will open that fix separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
