laxmanmamididd opened a new pull request, #7328: URL: https://github.com/apache/datafusion-comet/pull/7328
## Which issue does this PR close? Backport of #6900 to `branch-1.1`. Closes #6779 on this branch. ## Rationale for this change `branch-1.1` has the bug fixed on `main` by #6900: a broadcast hash join drops matching rows when the build side emits sliced string or binary batches. Native `DISTINCT`, grouped aggregates and window operators emit such batches, and `coalesceBroadcastBatches` appends them with `VectorSchemaRootAppender`, which assumes offsets that start at zero. We hit it on a production Spark job on Comet 1.1.0. Output was missing 24 of 724,405 rows. At an inner join on a 16-character string key, vanilla Spark produced 30,410 rows and `CometBroadcastHashJoin` produced 30,407 from identical inputs. With `spark.comet.exec.broadcastHashJoin.enabled=false` the output matched vanilla exactly. ## What changes are included in this PR? Cherry-pick of 66c63bd307b1 (#6900). The `ffi.md` hunk is placed above this branch's "Wrapper Object Lifecycle" heading because the heading differs from `main`. ## How are these changes tested? `UtilsSuite` passes on `branch-1.1` with `-Pspark-4.0` on JDK 17, and the new test `serializeBatches rebases offsets that start past zero` fails without the change to `Utils.scala`. `CometJoinSuite` needs the native library and has not been run locally; CI will run it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
