ewoodbury opened a new issue, #2849: URL: https://github.com/apache/datafusion-ballista/issues/2849
**Describe the bug** With the static planner, `lower_null_aware_join` puts an inline `CoalescePartitionsExec` on the probe side of a null-aware anti hash join so the join runs in one task. When an aggregate sits above the join, the `EnsureRequirements` pass run on the join's stage removes that coalesce, since the join does not require it. The stage is then split into tasks and returns wrong counts, with no error. **To Reproduce** Standalone cluster with `ballista.planner.adaptive.enabled = false`, `datafusion.optimizer.prefer_hash_join = true`, `ballista.optimizer.not_in_subquery_rewrite = false`, and `ballista.scheduler.max_partitions_per_task = 1`. Run a `GROUP BY` over a `NOT IN` subquery on a multi-file table. Counts came back as 1190/1155/1155 against 170/165/165 from DataFusion. **Expected behavior** The same counts as DataFusion. **Additional context** The nested-loop join fix in #2848 avoids this by giving the probe side a one-partition stage of its own instead of an inline coalesce. The same approach should work here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
