ewoodbury opened a new issue, #2847: URL: https://github.com/apache/datafusion-ballista/issues/2847
**Describe the bug** A nested-loop join whose type emits build-side rows (`Left`, `Full`, `LeftSemi`, `LeftAnti`, `LeftMark`) returns duplicate or spurious rows, with no error, when Ballista splits its stage into more than one task. Each task tracks matched build rows only for its own slice of the probe side, so it emits build rows as unmatched that matched in another task, emits a semi row once per task it matched in, and a mark row once per task. Both the adaptive and the static planner are affected. **To Reproduce** Standalone cluster with `ballista.scheduler.max_partitions_per_task = 1` (on a real cluster, any split stage does the same), and `orders` read from four CSV files: ```sql SELECT a.o_id, b.o_id FROM orders a LEFT JOIN orders b ON b.o_id < a.o_id AND b.o_id < 2 WHERE a.o_id < 3 ORDER BY 1, 2 ``` DataFusion returns 4 rows. Ballista returns 13: `(0, NULL)` four times, plus `(1, NULL)` and `(2, NULL)` rows that should not exist. A `NOT EXISTS` with a non-equi correlation returns rows the subquery matched. **Expected behavior** The same rows as DataFusion. **Additional context** #1055 is the same problem for broadcast hash joins. Hash joins are guarded (unsafe `CollectLeft` is demoted, null-aware anti joins run their probe in one task); nested-loop joins are not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
