viirya opened a new issue, #24764: URL: https://github.com/apache/datafusion/issues/24764
### Is your feature request related to a problem or challenge? The memory-limited fallback in `NestedLoopJoinExec` cannot serve join types whose final emission reads the visited-left bitmap (`LEFT`, `LEFT SEMI`, `LEFT ANTI`, `LEFT MARK`, `FULL`) when the right side has more than one partition. Instead of spilling, those queries fail with `ResourcesExhausted`. The cause is that the fallback path builds a **per-partition** `JoinLeftData` for each left chunk with `probe_threads_counter == 1` ([`nested_loop_join.rs`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/joins/nested_loop_join.rs)), while the single-pass path shares one `JoinLeftData` seeded with `collect_left_input(.., probe_threads_count)`. With a private bitmap per partition, each partition sees only its own right rows, so unmatched left rows would be emitted once per partition and rows matched only in another partition would be emitted as unmatched. Because those results are wrong, the fallback is deliberately refused for that combination and the error is surfaced instead: ```rust let left_emission_multi_partition = need_produce_result_in_final(self.join_type) && right_partition_count > 1; let can_spill = context.runtime_env().disk_manager.tmp_files_enabled() && !left_emission_multi_partition; ``` So the limitation is a correctness-preserving guard, not a silent bug — but it means a `LEFT`/`FULL` nested loop join over a partitioned right side has no spilling path at all and simply cannot run under memory pressure. Note that the in-code comment currently points readers to `https://github.com/apache/datafusion/issues/22038`, but 22038 is a **pull request**, not an issue, and no issue tracked this gap. This issue exists to be that tracking issue. ### Describe the solution you'd like Coordinate the left-side visited state across right partitions in the fallback path, so each chunk's `JoinLeftData` (visited bitmap plus probe-thread counter) is shared by every partition exactly as in the single-pass path. Then the gate above can be removed and these join types spill instead of erroring. ### Describe alternatives you've considered - **Keep the guard.** Zero risk, but `LEFT`/`FULL` NLJ over a partitioned right side stays unable to run under memory pressure. - **Force a single right partition** for the affected join types. Preserves correctness and enables spilling, but removes probe-side parallelism for exactly the plans that are already under memory pressure. - **Widen `need_produce_result_in_final`.** Wrong layer: it has ~16 call sites across NLJ / HashJoin / PWMJ, so changing it to influence this decision affects unrelated operators. ### Additional context PR #22038 implements the coordinated approach: a plan-level `FallbackCoordinator` loads each chunk once via a leader partition and publishes it as a shared `Arc<JoinLeftData>` whose probe-thread counter is seeded with `right_partition_count`, so the last partition to finish a chunk emits its unmatched left rows. It has just been rebased onto current `main`. One consequence worth flagging for downstream projects: the coordination assumes all right partitions run **in the same process**. A distributed engine that executes each partition as an independent task would get one coordinator per task, so the shared counter would never reach zero and the fallback would stall. PR #22038 therefore adds `datafusion.execution.enable_nlj_coordinated_fallback` (default `true`) so such engines can opt out and keep today's fail-fast behavior for the affected join types. Related: #24746 (the memory-limited path dropping deferred unmatched probe-side rows when the left side is exhausted) is a separate, already-fixed issue in the same code path; its `EmitGlobalRightUnmatched` routing is preserved by #22038. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
