ranflarion opened a new pull request, #24675:
URL: https://github.com/apache/datafusion/pull/24675
## Which issue does this PR close?
- Closes #22641.
## Rationale for this change
The memory-limited fallback returns wrong rows for join types whose final
emission reads the visited-left bitmap (LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK)
when the probe side has more than one partition. The fallback rebuilds a
per-partition `JoinLeftData` per left chunk with `probe_threads_counter == 1`
and a per-partition bitmap, so every partition believes it is the last to
finish probing and emits its own unmatched set, from a bitmap that only saw its
own right rows. Unmatched rows are emitted once per partition, and rows matched
only in another partition are emitted as unmatched.
Proof, using this file's own `build_table` / `prepare_join_filter` /
`multi_partitioned_join_collect` fixtures (4 right partitions) with a 100-byte
memory limit versus unbounded:
```rust
for join_type in [JoinType::Left, JoinType::LeftAnti] {
// unbounded memory -> correct; 100-byte pool -> memory-limited fallback
let (_, correct, _) = multi_partitioned_join_collect(left, right,
&join_type, Some(filter), ample_ctx).await?;
let (_, wrong, _) = multi_partitioned_join_collect(left, right,
&join_type, Some(filter), tight_ctx).await?;
}
// Left: correct=19 memory_limited=49
// LeftAnti: correct=1 memory_limited=31
```
The existing gate covers only FULL. This extends it to every
`need_produce_result_in_final` type, so an over-budget build side fails with
`ResourcesExhausted` instead of returning wrong rows. Right-side emission types
(RIGHT, RIGHT SEMI, RIGHT ANTI, RIGHT MARK) keep the fallback: each partition
owns its right rows exclusively, so their bitmaps are complete per partition.
Proper cross-partition coordination of the left bitmap, which would re-enable
the fallback for these types, is #22038.
## What changes are included in this PR?
The `full_join_multi_partition` condition in `NestedLoopJoinExec::execute`
becomes `need_produce_result_in_final(self.join_type) && right_partition_count
> 1`, with the comment updated to describe the failure mode.
`test_overallocation` moves LEFT/LEFT SEMI/LEFT ANTI/LEFT MARK from the
succeed-via-fallback group (which collected but never checked the row values)
into the must-OOM group alongside FULL.
## Are these changes tested?
`test_overallocation` now asserts the `ResourcesExhausted` refusal for all
five gated join types with a multi-partition probe side, and still asserts
fallback success for Inner and the right-emission types. All 42
`nested_loop_join` tests pass; `./dev/rust_lint.sh` is clean.
## Are there any user-facing changes?
Left-family nested loop joins with a multi-partition probe side whose build
side exceeds the memory budget now fail with `ResourcesExhausted` instead of
returning incorrect results. No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
