jayzhan211 opened a new pull request, #25490:
URL: https://github.com/apache/datafusion/pull/25490
## Which issue does this PR close?
- No separate issue. Split out of #25217 so it can be reviewed on its own.
## Rationale for this change
#25217 lets a hash join that runs out of memory finish as a sort-merge join:
it sorts both inputs itself and then needs to run them through the same join
streams `SortMergeJoinExec` uses. Today the only way to build those streams
is
inside `SortMergeJoinExec::execute`, which starts from two child
`ExecutionPlan`s rather than two already-sorted streams.
This PR separates "join two sorted streams" from "execute my children", so
the
first half can be reused. It is a refactor with no new caller yet; it is
split
out to keep the sort-merge join changes out of the (much larger) hash join
PR.
## What changes are included in this PR?
All in `sort_merge_join/exec.rs`:
- New `pub(crate) fn sort_merge_join_stream(SortMergeJoinInputs, &metrics,
&context)`.
It holds what `execute` did after executing its children: pick the streamed
and buffered side by join type, register the (spillable) memory consumer,
build the spill manager, and choose `BitwiseSortMergeJoinStream` or
`MaterializingSortMergeJoinStream` by join type family.
- New `pub(crate) struct SortMergeJoinInputs` carrying the two sorted
streams,
join keys, filter, join type, sort options, null equality, schema and
partition.
- `SortMergeJoinExec::execute` keeps the partition-count check, executes both
children, calls the new function and applies the embedded projection.
Nothing is `pub`, so there is no public API change.
One ordering detail: `execute` used to call `execute()` on the streamed child
first and the buffered child second; it now calls left then right, because
the
side selection moved into the shared function. For right-family joins that
swaps the order of the two calls. Both still happen before anything is
polled.
## What is the testing strategy for this PR?
No behaviour change is intended, so this relies on the existing
`sort_merge_join` unit tests, which all pass unchanged. That includes
`stream_registers_as_a_spillable_consumer` and
`fair_spill_pool_leaves_room_for_the_streamed_sort` from #25250, which check
that the memory consumer registered by the moved code is still spillable.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
