ranflarion opened a new issue, #24676:
URL: https://github.com/apache/datafusion/issues/24676
### Describe the bug
`NestedLoopJoinExec`'s memory-limited fallback re-executes the build-side
child. When the in-memory load fails with `ResourcesExhausted`,
`initiate_fallback` calls `self.left.execute(0, ctx)` a second time and spills
that stream to disk — but the first execution already consumed part (or all) of
the input. For an input that cannot be executed twice, the batches the first
pass consumed are gone:
- If the second execution yields nothing, the fallback fails with
`Internal("Left side produced no data to spill")`.
- If the second execution resumes a shared stream (e.g. an `ExecutionPlan`
wrapping an external one-shot reader — FFI sources, engines embedding
DataFusion that feed operators from a network or JVM iterator), the join
silently returns only the rows the first pass had not yet consumed. We observed
this embedding DataFusion 54.1.0: a 200k-row build side against an 8MB pool
returned 134,464 rows with no error.
Even for replayable inputs the build side is fully executed twice, paying
the child's cost again just to write it to disk.
### To Reproduce
Unit-shape repro against `main`, in `nested_loop_join.rs`'s test module:
wrap the existing `build_left_table()` in an `ExecutionPlan` whose `execute`
returns the real stream on the first call and an empty stream afterwards, then
run any join through `join_collect` with a 50-byte memory limit (as
`test_nlj_memory_limited_inner_join` does). The fallback engages, re-executes
the child, gets the empty stream, and fails:
```
Internal("Left side produced no data to spill")
```
With a stream that resumes instead of restarting, the same shape loses the
consumed prefix silently.
### Expected behavior
The fallback should spill from the pass that already consumed the stream:
when a reservation fails, write the batches buffered so far plus the remainder
of the same stream to the spill file, instead of executing the child again.
This executes the build side exactly once whether or not it fits, works for
non-replayable inputs, and removes the double execution for replayable ones.
`SpillState::Pending` then no longer needs to carry the child plan.
We run this design in production against DataFusion 54.1.0 and can offer it
as a PR.
### Additional context
Related: #15760 (memory-limited NLJ), #22038 (cross-partition left bitmap),
#23076 (concat memory amplification in the same load path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
