pepijnve commented on PR #19360:
URL: https://github.com/apache/datafusion/pull/19360#issuecomment-3662660485
For cross join specifically, as far as I can tell we're getting stuck
looping in `BuildBatches` state. I can't really explain why the proposed fix
would make that loop exit since the coop wrapper is added around the inner
build batches loop. The task budget should only be getting decremented once and
then still get stuck in the same loop. A more precise fix for this problem
would be to decrement the tokio task budget on each loop iteration explicitly.
```
--- a/datafusion/physical-plan/src/joins/cross_join.rs (revision
2bea7968977fd9e2f78c766ca553a45068c15048)
+++ b/datafusion/physical-plan/src/joins/cross_join.rs (date
1765923164223)
@@ -581,6 +581,7 @@
handle_state!(ready!(self.fetch_probe_batch(cx)))
}
CrossJoinStreamState::BuildBatches(_) => {
+
ready!(tokio::task::coop::poll_proceed(cx)).made_progress();
let poll = handle_state!(self.build_batches());
self.join_metrics.baseline.record_poll(poll)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
