zhuqi-lucas commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2951559003
> I found some time to work on this tonight and it looks good to me now. > > To summarize where we are: > > * We add yields to all leaf nodes, but no yields to any intermediate node. > * We added a bunch of tests to cover some corner cases and all of them pass. > * There is a single new `with_cooperative_yields` API, which returns a cooperatively yielding version of a plan object (if it exists). If it doesn't exist for a leaf node, we add an auxiliary operator to handle yielding. > > Future work: > > * We will study input-side pipelining behaviors and improve the pipelining API, so that we only trigger explicit yielding when it is necessary. Given the small number of leaf nodes, we are not that far off from optimality even as is, which is great. We have some ideas on what to try here, but the current state seems quite good -- so we can merge it to fix downstream issues as we make further progress. > * We will think about supporting cases involving non-volcano (i.e. spill) data flow. > > @zhuqi-lucas and @alamb, PTAL Thank you , i agree that we are in good state, because this PR can help both datafusion operator and also custom defined operator automatically. I will also help investigate following case, may be as a follow-up ticket, thanks! > We will think about supporting cases involving non-volcano (i.e. spill) data flow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
