adriangb commented on PR #9578: URL: https://github.com/apache/arrow-rs/pull/9578#issuecomment-4092762488
> > > Probably won't show up in benchmark, but still would be nice. A more general solution would be https://github.com/apache/arrow-rs/pull/9414/changes perhaps > > > > > > Should I spend time reviewing that one instead / in addition? I am losing track of all the PRs > > > > Probably won't show up in benchmark, but still would be nice. A more general solution would be https://github.com/apache/arrow-rs/pull/9414/changes perhaps > > > > > > Should I spend time reviewing that one instead / in addition? I am losing track of all the PRs > > Perhaps I/we want to first test it in DataFusion? @adriangb also wanted to do aptive filtering which could be similar or work together. My intuition (substantiated by some now somewhat outdated benchmarks) is that with morselization and using morsels as the unit of adaptivity we can and probably should do all of the adaptive filter execution in DataFusion. I plan to fast follow the morselization work with the adaptive filtering work because (1) it will work much better with the smaller units of adaptivity, (2) I don't want to bog reviewers down with too many options but (3) I need to make sure that any API changes needed (I expect few to none) can be done before the API changes from morsels are set in stone. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
