Rich-T-kid commented on PR #11174: URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6223309703
Working on this at the moment. The issue is that to the size of the indices isn't a very useful hint. We'd need to traverse the entire indices array once & then make a decision but this might drag down the performance for low selectivity cases. The goal of this new method is to replace the take + push batch, that repartitionExex does so the low selectivity speed ups should be prioritized. Since data is typically 1/8 or 1/16 or 1/32.... for output partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
