pepijnve commented on issue #18075: URL: https://github.com/apache/datafusion/issues/18075#issuecomment-3422326710
Copying this over from an `arrow-rs` discord discussion. Perhaps we can implement case when more efficiently using interleave. The current implementation strategy builds up a result one branch at a time as in this diagram (I made some mistakes in the labels, but I think the idea is clear enough). <img width="2510" height="2256" alt="Image" src="https://github.com/user-attachments/assets/b71a898c-1a7a-45a4-87a0-a8bbf4d62cc5"; /> What if instead of merging after every branch we keep the partial results separate and instead update an index `Vec<(usize,usize)>` as we go. To be able to do that efficiently, we augment the record batch with a "row number" array that gets filtered along with the record batch itself. That gives us the offsets in the index vec to update. When evaluation is complete everything gets bundled up into the final result array with a single interleave operation. <img width="2092" height="2374" alt="Image" src="https://github.com/user-attachments/assets/86ae8caa-2d82-4146-90c0-84fac8906a21"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
