Rich-T-kid commented on PR #9865: URL: https://github.com/apache/arrow-rs/pull/9865#issuecomment-4390303090
original - this causes the issues that #7710 aims to resolve. <img width="1724" height="958" alt="Image 5-6-26 at 11 24 AM" src="https://github.com/user-attachments/assets/b3772ed5-92d7-44a6-8407-ad5b2d64d694"; /> First approach - using values.to_data().slice(i,1) in the hot loop <img width="1582" height="1418" alt="Image 5-6-26 at 11 25 AM" src="https://github.com/user-attachments/assets/ca18ba35-1eae-4f6a-845b-521159b33c11"; /> Super slow Second approach - factor out the values to .slice(i,1) to a separate function where its computes the comparissions ahead of time in a boolean array. **this was slightly better than the first approach but still much slower than the initial** Final approach - moved around some code in ord.rs so that arrow-select & arrow-ord dont cause a depency chain. This uses ```make_comparator(values,values,default)``` to quickly compare positions in the array. <img width="1582" height="1224" alt="Image 5-6-26 at 12 54 PM" src="https://github.com/user-attachments/assets/3ab90ff2-0bad-4e4a-9a11-1771856d3906"; /> This approach is still slower than the initial approach but this is expected. comparing values takes time. cc @Jefffrey cc @asubiotto I think you also refered to a similar approach for interleave. assuming this PR gets merged it would make ```make_compactor``` visible to interleave as well making the dedupe in interleave a lot simpler. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
