alamb commented on PR #24102: URL: https://github.com/apache/datafusion/pull/24102#issuecomment-5417025308
> > BTW I made a small test that creates a 1.6GB file and then does a query > > @alamb Thanks for putting this together! I tried the same test locally with only 4 values, which exercises the direct-comparison path. The filter itself was about 2x faster, and the full query improved from ~128ms to ~111ms. With 100 values, I also see only a few percent end-to-end improvement, consistent with your results. I think a few percent end to end is a pretty good result, to be clear. In fact I was debugging a query just today that has an `IN` list with 100 elements 😆 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
