viirya commented on PR #25679: URL: https://github.com/apache/datafusion/pull/25679#issuecomment-6284230623
Thanks for this — the grouping idea fixes the pathology nicely. I rebased the last commit onto current main to check it against the repro sunchao raised on #26214 (https://github.com/apache/datafusion/pull/26214#discussion_r4318931218). Conflicts are small: the multi-column `mark` closure from #19857, `MapOffset` → `ProbeOffset` (#26203), and `equal_rows_arr` now needs the comparator slot (the build scope values are normalized, so `equal_rows_arr_with_normalized_left` fits). With 8192 × 8192 rows and a constant scope key, `a NOT IN (SELECT x FROM inner_t i WHERE i.y = o.b)` goes from ~206 ms to ~1 ms, including the variant where one outer row sits in a scope that is never marked. The issue's Q08 at 100k goes from ~367 ms to ~10 ms, and the unique-scope 2M × 2M shapes cost +2–6% wall, in line with your numbers. Now that #25560 has landed, do you plan to take this out of draft? #26214 makes NOT NULL tuple elements of a multi-column `NOT IN` into scope keys, which routes more queries into this path, so I'd like to stack it on top of this. If you don't plan to continue with it, we'd be happy to open a PR that builds on your commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
