pan3793 commented on PR #55579: URL: https://github.com/apache/spark/pull/55579#issuecomment-6649385617
Thanks @LuciferYang. Re-verified at `33a96ff` with the same scratch tests as my [review](https://github.com/apache/spark/pull/55579#pullrequestreview-5733830527): 1. **Split size:** fixed. Same setup (10 partitions, about 4 MB each, `minPartitionNum=20`): | Query | Tasks | Largest split | |---|---|---| | V2 runtime-pruned, scalar subquery or DPP | 21 | 200,722 B | | V2, same partition selected at compile time | 21 | 200,722 B | | V1, scalar subquery or DPP | 21 | | 2. **Eval failure:** fixed. With ANSI on, both the DPP join and the scalar subquery return the correct 10 rows. The DPP scan keeps the `abc` directory and the join drops its rows. 3. **Empty array:** fixed. It returns the same partitions as `planInputPartitions()`. 4. **Screened-out filters:** fixed. A DPP filter degraded to `true` now reaches the scan (1 call, was 0). 5. **`CleanupDynamicPruningFilters`:** fixed. A non-file DSv2 relation keeps its DPP filter (1 before, 1 after). 6. **Description:** PR description is stale. 7. **Nits:** fixed. Your suites plus the DSv2 DPP suites pass locally: 365 tests, 0 failures. CI fails, merging master should fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
