gengliangwang commented on issue #27157: [SPARK-30475][SQL] File source V2: Push data filters for file listing URL: https://github.com/apache/spark/pull/27157#issuecomment-575848049 @guykhazma Sorry to reply late. I was thinking about another approach, but I can't come up with a better one yet. My major concern is that the filters are supposed to be pushed down in the `FileScanBuilder`. It is wired to push down again for in the `FileScan`. Technically, the partition filters should be pushed down in `FileScanBuilder` as well. However, the current DSV2 API exposes the filters as `Filter` only instead of `Expression`. The coverage of `Filter` is limited. That's why I push the partition filters into FileScan in https://github.com/apache/spark/pull/27112. Keeping the behavior in V2 is also important. I will merge this one. We can improve the approach in the future.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
