MaxGekk edited a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573390974 @cloud-fan The problem was in the line https://github.com/apache/spark/pull/26973/commits/f0aa0a88bfa0c87007f8781ba7fac8f9cd3057ba#diff-faa3cfad03552057c3cb431c5ce87f03L52 where I returned an empty array (https://github.com/apache/spark/commit/f0aa0a88bfa0c87007f8781ba7fac8f9cd3057ba#diff-44a98c4a53980cb04e57f0489b257a37L141). The comment for pushFilters says that I should return filters that should be evaluated after scanning https://github.com/apache/spark/blob/053dd858d38e6107bc71e0aa3a4954291b74f8c8/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.java#L31 I guessed if I consumed all filters then I should return nothing. But looking at ORC and Parquet, I noticed that they return all filters: https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala#L65 https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcScanBuilder.scala#L65
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
