[ https://issues.apache.org/jira/browse/SPARK-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295951#comment-14295951 ]
Apache Spark commented on SPARK-5465: ------------------------------------- User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/4255 > Data source version of Parquet doesn't push down And filters properly > --------------------------------------------------------------------- > > Key: SPARK-5465 > URL: https://issues.apache.org/jira/browse/SPARK-5465 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 1.2.0, 1.2.1 > Reporter: Cheng Lian > Priority: Blocker > > The current implementation combines all predicates and then tries to convert > it to a single Parquet filter predicate. In this way, the Parquet filter > predicate can not be generated if any component of the original filters can > not be converted. (code lines > [here|https://github.com/apache/spark/blob/a731314c319a6f265060e05267844069027804fd/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala#L197-L201]). > For example, {{a > 10 AND a < 20}} can be successfully converted, while {{a > > 10 AND a < b}} can't because Parquet doesn't accept filters like {{a < b}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org