[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998014#comment-14998014 ]
Hyukjin Kwon commented on SPARK-11621: -------------------------------------- I would like to work this. > ORC filter pushdown not working properly after new unhandled filter interface. > ------------------------------------------------------------------------------ > > Key: SPARK-11621 > URL: https://issues.apache.org/jira/browse/SPARK-11621 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.0 > Reporter: Hyukjin Kwon > > After we get the new interface to get rid of filters predicate-push-downed > which are processed in datasource-level > (https://github.com/apache/spark/pull/9399), it dose not push down filters > for ORC. > This is because at {{DataSourceStrategy}}, it is classified to scanning > non-partitioned HadoopFsRelation, and all the filters are treated as > unhandled filters. > Also, since ORC does not support to filter fully record by record but instead > rough results came out, the filters for ORC should not go to unhandled > filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org