[
https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-11621:
---------------------------------
Description:
After the new interface to get rid of filters predicate-push-downed which are
already processed in datasource-level
(https://github.com/apache/spark/pull/9399), it dose not push down filters for
ORC.
This is because at {{DataSourceStrategy}}, all the filters are treated as
unhandled filters.
Also, since ORC does not support to filter fully record by record but instead
rough results came out, the filters for ORC should not go to unhandled filters.
was:
After the new interface to get rid of filters predicate-push-downed which are
already processed in datasource-level
(https://github.com/apache/spark/pull/9399), it dose not push down filters for
ORC.
This is because at {{DataSourceStrategy}}, it is classified to scanning
non-partitioned HadoopFsRelation, and all the filters are treated as unhandled
filters.
Also, since ORC does not support to filter fully record by record but instead
rough results came out, the filters for ORC should not go to unhandled filters.
> ORC filter pushdown not working properly after new unhandled filter interface.
> ------------------------------------------------------------------------------
>
> Key: SPARK-11621
> URL: https://issues.apache.org/jira/browse/SPARK-11621
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Hyukjin Kwon
>
> After the new interface to get rid of filters predicate-push-downed which are
> already processed in datasource-level
> (https://github.com/apache/spark/pull/9399), it dose not push down filters
> for ORC.
> This is because at {{DataSourceStrategy}}, all the filters are treated as
> unhandled filters.
> Also, since ORC does not support to filter fully record by record but instead
> rough results came out, the filters for ORC should not go to unhandled
> filters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
