[
https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15115204#comment-15115204
]
Hyukjin Kwon edited comment on SPARK-12890 at 1/25/16 1:46 PM:
---------------------------------------------------------------
Actually I don't still understand what is an issue here. This might not be
related with merging schemas as it is disabled by default and any filter is not
being pushed down here. It does not automatically create a filter for a
function and pushes down it as far as I know.
I mean, the referenced column would be {{date}} and given filters would be
empty. So it tries to read all the files regardless of file format as long as
it supports to partitioned files.
was (Author: hyukjin.kwon):
Actually I don't still understand what is an issue here. This might not be
related with merging schemas as it is disabled by default and any filter is not
being pushed down here. It does not automatically create a filter and pushes
down it as far as I know.
I mean, the referenced column would be {{date}} and given filters would be
empty. So it tries to read all the files regardless of file format as long as
it supports to partitioned files.
> Spark SQL query related to only partition fields should not scan the whole
> data.
> --------------------------------------------------------------------------------
>
> Key: SPARK-12890
> URL: https://issues.apache.org/jira/browse/SPARK-12890
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Prakash Chockalingam
>
> I have a SQL query which has only partition fields. The query ends up
> scanning all the data which is unnecessary.
> Example: select max(date) from table, where the table is partitioned by date.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
