[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114794#comment-15114794 ]
Hyukjin Kwon edited comment on SPARK-12890 at 1/25/16 5:49 AM: --------------------------------------------------------------- In that case, it will not read all the data but only footers (metadata) for each file, {{_METADATA}} or {{_COMMON_METADATA}} as the requested columns would be empty because the required column is a partition column. was (Author: hyukjin.kwon): In that case, it will not read all the data but only footer (metadata), {{_METADATA}} or {{_COMMON_METADATA}} as the requested columns would be empty because the required column is a partition column. > Spark SQL query related to only partition fields should not scan the whole > data. > -------------------------------------------------------------------------------- > > Key: SPARK-12890 > URL: https://issues.apache.org/jira/browse/SPARK-12890 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Prakash Chockalingam > > I have a SQL query which has only partition fields. The query ends up > scanning all the data which is unnecessary. > Example: select max(date) from table, where the table is partitioned by date. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org