Eric Liang created SPARK-16596: ---------------------------------- Summary: Refactor DataSourceScanExec to do partition discovery at execution instead of planning time Key: SPARK-16596 URL: https://issues.apache.org/jira/browse/SPARK-16596 Project: Spark Issue Type: Improvement Components: SQL Reporter: Eric Liang Priority: Minor
Partition discovery is rather expensive, so we should do it at execution time instead of during physical planning. Right now there is not much benefit since ListingFileCatalog will read scan for all partitions at planning time anyways, but this can be optimized in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org