[ https://issues.apache.org/jira/browse/SPARK-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin closed SPARK-11448. ------------------------------- Resolution: Auto Closed > We should skip caching part-files in ParquetRelation when configured to merge > schema and respect summaries > ---------------------------------------------------------------------------------------------------------- > > Key: SPARK-11448 > URL: https://issues.apache.org/jira/browse/SPARK-11448 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Liang-Chi Hsieh > > We now cache part-files, metadata, common metadata in ParquetRelation as > currentLeafStatuses. However, when configured to merge schema and respect > summaries, dataStatuses (`FileStatus` objects of all part-files) are not > necessary anymore. We should skip them when caching in driver side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org