This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 5d672b7 [SPARK-26763][SQL] Using fileStatus cache when filterPartitions 5d672b7 is described below commit 5d672b7f3e07cfd7710df319fc6c7d2b9056a068 Author: Xianyang Liu <xianyang....@intel.com> AuthorDate: Tue Jan 29 23:11:11 2019 +0800 [SPARK-26763][SQL] Using fileStatus cache when filterPartitions ## What changes were proposed in this pull request? We should pass the existed `fileStatusCache` to `InMemoryFileIndex` even though there aren't partition columns. ## How was this patch tested? Existed test. Extra tests can be added if there is a requirement. Closes #23683 from ConeyLiu/filestatuscache. Authored-by: Xianyang Liu <xianyang....@intel.com> Signed-off-by: Wenchen Fan <wenc...@databricks.com> --- .../org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala index a66a076..8736d07 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala @@ -84,8 +84,8 @@ class CatalogFileIndex( new PrunedInMemoryFileIndex( sparkSession, new Path(baseLocation.get), fileStatusCache, partitionSpec, Option(timeNs)) } else { - new InMemoryFileIndex( - sparkSession, rootPaths, table.storage.properties, userSpecifiedSchema = None) + new InMemoryFileIndex(sparkSession, rootPaths, table.storage.properties, + userSpecifiedSchema = None, fileStatusCache = fileStatusCache) } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org