This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5d672b7 [SPARK-26763][SQL] Using fileStatus cache when
filterPartitions
5d672b7 is described below
commit 5d672b7f3e07cfd7710df319fc6c7d2b9056a068
Author: Xianyang Liu <xianyang....@intel.com>
AuthorDate: Tue Jan 29 23:11:11 2019 +0800
[SPARK-26763][SQL] Using fileStatus cache when filterPartitions
## What changes were proposed in this pull request?
We should pass the existed `fileStatusCache` to `InMemoryFileIndex` even
though there aren't partition columns.
## How was this patch tested?
Existed test. Extra tests can be added if there is a requirement.
Closes #23683 from ConeyLiu/filestatuscache.
Authored-by: Xianyang Liu <xianyang....@intel.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
index a66a076..8736d07 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
@@ -84,8 +84,8 @@ class CatalogFileIndex(
new PrunedInMemoryFileIndex(
sparkSession, new Path(baseLocation.get), fileStatusCache,
partitionSpec, Option(timeNs))
} else {
- new InMemoryFileIndex(
- sparkSession, rootPaths, table.storage.properties, userSpecifiedSchema
= None)
+ new InMemoryFileIndex(sparkSession, rootPaths, table.storage.properties,
+ userSpecifiedSchema = None, fileStatusCache = fileStatusCache)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
