[
https://issues.apache.org/jira/browse/SPARK-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Liang updated SPARK-17994:
-------------------------------
Description:
In SPARK-16980, we removed the full in-memory cache of table partitions in
favor of loading only needed partitions from the metastore. This greatly
improves the initial latency of queries that only read a small fraction of
table partitions.
However, since the metastore does not store file statistics, we need to
discover those from remote storage. With the loss of the in-memory file status
cache this has to happen on each query, increasing the latency of repeated
queries over the same partitions.
The proposal is to add back a per-table cache of partition contents, i.e.
Map[Path, Array[FileStatus]]. This cache would be retained per-table, and can
be invalidated through refreshTable() and refreshByPath(). Unlike the prior
cache, it can be incrementally updated as new partitions are read.
cc [~michael]
was:
In SPARK-16980, we removed the full in-memory cache of table partitions in
favor of loading only needed partitions from the metastore. This greatly
improves the initial latency of queries that only read a small fraction of
table partitions.
However, since the metastore does not store file statistics, we need to
discover those from remote storage. With the loss of the in-memory file status
cache this has to happen on each query, increasing the latency of repeated
queries over the same partitions.
The proposal is to add back a per-table cache of partition contents, i.e.
Map[Path, Array[FileStatus]]. This cache would be retained per-table, and would
be invalidated through refreshTable() and refreshByPath(). Unlike the prior
cache, it can be incrementally updated as new partitions are read.
cc [~michael]
> Add back a file status cache for catalog tables
> -----------------------------------------------
>
> Key: SPARK-17994
> URL: https://issues.apache.org/jira/browse/SPARK-17994
> Project: Spark
> Issue Type: Sub-task
> Reporter: Eric Liang
>
> In SPARK-16980, we removed the full in-memory cache of table partitions in
> favor of loading only needed partitions from the metastore. This greatly
> improves the initial latency of queries that only read a small fraction of
> table partitions.
> However, since the metastore does not store file statistics, we need to
> discover those from remote storage. With the loss of the in-memory file
> status cache this has to happen on each query, increasing the latency of
> repeated queries over the same partitions.
> The proposal is to add back a per-table cache of partition contents, i.e.
> Map[Path, Array[FileStatus]]. This cache would be retained per-table, and can
> be invalidated through refreshTable() and refreshByPath(). Unlike the prior
> cache, it can be incrementally updated as new partitions are read.
> cc [~michael]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
