[
https://issues.apache.org/jira/browse/HIVE-30215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30215:
----------------------------------
Description:
HiveMetaStoreClientWithLocalCache keys its entries on a
TableWatermark(validWriteIdList, tableId), and isValid() requires a non-null
write-id list. A non-transactional table therefore can never be cached. Six
call sites (listPartitionsByExpr, getPartitionsSpecByExpr, getAggrColStatsFor,
getPartitionsByNames, listPartitionNames, getTableColumnStatistics) fetch the
table id first and test validity afterwards. This client sits inside the
query-level cache (Hive.createMetaStoreClient), so each such getTable is a
metastore round-trip made only to find out that nothing can be cached.
On a TPC-DS 1 TB run on external Parquet tables, 9,356 of 16,111 get_table
calls came immediately before one of these sites. That is about 0.23 s of
compile per query, ~23 s per 99-query
> HS2 HMS cache: every partition and statistics lookup of a non-transactional
> table costs an extra get_table call
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30215
> URL: https://issues.apache.org/jira/browse/HIVE-30215
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HiveMetaStoreClientWithLocalCache keys its entries on a
> TableWatermark(validWriteIdList, tableId), and isValid() requires a non-null
> write-id list. A non-transactional table therefore can never be cached. Six
> call sites (listPartitionsByExpr, getPartitionsSpecByExpr,
> getAggrColStatsFor, getPartitionsByNames, listPartitionNames,
> getTableColumnStatistics) fetch the table id first and test validity
> afterwards. This client sits inside the query-level cache
> (Hive.createMetaStoreClient), so each such getTable is a metastore round-trip
> made only to find out that nothing can be cached.
>
>
>
>
>
> On a TPC-DS 1 TB run on external Parquet tables, 9,356 of 16,111 get_table
> calls came immediately before one of these sites. That is about 0.23 s of
> compile per query, ~23 s per 99-query
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
