[
https://issues.apache.org/jira/browse/SPARK-31112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
deshanxiao updated SPARK-31112:
-------------------------------
Description:
Now, we use HiveClientImpl to access hive metastore. However, a long running
rpc in hive will block all of the query. Currently, we use the member of
externalCatalog in ShardState to access. It's singleton.
Maybe, we can use multiple extrenal catalog instance to speed up metastore
access in read-only situation.
Original:
Query 1:
DatabaseExist -> getTable -> getPartiton (6s)
Query 2:
DatabaseExist -> getTable -> getPartiton (5s)
Total cost: 11s
Now:
Query 1:
DatabaseExist -> getTable -> getPartiton (6s)
Query 2:
DatabaseExist -> getTable -> getPartiton (5s)
Total cost: 6s
was:Now, we use HiveClientImpl to access hive metastore. However, a long
running rpc in hive will block all of the query. Currently, we use the member
of externalCatalog in ShardState to access. Maybe, we can use multiple extrenal
catalog instance to speed up metastore access in read-only situation.
> Use multiple extrenal catalog to speed up metastore access
> ----------------------------------------------------------
>
> Key: SPARK-31112
> URL: https://issues.apache.org/jira/browse/SPARK-31112
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: deshanxiao
> Priority: Major
>
> Now, we use HiveClientImpl to access hive metastore. However, a long running
> rpc in hive will block all of the query. Currently, we use the member of
> externalCatalog in ShardState to access. It's singleton.
> Maybe, we can use multiple extrenal catalog instance to speed up metastore
> access in read-only situation.
> Original:
> Query 1:
> DatabaseExist -> getTable -> getPartiton (6s)
> Query 2:
> DatabaseExist -> getTable -> getPartiton (5s)
> Total cost: 11s
> Now:
> Query 1:
> DatabaseExist -> getTable -> getPartiton (6s)
> Query 2:
> DatabaseExist -> getTable -> getPartiton (5s)
> Total cost: 6s
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
