[
https://issues.apache.org/jira/browse/SPARK-32118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144399#comment-18144399
]
Miguel Angel Fernandez Diaz commented on SPARK-32118:
-----------------------------------------------------
We hit this contention in a long-running application (a Thrift-server-like
service with many concurrent sessions). The metastore database sits behind a
connection pooler. One thread waited for a connection of the metastore pool
while holding the \{{HiveExternalCatalog}} lock, and every other session that
read the catalog stayed BLOCKED behind it for as long as the call lasted.
I have opened a PR with a smaller change than the one this ticket asks for:
https://github.com/apache/spark/pull/60446
It does not make catalog operations concurrent. It replaces the monitor used by
\{{HiveExternalCatalog.withClient}} with a \{{ReentrantLock}} and adds an
opt-in static conf, \{{spark.sql.hive.externalCatalog.lockTimeoutMs}} (default
0, disabled). When it is set, a caller that cannot get the lock in time fails
with an \{{AnalysisException}} instead of waiting with no bound. The default
behavior does not change.
The reason for this approach is that it does not depend on the Hive client
being thread-safe, which was the open concern in the previous PR
(https://github.com/apache/spark/pull/28938).
Two questions for the maintainers:
1. Is this ticket the right place for the change, or should I open a separate
one, since it mitigates the problem but does not remove the lock?
2. Would a follow-up that runs metastore calls concurrently through a pool of
Hive clients (one client per thread, so no shared client state) be acceptable?
I can work on it if there is interest.
> Use fine-grained read write lock for each database in HiveExternalCatalog
> -------------------------------------------------------------------------
>
> Key: SPARK-32118
> URL: https://issues.apache.org/jira/browse/SPARK-32118
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0, 3.1.0
> Reporter: Lantao Jin
> Priority: Major
> Labels: pull-request-available
>
> In HiveExternalCatalog, all metastore operations are synchronized by a same
> object lock. In a heavy traffic Spark thriftserver or Spark Driver, users's
> queries may be stuck by any a long operation. For example, if a user is
> accessing a table which contains mass partitions, the operation
> loadDynamicPartitions() holds the object lock for a long time. All queries
> are blocking to wait for the lock.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]