[
https://issues.apache.org/jira/browse/IMPALA-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080647#comment-18080647
]
ASF subversion and git services commented on IMPALA-14949:
----------------------------------------------------------
Commit 0802e295219fb45fd1021d46f945cc7dbc5f3935 in impala's branch
refs/heads/master from jasonmfehr
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0802e2952 ]
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Reviewed-on: http://gerrit.cloudera.org:8080/24275
Reviewed-by: Yida Wu <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Catalogd Deadlock During Initial Invalidate
> -------------------------------------------
>
> Key: IMPALA-14949
> URL: https://issues.apache.org/jira/browse/IMPALA-14949
> Project: IMPALA
> Issue Type: Bug
> Reporter: Jason Fehr
> Assignee: Jason Fehr
> Priority: Critical
>
> A potential deadlock exists in catalogd during startup when the initial
> global invalidate all happens under heavy load (for example, when catalogd
> restarts after it was killed for running out of memory).
> Sequence of events:
> 1. Initial invalidate all takes too long and releases versionLock_
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2620].
> 2. Another catalogd operation (such as getPartialCatalogObject) takes a read
> lock on versionLock_ (for example,
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2808]).
> 3. The catalogd operationr then waits for the initial invalidate all to
> finish (for example,
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L523])
> 4. The Initial invalidate all cannot continue as it needs the write lock of
> versionLock_
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2625].
> Caused by: https://gerrit.cloudera.org/c/22640/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
