[
https://issues.apache.org/jira/browse/IMPALA-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-14949:
--------------------------------
Description:
A potential deadlock exists in catalogd during startup when the initial global
invalidate all happens under heavy load (for example, when catalogd restarts
after it was killed for running out of memory).
Sequence of events:
1. Initial invalidate all takes too long and releases versionLock_
[here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2620].
2. Another catalogd operation (such as getPartialCatalogObject) takes a read
lock on versionLock_ (for example,
[here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2808]).
3. The catalogd operationr then waits for the initial invalidate all to finish
(for example,
[here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L523])
4. The Initial invalidate all cannot continue as it needs the write lock of
versionLock_
[here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2625].
was:
A potential deadlock exists in catalogd during startup when the initial global
invalidate all happens under heavy load (for example, when catalogd restarts
after it was killed for running out of memory).
Sequence of events:
1. Initial invalidate all takes too long and releases versionLock_
[here](https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2620).
2. Another catalogd operation (such as getPartialCatalogObject) takes a read
lock on versionLock_ (for example,
[here](https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2808)).
3. The catalogd operationr then waits for the initial invalidate all to finish
(for example,
[here](https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L523))
4. The Initial invalidate all cannot continue as it needs the write lock of
versionLock_
[here](https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2625).
> Catalogd Deadlock During Initial Invalidate
> -------------------------------------------
>
> Key: IMPALA-14949
> URL: https://issues.apache.org/jira/browse/IMPALA-14949
> Project: IMPALA
> Issue Type: Bug
> Reporter: Jason Fehr
> Priority: Critical
>
> A potential deadlock exists in catalogd during startup when the initial
> global invalidate all happens under heavy load (for example, when catalogd
> restarts after it was killed for running out of memory).
> Sequence of events:
> 1. Initial invalidate all takes too long and releases versionLock_
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2620].
> 2. Another catalogd operation (such as getPartialCatalogObject) takes a read
> lock on versionLock_ (for example,
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2808]).
> 3. The catalogd operationr then waits for the initial invalidate all to
> finish (for example,
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L523])
> 4. The Initial invalidate all cannot continue as it needs the write lock of
> versionLock_
> [here|https://github.com/apache/impala/blob/89b3307e377351e6920929dec75ee08bfc9f5f4a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2625].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
