Hello Marcel Kornacker, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5710 to look at the new patch set (#5). Change subject: IMPALA-4449: Revisit table locking pattern in the catalog ...................................................................... IMPALA-4449: Revisit table locking pattern in the catalog This commit fixes an issue where multiple long-running operations on the same catalog object (e.g. table) can block other catalog operations from making progress. Problem: IMPALA-1480 introduced table level locking that in conjunction with the global catalog lock ensures serialized access to catalog table objects. In some cases (e.g. multiple long running operations on same table), the locking pattern used resulted in the catalog lock being held for a long period of time, thus blocking other catalog operations from making any progress. That resulted in high response times and the system appearing to be hung. Solution: Change the locking pattern in the catalog for protecting table objects so that no operation will hold the catalog lock for a long time if it fails to acquire a table lock. The operation that attempts to acquire a table lock and fails to do so must release the catalog lock and retry. The use of fair locks prevent starvation from happening. The only operation that doesn't follow this retry logic is the getCatalogObjects() call that retrieves a snapshot of the catalog metadata for transmitting to the statestore. Testing: I manually tested this change by running concurrency tests using JMeter and verified that the throughput of catalog operations on a specific table is not affected by other concurrent long running operations (e.g. refresh) on a different table. Change-Id: Id08e21da31deb1f003b3cada4517651f3b3b2bb2 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 3 files changed, 183 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/5710/5 -- To view, visit http://gerrit.cloudera.org:8080/5710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id08e21da31deb1f003b3cada4517651f3b3b2bb2 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Marcel Kornacker <mar...@cloudera.com>