Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21803 )
Change subject: IMPALA-915: Support cancel queries in frontend ...................................................................... Patch Set 24: (7 comments) It seems cancelling the query in impala-shell http://gerrit.cloudera.org:8080/#/c/21803/24/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/21803/24/be/src/runtime/query-driver.cc@a94 PS24, Line 94: Could you explain why we don't need this? http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3825 PS24, Line 3825: throw new CatalogException("Error running getPartialCatalogObject(): ", e); Can we improve this when the InterruptedException is thrown at L3796? http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@575 PS24, Line 575: isPiggybacked = true; : Future<ValueType> existing = (Future<ValueType>)inCache; : ValueType ret = Uninterruptibles.getUninterruptibly(existing); The piggyback mechanism makes different queries be able to share getPartialCatalogObject RPCs. Currently the getPartialCatalogObject RPC thread is registered in catalogd side using the first query id. If user wants to cancel the other queries, that thread in catalogd won't be interrupted. Not sure if we can fix this by not using Uninterruptibles.getUninterruptibly(). http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/service/Canceller.java File fe/src/main/java/org/apache/impala/service/Canceller.java: http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/service/Canceller.java@42 PS24, Line 42: threads generating TExecRequest nit: now we have threads handling catalogd RPCs as well http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21803/24/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@430 PS24, Line 430: public TDdlExecResponse execDdlRequest(TDdlExecRequest ddlRequest) This is the entrance for DDL requests. They need the Canceller logics as well. However, only threads waiting for internal locks can be interrupted. For threads waiting for external RPCs (e.g. HMS RPCs), I think we have no way to cancel them. Since the current patch is "Support cancel queries in frontend", we can try to do this in a future patch to support cancel queries in catalogd. I think we can try triggering external RPCs in seperated threads so the current thread can be interrupted. http://gerrit.cloudera.org:8080/#/c/21803/24/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/21803/24/tests/custom_cluster/test_web_pages.py@567 PS24, Line 567: To test the piggyback mechanism, we can add a new test that has a 3rd query like "show create table tpcds.store_sales" running on coordinator1, and try cancelling the 3rd query. Currently it can't be cancelled immediately since it's waiting for the getPartialCatalogObject RPC of the 2nd query to finish. http://gerrit.cloudera.org:8080/#/c/21803/24/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/21803/24/tests/shell/test_shell_interactive.py@333 PS24, Line 333: def test_cancellation(self, vector): Can we add tests for impala-shell? It seems we can't cancel the query in CREATED state yet. I tried the case of slow REFRESH in coordinator0 and DESCRIBE in coordinator1 in local-catalog mode. I can't cancel the DESCRIBE query in impala-shell with Ctrl+C. -- To view, visit http://gerrit.cloudera.org:8080/21803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d25d4c7fb0b8dcc7dad9510db1e8dca220eeb86 Gerrit-Change-Number: 21803 Gerrit-PatchSet: 24 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Comment-Date: Sat, 14 Dec 2024 03:00:10 +0000 Gerrit-HasComments: Yes
