[
https://issues.apache.org/jira/browse/IMPALA-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775944#comment-17775944
]
Michael Smith commented on IMPALA-12493:
----------------------------------------
The log sequence:
{code}
I1010 05:06:28.007423 263772 status.cc:129] Cancelled from Impala's debug web
interface by client at XXX.XX.XXX.248:54879
I1010 05:06:28.008111 263772 impala-server.cc:1416] UnregisterQuery():
query_id=c64d690d72b8db5c:3e8be7e500000000
{code}
would only happen via a call to the HTTP endpoint /cancel_query. That
corresponds to the Cancel or Close button in the web UI, or they could be
hitting it programatically. There's nothing else inside Impala that would hit
that endpoint.
{code}
I1010 05:06:28.008137 263772 status.cc:129] Query not yet running
{code}
is an error state that means the request to cancel the query failed. It should
not change the state of the query. However before the cancel request gets to
that point, it's updated the QueryDriver to believe the query has been
finalized at
https://github.com/apache/impala/blob/4.3.0/be/src/runtime/query-driver.cc#L433.
Subsequent attempts to cancel the query exit early because that atomic has
already been set to true. That's what produces messages like
{code}
I1010 05:06:31.701305 260373 impala-server.cc:1566] Invalid or unknown query
handle: c64d690d72b8db5c:3e8be7e500000000.
{code}
After that you're stuck, there's no way to cancel the query and the coordinator
won't finalize the query because it believe's it's already finalized, i.e.
{code}
impala-server.cc:1829] CancelFromThreadPool(): query
c64d690d72b8db5c:3e8be7e500000000 already unregistered.
{code}
I can reproduce this issue by introducing an artificial sleep at
https://github.com/apache/impala/blob/4.3.0/fe/src/main/java/org/apache/impala/service/Frontend.java#L2330
to replicate slow table load.
IMPALA-915 would be one way to address this (by correctly implementing Cancel
during analysis). Another way would be to ensure we don't mark the query as
finalized if attempting to finalize it fails (possibly for certain classes of
failure).
> Impala Query cancelled while Analyzing or Compiling partially closes but
> query remains on Coordinator
> -----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12493
> URL: https://issues.apache.org/jira/browse/IMPALA-12493
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.0.0
> Reporter: Ezra Zerihun
> Priority: Major
>
> Customer checks Coordinator Web UI and on /queries page they either see:
> * query stuck in Running state indefinitely
> * query stuck in Waiting to be Closed state indefinitely
> Trying to close the query with Cancel button on /queries page or on Impala >
> Queries page both have no effect and the query does not close from Web UI
> /queries. So the queries remains on /queries page indefinitely, unless we
> restart the Impala Daemon.
>
> Looking at Impala Daemon logs we see below. Query begins Analyzing and takes
> a few minutes. While this is processing, the query is closed from the Web UI.
> This leaves the query in a partially Closed state, where we see "Invalid or
> unknown query handle" repeat indefinitely but the query still is trying to
> run on Coordinator.
>
> {quote}I1010 05:04:25.418613 260373 impala-server.cc:1332]
> c64d690d72b8db5c:3e8be7e500000000] Registered query
> query_id=c64d690d72b8db5c:3e8be7e500000000
> session_id=a0466ab54306c776:cf2f1660dd18149e
> I1010 05:04:25.418879 260373 Frontend.java:1623]
> c64d690d72b8db5c:3e8be7e500000000] Analyzing query: create table ...
> ...
> I1010 05:06:28.007423 263772 status.cc:129] Cancelled from Impala's debug web
> interface by client at XXX.XX.XXX.248:54879
> @ 0xe582e3
> @ 0x1450bdf
> @ 0x176e5ac
> @ 0x1770b62
> @ 0x1786bf0
> @ 0x1789317
> @ 0x7f62459a6ea4
> @ 0x7f62425e4b2c
> I1010 05:06:28.008111 263772 impala-server.cc:1416] UnregisterQuery():
> query_id=c64d690d72b8db5c:3e8be7e500000000
> I1010 05:06:28.008137 263772 status.cc:129] Query not yet running
> @ 0xe582e3
> @ 0x15741f8
> @ 0x15744ee
> @ 0x1aef419
> @ 0x1480aae
> @ 0x1450c13
> @ 0x176e5ac
> @ 0x1770b62
> @ 0x1786bf0
> @ 0x1789317
> @ 0x7f62459a6ea4
> @ 0x7f62425e4b2c
> ...
> I1010 05:06:30.887735 260373 AnalysisContext.java:468]
> c64d690d72b8db5c:3e8be7e500000000] Analysis took 64 ms
> I1010 05:06:30.890645 260373 BaseAuthorizationChecker.java:113]
> c64d690d72b8db5c:3e8be7e500000000] Authorization check took 2 ms
> I1010 05:06:30.890789 260373 Frontend.java:1666]
> c64d690d72b8db5c:3e8be7e500000000] Analysis and authorization finished.
> I1010 05:06:31.652444 260373 client-request-state.cc:754]
> c64d690d72b8db5c:3e8be7e500000000] DDL exec mode=asynchronous
> I1010 05:06:31.700479 260373 impala-server.cc:1372] Query
> c64d690d72b8db5c:3e8be7e500000000 has idle timeout of 15m
> I1010 05:06:31.700511 260373 impala-server.cc:1378] Query
> c64d690d72b8db5c:3e8be7e500000000 has execution time limit of 3h
> I1010 05:06:31.701305 260373 impala-server.cc:1566] Invalid or unknown query
> handle: c64d690d72b8db5c:3e8be7e500000000.
> I1010 05:06:31.703812 260373 impala-server.cc:2446] Connection
> a0466ab54306c776:cf2f1660dd18149e from client XXX.XX.XXX.220:59592 to server
> beeswax-frontend closed. The connection had 1 associated session(s).
> I1010 05:06:31.703851 260373 impala-server.cc:1616] Closing session:
> a0466ab54306c776:cf2f1660dd18149e
> I1010 05:06:31.703873 260373 impala-server.cc:1416] UnregisterQuery():
> query_id=c64d690d72b8db5c:3e8be7e500000000
> I1010 05:06:31.703892 260373 impala-server.cc:1566] Invalid or unknown query
> handle: c64d690d72b8db5c:3e8be7e500000000.
> E1010 05:06:31.703907 260373 impala-server.cc:1457] Query de-registration for
> query_id=\{0} failed: \{1}
> I1010 05:06:31.704043 260373 impala-server.cc:1663] Closed session:
> a0466ab54306c776:cf2f1660dd18149e
> ...
> I1010 05:06:44.944639 263786 status.cc:129] Cancelled from Impala's debug web
> interface by client at XXX.XX.XXX.248:54883
> @ 0xe582e3
> @ 0x1450bdf
> @ 0x176e5ac
> @ 0x1770b62
> @ 0x1786bf0
> @ 0x1789317
> @ 0x7f62459a6ea4
> @ 0x7f62425e4b2c
> I1010 05:06:44.945287 263786 impala-server.cc:1416] UnregisterQuery():
> query_id=c64d690d72b8db5c:3e8be7e500000000
> I1010 05:06:44.945313 263786 impala-server.cc:1566] Invalid or unknown query
> handle: c64d690d72b8db5c:3e8be7e500000000.
> ...
> I1010 05:14:15.958953 263810 coordinator.cc:533]
> c64d690d72b8db5c:3e8be7e500000000] started execution on 757 backends for
> query_id=c64d690d72b8db5c:3e8be7e500000000
> I1010 05:14:15.964103 263811 coordinator.cc:883] Coordinator waiting for
> backends to finish, 756 remaining. query_id=c64d690d72b8db5c:3e8be7e500000000
> ...
> I1010 05:14:20.445523 1088732 impala-server.cc:1579] Invalid or unknown query
> handle: c64d690d72b8db5c:3e8be7e500000000.
> I1010 05:14:20.445645 1088732 control-service.cc:179] ReportExecStatus():
> Received report for unknown query ID (probably closed or cancelled):
> c64d690d72b8db5c:3e8be7e500000000 remote host=XXX.XX.XXX.9:60548
> ...
> I1010 05:14:20.446139 1088704 impala-server.cc:1579] Invalid or unknown query
> handle: c64d690d72b8db5c:3e8be7e500000000.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
