[
https://issues.apache.org/jira/browse/IMPALA-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526742#comment-17526742
]
Wenzhe Zhou edited comment on IMPALA-11263 at 4/24/22 5:35 PM:
---------------------------------------------------------------
In function Coordinator::BackendState::Cancel(), if exec_rpc_sent_ equals true
and exec_done_ equals false (that means Coordinator::BackendState::ExecAsync()
is called, but callback function Coordinator::BackendState::ExecCompleteCb() is
not called), we will call RpcController::Cancel() to cancel Exec() RPC, then
call Coordinator::BackendState::WaitOnExecLocked() to wait RPC callback
function Coordinator::BackendState::ExecCompleteCb() to be called.
>From above log message, Coordinator::BackendState::Cancel() for the 4-th
>backend hang after calling WaitOnExecLocked(). That means the RPC callback
>function was not called even after the timeout.
Coordinator::BackendState::ExecCompleteCb() should be called if RPC is finished
successfully, finished with error, cancelled, or timeout.
Read KRPC code and found that it's possible that KRPC callback function is not
called in a corner case - The RPC is cancelled while it's in SENDING state and
get socket write error.
Impala coordinator calls RpcController::Cancel() to schedule a RPC cancellation
task for reactor thread pool. When reactor thread executes the cancellation
task with function ReactorThread::CancelOutboundCall(), the function calls
Connection::CancelOutboundCall(), then calls OutboundCall::Cancel().
Connection::CancelOutboundCall() reset car->call as null pointer which will
lead Connection::HandleOutboundCallTimeout() to skip calling
OutboundCall::SetTimedOut(). OutboundCall::Cancel() will not call
OutboundCall::SetCancelled() if the OutboundCall is in SENDING state.
OutboundCall::SetCancelled() will be called until OutboundCall:SetSent() is
called when the state is transferred from SENDING to SENT. So if a RPC is
cancelled, OutboundCall::SetTimedOut() will not be called for its OutboundCall
object when the timeout is handled in Connection::HandleOutboundCallTimeout(),
and OutboundCall::SetCancelled() will not be called until
OutboundCall:SetSent() is called when OutboundCall object is in SENDING state.
OutboundCall:SetSent() is called by function
CallTransferCallbacks::NotifyTransferFinished() if notification of transfer
finishing is received after sending a RPC call on the wire.
Connection::ProcessOutboundTransfers() call OutboundCall::SetSending() to set
OutboundCall's state as SENDING when starting transfer RPC. It then calls
OutboundTransfer::SendBuffer() to send data through socket.
OutboundTransfer::SendBuffer() calls socket->Writev() to send data. If
socket->Writev() return error, the SendBuffer() function will return error
without calling CallTransferCallbacks::NotifyTransferFinished() so
OutboundCall::SetSent() will not be called. This lead to
OutboundCall::SetCancelled() is not called for the OutboundCall object.
Connection::ProcessOutboundTransfers() then calls
ReactorThread::DestroyConnection() to destroy the connection.
ReactorThread::DestroyConnection() calls Connection::Shutdown() to clear all
outbound calls which have been sent and were awaiting a response. But for a RPC
being cancelled, its car->call is already reset as null pointer so
OutboundCall::SetFailed() will not be called for the OutboundCall object.
To summary, Connection::CancelOutboundCall() reset car->call as null pointer,
which will lead Connection::HandleOutboundCallTimeout() to skip calling
OutboundCall::SetTimedOut(), and Connection::Shutdown() to skip calling
OutboundCall::SetFailed(). socket->Writev() error causes
OutboundCall::SetSent() not been called, hence OutboundCall::SetCancelled() not
been called.
Since OutboundCall::SetFailed(), OutboundCall::SetCancelled() and
OutboundCall::SetTimedOut() are not called for the OutboundCall object, the
object cannot be transferred from SENDING state to a finished state, so that
RPC callback function will not be called.
This is the root cause that RPC callback function
Coordinator::BackendState::ExecCompleteCb() was not called when RPC was
cancelled. It caused Coordinator::BackendState::WaitOnExecLocked() to wait
indefinitely.
was (Author: wzhou):
In function Coordinator::BackendState::Cancel(), if exec_rpc_sent_ equals true
and exec_done_ equals false (that means Coordinator::BackendState::ExecAsync()
is called, but callback function Coordinator::BackendState::ExecCompleteCb() is
not called), we will call RpcController::Cancel() to cancel Exec() RPC, then
call Coordinator::BackendState::WaitOnExecLocked() to wait RPC callback
function Coordinator::BackendState::ExecCompleteCb() to be called.
>From above log message, Coordinator::BackendState::Cancel() for the 4-th
>backend hang after calling WaitOnExecLocked(). That means the RPC callback
>function was not called even after the timeout.
Coordinator::BackendState::ExecCompleteCb() should be called if RPC is finished
successfully, finished with error, cancelled, or timeout.
Read KRPC code and found that it's possible that KRPC callback function is not
called in a corner case - The RPC is cancelled while it's in SENDING state and
get socket write error.
Impala coordinator calls RpcController::Cancel() to schedule a RPC cancellation
task for reactor thread pool. When reactor thread executes the cancellation
task with function ReactorThread::CancelOutboundCall(), the function calls
Connection::CancelOutboundCall(), then calls OutboundCall::Cancel().
Connection::CancelOutboundCall() reset car->call as null pointer which will
lead Connection::HandleOutboundCallTimeout() to skip calling
OutboundCall::SetTimedOut(). OutboundCall::Cancel() will not call
OutboundCall::SetCancelled() if the OutboundCall is in SENDING state.
OutboundCall::SetCancelled() will be called until OutboundCall:SetSent() is
called when the state is transferred from SENDING to SENT. So if a RPC is
cancelled, OutboundCall::SetTimedOut() will not be called for its OutboundCall
object when the timeout is handled in Connection::HandleOutboundCallTimeout(),
and OutboundCall::SetCancelled() will not be called until
OutboundCall:SetSent() is called when OutboundCall object is in SENDING state.
OutboundCall:SetSent() is called by function
CallTransferCallbacks::NotifyTransferFinished() if notification of transfer
finishing is received after sending a RPC call on the wire.
Connection::ProcessOutboundTransfers() call OutboundCall::SetSending() to set
OutboundCall's state as SENDING when starting transfer RPC. It then calls
OutboundTransfer::SendBuffer() to send data through socket.
OutboundTransfer::SendBuffer() calls socket->Writev() to send data. If
socket->Writev() return error, the SendBuffer() function will return error
without calling CallTransferCallbacks::NotifyTransferFinished() so
OutboundCall::SetSent() will not be called. This lead to
OutboundCall::SetCancelled() is not called for the OutboundCall object.
Connection::ProcessOutboundTransfers() then calls
ReactorThread::DestroyConnection() to destroy the connection.
ReactorThread::DestroyConnection() calls Connection::Shutdown() to clear all
outbound calls which have been sent and were awaiting a response. But for a RPC
being cancelled, its car->call is already reset as null pointer so
OutboundCall::SetFailed() will not be called for the OutboundCall object.
To summary, Connection::CancelOutboundCall() reset car->call as null pointer,
which will lead Connection::HandleOutboundCallTimeout() to skip calling
OutboundCall::SetTimedOut(), and Connection::Shutdown() to skip calling
OutboundCall::SetFailed(). socket->Writev() error causes
OutboundCall::SetSent() not been called, hence OutboundCall::SetCancelled() not
been called.
Since OutboundCall::SetFailed(), OutboundCall::SetCancelled() and
OutboundCall::SetTimedOut() are not called for the OutboundCall object, the
object cannot be transferred from SENDING state to a finished state, hence RPC
callback function will not be called.
This is the root cause that RPC callback function
Coordinator::BackendState::ExecCompleteCb() was not called when RPC was
cancelled. It caused Coordinator::BackendState::WaitOnExecLocked() to wait
indefinitely.
> Coordinator hang when cancelling a query
> ----------------------------------------
>
> Key: IMPALA-11263
> URL: https://issues.apache.org/jira/browse/IMPALA-11263
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Wenzhe Zhou
> Assignee: Wenzhe Zhou
> Priority: Major
>
> In a rare case, callback function Coordinator::BackendState::ExecCompleteCb()
> was not called for the corresponding ExecQueryFInstances RPC somehow. This
> caused coordinator waited indefinitely when cancelling the query.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
