[
https://issues.apache.org/jira/browse/IMPALA-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071162#comment-18071162
]
Quanlong Huang commented on IMPALA-14872:
-----------------------------------------
To be specific, here is the callstack of a thread that holds the BackendState
lock while sending the CancelQueryFInstances RPC. Note that this comes from an
old version of Impala. After IMPALA-14271, the lock is released while sending
the RPC:
[https://gerrit.cloudera.org/c/23264/5/be/src/runtime/coordinator-backend-state.cc].
But Coordinator::BackendState::Cancel() still holds the lock while doing other
stuffs.
{noformat}
#0 sys_futex
#1 kudu::Notification::WaitForNotification
#2 kudu::rpc::Proxy::SyncRequest
#3 impala::ControlServiceProxy::CancelQueryFInstances
#4 impala::RpcMgr::DoRpcWithRetry
#5 impala::Coordinator::BackendState::Cancel
#6 impala::Coordinator::CancelBackends
#7 impala::Coordinator::HandleExecStateTransition
#8 impala::Coordinator::SetNonErrorTerminalState
#9 impala::Coordinator::Cancel
#10 impala::ClientRequestState::Cancel
#11 impala::ImpalaServer::CancelFromThreadPool
#12 boost::function2<void, int, impala::CancellationWork const&>::operator()
#13 impala::ThreadPool<impala::CancellationWork>::WorkerThread
#14 boost::function0<void>::operator()
#15 impala::Thread::SuperviseThread{noformat}
Here is the callstack of the UnresponsiveBackendThread that waits for such a
lock:
{noformat}
#0 __lll_lock_wait
#1 pthread_mutex_lock@@GLIBC_2.2.5
#2 __gthread_mutex_lock
#3 std::mutex::lock
#4 std::unique_lock<std::mutex>::lock
#5 std::unique_lock<std::mutex>::unique_lock
#6 impala::Coordinator::BackendState::IsDone
#7 impala::Coordinator::GetMaxBackendStateLagMs
#8 operator()
#9 std::function<void(std::shared_ptr<impala::QueryDriver> const&)>::operator()
#10 impala::GenericShardedQueryMap<impala::TUniqueId,
std::shared_ptr<impala::QueryDriver> >::DoFuncForAllEntries
#11 impala::ImpalaServer::UnresponsiveBackendThread
#12 boost::function0<void>::operator()
#13 impala::Thread::SuperviseThread{noformat}
[https://github.com/apache/impala/blob/b650fda439c578d4067bfbc5694c348cb8a8c4e6/be/src/service/impala-server.cc#L3067]
[https://github.com/apache/impala/blob/b650fda439c578d4067bfbc5694c348cb8a8c4e6/be/src/runtime/coordinator-backend-state.cc#L423]
> Don't hold query_driver_map_ lock while calling GetMaxBackendStateLagMs
> -----------------------------------------------------------------------
>
> Key: IMPALA-14872
> URL: https://issues.apache.org/jira/browse/IMPALA-14872
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.5.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> {{query_driver_map_}} is a highly contended data structure. Accessing it
> acquires a lock that prevents any other query progress on queries that share
> a shard, and by default there are only 4 shards.
> {{GetMaxBackendStateLagMs}} acquires a lock on each {{BackendState}} object,
> which may also be in use (notably while sending a
> {{CancelQueryFInstancesRequestPB}} RPC in Cancel()), for a coordinator within
> {{DoFuncForAllEntries}} in
> https://github.com/apache/impala/blob/4.5.0/be/src/service/impala-server.cc#L2983.
> Let's stop doing that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
