Mohammad Arshad created HDFS-17959:
--------------------------------------
Summary: Hive query execution hangs due to HDFS Client race
condition where RPC threads block on a zombie connection's queue
Key: HDFS-17959
URL: https://issues.apache.org/jira/browse/HDFS-17959
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 3.4.1
Reporter: Mohammad Arshad
Assignee: Mohammad Arshad
Hive query execution was blocked on HiveServer2 due to a race condition in the
HDFS client.
*Observation:*
I collected and analyzed a thread dump from HiveServer2 (attached to this
JIRA). The analysis showed that multiple threads were adding items to the
rpcRequestQueue, but no thread was polling from it. As a result, the threads
adding items could not proceed and remained blocked on the
SynchronousQueue.offer() method, which in turn blocked Hive query execution.
*Thread Dump Analysis Details:*
Multiple threads were stuck on queue 0x00007ef98c380800, but no IPC Parameter
Sending Thread (RpcRequestSender) exists to poll this queue.
The four sender threads present in the dump poll four different queues, all
recently created (13–33 seconds old).
The stuck queue 0x00007ef98c380800 belongs to an older Connection whose sender
thread is dead (a zombie queue).
The blocked threads are stuck in the following code path:
{code:java}
at
java.util.concurrent.SynchronousQueue.offer([email protected]/SynchronousQueue.java:857)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1187)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
