Jiechuan Chen created TEZ-4416:
----------------------------------

             Summary: Dead lock triggered by ShuffleScheduler
                 Key: TEZ-4416
                 URL: https://issues.apache.org/jira/browse/TEZ-4416
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.10.1
            Reporter: Jiechuan Chen
         Attachments: container.jstack, screenshot.PNG

How this bug is found:

I was executing a sql with Hive on tez on a cluster that has low disk capacity. 
An exception was thrown during the execution of one of the reducer, due to the 
failure of reading intermediate files. The task didn't stop normally, but keep 
hanging for a long while. Therefore, I printed out the jstack and did some 
investigation. Here's what I found.

(The jstack file and the screenshot of corresponding jstack segment are 
attached below.)

 

How this dead lock is triggered:
 # Fail to copy files on hdfs, which will trigger copyFailed() from 
FetcherOrderedGrouped.copyFromHost(), which is a synchronized method on 
ShuffleScheduler instance. 
 # Method called from 1 will eventually goes to ShuffleScheduler.close(), in 
which it tries to kill the Referee's thread by calling referee.interrupt() and 
referee.join().
 # Meanwhile, Referee is waiting for ShuffleScheduler's lock in its run() 
method, which is hold by the method called from 1. Hence a dead lock happens.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Reply via email to