Jiechuan Chen created TEZ-4416: ---------------------------------- Summary: Dead lock triggered by ShuffleScheduler Key: TEZ-4416 URL: https://issues.apache.org/jira/browse/TEZ-4416 Project: Apache Tez Issue Type: Bug Affects Versions: 0.10.1 Reporter: Jiechuan Chen Attachments: container.jstack, screenshot.PNG
How this bug is found: I was executing a sql with Hive on tez on a cluster that has low disk capacity. An exception was thrown during the execution of one of the reducer, due to the failure of reading intermediate files. The task didn't stop normally, but keep hanging for a long while. Therefore, I printed out the jstack and did some investigation. Here's what I found. (The jstack file and the screenshot of corresponding jstack segment are attached below.) How this dead lock is triggered: # Fail to copy files on hdfs, which will trigger copyFailed() from FetcherOrderedGrouped.copyFromHost(), which is a synchronized method on ShuffleScheduler instance. # Method called from 1 will eventually goes to ShuffleScheduler.close(), in which it tries to kill the Referee's thread by calling referee.interrupt() and referee.join(). # Meanwhile, Referee is waiting for ShuffleScheduler's lock in its run() method, which is hold by the method called from 1. Hence a dead lock happens. -- This message was sent by Atlassian Jira (v8.20.7#820007)