[
https://issues.apache.org/jira/browse/TEZ-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4334:
------------------------------
Summary: Fix deadlock in ShuffleScheduler between ShuffleScheduler.close()
and the ShufflePenaltyReferee thread (was: Fix deadlock in ShuffleScheduler)
> Fix deadlock in ShuffleScheduler between ShuffleScheduler.close() and the
> ShufflePenaltyReferee thread
> ------------------------------------------------------------------------------------------------------
>
> Key: TEZ-4334
> URL: https://issues.apache.org/jira/browse/TEZ-4334
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Sungwoo Park
> Assignee: Sungwoo Park
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Deadlock can be generated between a thread calling ShuffleScheduler.close()
> and the ShufflePenaltyReferee thread.
> Example (produced with an earlier version):
> {{"Fetcher_O { attempt_1611850856294_0026_1_03_000000_0_10344 Reducer_3} #13"
> #2669 daemon prio=5 os_prio=0 tid=0x00002b9de869d000 nid=0xf99 in
> Object.wait() [0x00002b9de4983000]
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.close(ShuffleScheduler.java:481)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.cleanupShuffleScheduler(Shuffle.java:352)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.cleanupShuffleSchedulerIgnoreErrors(Shuffle.java:343)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.reportException(Shuffle.java:407)
> - locked <0x00002b96bbb9d7a8> (a
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.isShuffleHealthy(ShuffleScheduler.java:1033)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.copyFailed(ShuffleScheduler.java:781)
> - locked <0x00002b96b98a7860> (a
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:414)
> "ShufflePenaltyReferee {Reducer_3}" #2645 daemon prio=5 os_prio=0
> tid=0x00002b9560fae800 nid=0xf7d waiting for monitor entry
> [0x00002b9de733b000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler$Referee.run(ShuffleScheduler.java:1322)
> - waiting to lock <0x00002b96b98a7860> (a
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler)}}
> We can fix the deadlock with:
> 1) do not hold ShuffleScheduler.this when calling
> exceptionReporter.reportException()
> 2) remove synchronized in copyFailed()
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
