[
https://issues.apache.org/jira/browse/FLINK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-13421:
----------------------------
Description:
When a TM lost and RM identified it first, it will notify JM about it through
JobMaster#notifyAllocationFailure.
We observed unexpected ConcurrentModificationException in this process, stack
as below:
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)
at
org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)
at
org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)
at
org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)
... 26 more
This can cause an allocated slot to be removed from SlotPool#allocatedSlots but
not all of its payload tasks get failed. Tasks may hang in scheduled forever,
not able to fail or timeout, as in the attached log.
was:
When a TM lost and RM identified it first, it will notify JM about it through
JobMaster#notifyAllocationFailure.
We observed unexpected ConcurrentModificationException in this process, stack
as below:
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)
at
org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)
at
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)
at
org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)
at
org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)
... 26 more
This can cause an allocated slot to be removed from SlotPool#allocatedSlots but
not all of its payload tasks get failed. Tasks may hang in scheduled forever,
not able to fail or timeout, as in the attached log.
It is not figured out yet that how a concurrent modification can happen. We do
not have a debug log for it and is not able to re-pro it with debug log enabled
yet.
However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its
children directly to avoid ConcurrentModificationException to occur in any case.
> Unexpected ConcurrentModificationException when RM notifies JM about
> allocation failure
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-13421
> URL: https://issues.apache.org/jira/browse/FLINK-13421
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.0
> Reporter: Zhu Zhu
> Priority: Blocker
> Attachments: jm_concurrentmodification.log
>
>
> When a TM lost and RM identified it first, it will notify JM about it through
> JobMaster#notifyAllocationFailure.
> We observed unexpected ConcurrentModificationException in this process, stack
> as below:
>
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)
> at
> org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)
> ... 26 more
>
> This can cause an allocated slot to be removed from SlotPool#allocatedSlots
> but not all of its payload tasks get failed. Tasks may hang in scheduled
> forever, not able to fail or timeout, as in the attached log.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
