[
https://issues.apache.org/jira/browse/TEZ-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated TEZ-1087:
-------------------------------
Attachment: TEZ-1087-1.patch
Looks like iterator.remove() gets called twice in some cases-
{code}
InputAttemptIdentifier input = inputIter.next();
// Avoid adding attempts which have already completed.
if (completedInputSet.contains(input.getInputIdentifier())) {
inputIter.remove();
}
// Avoid adding attempts which have been marked as OBSOLETE
if (obsoletedInputs.contains(input)) {
inputIter.remove();
obsoletedInputs.remove(input);
}
{code}
Attaching a patch that addresses it. I am fixing this specific issue without
understanding the overall code. Please let me know if this is not a proper fix.
> ShuffleManager fails with IllegalStateException
> -----------------------------------------------
>
> Key: TEZ-1087
> URL: https://issues.apache.org/jira/browse/TEZ-1087
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Cheolsoo Park
> Attachments: TEZ-1087-1.patch
>
>
> While running a Pig-on-Tez job, I ran into this error-
> {code}
> at java.util.ArrayList$Itr.remove(ArrayList.java:844)
> at
> org.apache.tez.runtime.library.shuffle.common.impl.ShuffleManager.constructFetcherForHost(ShuffleManager.java:314)
> at
> org.apache.tez.runtime.library.shuffle.common.impl.ShuffleManager.access$1200(ShuffleManager.java:80)
> at
> org.apache.tez.runtime.library.shuffle.common.impl.ShuffleManager$RunShuffleCallable.call(ShuffleManager.java:260)
> at
> org.apache.tez.runtime.library.shuffle.common.impl.ShuffleManager$RunShuffleCallable.call(ShuffleManager.java:213)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
