[ https://issues.apache.org/jira/browse/SPARK-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393507#comment-15393507 ]
Sean Owen commented on SPARK-16709: ----------------------------------- I'm not sure it's different. Can you try a version that SPARK-14915 is merged in, like 1.6.2 or later? Otherwise I'm pretty sure the CDE path is the same problem here. Or: provide a pull request that illustrates what you think the fix is. > Task with commit failed will retry infinite when speculation set to true > ------------------------------------------------------------------------ > > Key: SPARK-16709 > URL: https://issues.apache.org/jira/browse/SPARK-16709 > Project: Spark > Issue Type: Bug > Affects Versions: 1.6.0 > Reporter: Hong Shen > Attachments: commit failed.png > > > In our cluster, we set spark.speculation=true, but when a task throw > exception at SparkHadoopMapRedUtil.performCommit(), this task can retry > infinite. > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala#L83 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org