[
https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898003#comment-15898003
]
Steve Loughran commented on SPARK-19790:
----------------------------------------
Thinking some more & looking at code snippets
# FileOutputFormat with algorithm 2 can recover from a failed task commit,
somehow. That code is too complex to make sense of now that it mixes two
algorithms with corecursion and stuff to run in client and server.
# The s3guard committer will have the task upload it's files in parallel, but
will not complete the multipart commit; the information to do this will be
persisted to HDFS for execution by the job committer.
# I plan a little spark extension which will do the same for files with
absolute destinations, this time passing the data back to the job committer.
Which means a failed task can be recovered from. All pending writes for that
task will need to be found (scan FS) and abort. Still an issue about ordering
of PUT vs save of upload data; some GC of pending commits to a dest dir would
be the way to avoid running up bills.
This is one of those coordination problems where someone with TLA+ algorithm
specification skills would be good, along with the foundation specs for
filesystems and object stores. Someone needs to find a CS student looking for a
project.
> OutputCommitCoordinator should not allow another task to commit after an
> ExecutorFailure
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-19790
> URL: https://issues.apache.org/jira/browse/SPARK-19790
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 2.1.0
> Reporter: Imran Rashid
>
> The OutputCommitCoordinator resets the allowed committer when the task fails.
>
> https://github.com/apache/spark/blob/8aa560b75e6b083b2a890c52301414285ba35c3d/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala#L143
> However, if a task fails because of an ExecutorFailure, we actually have no
> idea what the status is of the task. The task may actually still be running,
> and perhaps successfully commit its output. By allowing another task to
> commit its output, there is a chance that multiple tasks commit, which can
> result in corrupt output. This would be particularly problematic when
> commit() is an expensive operation, eg. moving files on S3.
> For other task failures, we can allow other tasks to commit. But with an
> ExecutorFailure, its not clear what the right thing to do is. The only safe
> thing to do may be to fail the job.
> This is related to SPARK-19631, and was discovered during discussion on that
> PR https://github.com/apache/spark/pull/16959#discussion_r103549134
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
