[
https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897863#comment-15897863
]
Steve Loughran commented on SPARK-19790:
----------------------------------------
The only time a task output committer should be making observable state changes
is during the actual commit operation. If it is doing things before that commit
operation, that's a bug in that it doesn't meet the goal "committer".
The Hadoop output committer has two stages here: the FileOutputFormat work and
then rename of files; together they are not a transaction, but on a real
filesystem: fast
The now deleted DirectOutputCommitter was doing things as it went along —but
that's why it got pulled.
That leaves: the Hadoop Output Committer committing work on object stores which
implement rename() as a copy, hence slow and with a large enough failure
window. HADOOP-13786 is going to make that window very small indeed, at least
for job completion.
One thing to look at here is the
{{org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter}} protocol, where
the committer can be asked whether or not it supports recovery, as well as
{{isCommitJobRepeatable}} to probe for a job commit being repeatable even if it
fails partway through. The committer gets to implement its policy there.
> OutputCommitCoordinator should not allow another task to commit after an
> ExecutorFailure
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-19790
> URL: https://issues.apache.org/jira/browse/SPARK-19790
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 2.1.0
> Reporter: Imran Rashid
>
> The OutputCommitCoordinator resets the allowed committer when the task fails.
>
> https://github.com/apache/spark/blob/8aa560b75e6b083b2a890c52301414285ba35c3d/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala#L143
> However, if a task fails because of an ExecutorFailure, we actually have no
> idea what the status is of the task. The task may actually still be running,
> and perhaps successfully commit its output. By allowing another task to
> commit its output, there is a chance that multiple tasks commit, which can
> result in corrupt output. This would be particularly problematic when
> commit() is an expensive operation, eg. moving files on S3.
> For other task failures, we can allow other tasks to commit. But with an
> ExecutorFailure, its not clear what the right thing to do is. The only safe
> thing to do may be to fail the job.
> This is related to SPARK-19631, and was discovered during discussion on that
> PR https://github.com/apache/spark/pull/16959#discussion_r103549134
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
