[ https://issues.apache.org/jira/browse/HADOOP-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631607#comment-16631607 ]
Steve Loughran commented on HADOOP-15782: ----------------------------------------- bq. Only one attempt is allowed to commit via canCommit and with deterministic output + atomic rename works anyway. yes, but if that attempt is considered to have failed, an alternative attempt may commit. At least, that's my reading of the code in Hadoop and spark. Now, provided that only one file is generated by a task attempt, and the output of any attempt can be accepted, then a long-delayed task commit *shouldnt* be harmful, at least if it happens while the job is in progress. If it happens after the job has completed, well, that's "unusual" I've never seen that happening, it is a failure mode to be considered if you want to be able to show that your algorithm is robust. The MR job committer explicitly checks before job commit that it's had a recent heartbeat with the YARN RM to avoid this and cluster partition problems at the job level; nothing worries about it for individual tasks. bq. even on HDFS v1 was noticeably non-atomic for large jobs and you need to check for _SUCCESS or have another service recording completion, v2 was a big improvement for Twitter. Yes: neither committer is fully atomic at some phases in its operation The S3A ones don't have atomic job commit either; just O(files) POST requests which can be done in parallel. They do at least deliver atomic task commit (at least I believe so...). bq. I am just about to familiarize myself with Spark's use of FOC it doesn't need to worry about job restart, so life is simpler. Still uses the MRv1 APIs though, which they should be weaned off (And in Hadoop MR: deprecated) > Clarify committers.md around v2 failure handling > ------------------------------------------------ > > Key: HADOOP-15782 > URL: https://issues.apache.org/jira/browse/HADOOP-15782 > Project: Hadoop Common > Issue Type: Bug > Components: documentation > Affects Versions: 3.1.0, 3.1.1 > Reporter: Gera Shegalov > Priority: Major > > The doc file > {{hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md}} > refers to the default file output committer (v2) as not supporting job and > task recovery throughout the doc: > {quote}or just by rerunning everything (The "v2" algorithm and Spark). > {quote} > This is incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org