yinan zhan created HADOOP-17584:
-----------------------------------
Summary: s3a committer
Key: HADOOP-17584
URL: https://issues.apache.org/jira/browse/HADOOP-17584
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.2.0
Reporter: yinan zhan
s3a magic committer isRecoverySupported() is false, so will restart all task
after application master restart for am jvm crashed, leaving pendingset in
magic path not to clear. pendingset name format is jobAttemptPath +
taskAttemptID.getTaskID() + ".pendingset". These pendingset files are
overwritted by new task commit.
But if in new am attempt, a speculative task overcomes origin task, so
pendingset file in last attempt may be hold for job commit, the data for commit
is wrong
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
