Steve Loughran created MAPREDUCE-7474: -----------------------------------------
Summary: [ABFS] Improve commit resilience and performance in Manifest Committer Key: MAPREDUCE-7474 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.3.6, 3.4.0 Reporter: Steve Loughran * Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled. * large burst of delete calls noted: are they needed relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only. Initial proposed changes * retry recovery on task commit rename, always (repeat save, delete, rename) * audit delete use and see if it can be pruned * maybe: rate limit some IO internally, but not delegate to abfs -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org