[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1642513305 Update: I have a pr of this for branch-3.3 which does everything but changing the default/documenting this change #5859 this is so those releases stop warning about incompatibility and to keep the code more in sync between branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1581239126 @dannycjones: you happy with the changes now? I've got Ayush's upvote already -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1573949366 safe when you don't have incompatible clients trying to write to the same directory tree, more specifically using rename and delete, which effectively means "hive, mapreduce, spark jobs etc". the s3a committer doesn't use rename to commit work, but when spark starts a job it will delete the dest directory -so if delete() mistakes a marker at the root for an empty dir, it may not delete the old data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1572316027 Thanks. I think the tests were all good but will rerun to be 100% sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1570827515 noted. well, let's target 3.4 at the very least and tag as incompatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5689: HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep"
steveloughran commented on PR #5689: URL: https://github.com/apache/hadoop/pull/5689#issuecomment-1569802418 @ayushtkn really? default values are immutable. not sure about that as a lot of things change implicitly, or for good reason "the defaults weren't good". while the names+meanings+units are all stable, default values are public/evolving -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
