steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode URL: https://github.com/apache/hadoop/pull/1810#issuecomment-575735320 Ok, my patch changes behaviour slightly (I'd actually thought it was a bug, but ITestS3GuardWriteBack thinks it is expected) With my patch, when you list a dir in nonauth mode, it adds records in DDB for any which don't exist, so helps to build up that full list of files. Currently, in nonauth, we only add changed files. What to do? We add files to S3Guard on creation, import etc, and in auth-mode we do build up that list. So why not nonauth Side issue: when we do that listing of a nonauth dir in auth mode, it lasts until the bit is cleared. (which deletes of files do, needlessly). But when reconciling the lists, we don't worry about files listed in S3Guard but not found in the FS. So we mark the dir as authoritative even though it could be that there are errors in the listing. Seems to me we should be looking at the TTL of entries in the original DDB listing and considering missing (expired) files as deleted. Oh, S3Guard is the pain of my life.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
