[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584229#comment-16584229 ] Hudson commented on HADOOP-14154: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14796 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14796/]) HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (fabbri: rev d7232857d8d1e10cdac171acdc931187e45fd6be) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DDBPathMetadata.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PathMetadataDynamoDBTranslation.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestPathMetadataDynamoDBTranslation.java > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, HADOOP-14154.005.patch, > HADOOP-14154.006.patch, HADOOP-14154.007.patch, all-logs.txt, > perf-eval-v1.diff, run-dir-perf-itest-v2.sh, run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583357#comment-16583357 ] genericqa commented on HADOOP-14154: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 3s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 24s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 63m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12935963/HADOOP-14154.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 16e80cf50e0d 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1290e3c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15050/testReport/ | | Max. process+thread count | 313 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15050/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > >
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583310#comment-16583310 ] Aaron Fabbri commented on HADOOP-14154: --- v7 patch: minor checkstyle cleanups. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, HADOOP-14154.005.patch, > HADOOP-14154.006.patch, HADOOP-14154.007.patch, all-logs.txt, > perf-eval-v1.diff, run-dir-perf-itest-v2.sh, run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583208#comment-16583208 ] genericqa commented on HADOOP-14154: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 16s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 7 new + 5 unchanged - 0 fixed = 12 total (was 5) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 32s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 35s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 63m 7s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12935938/HADOOP-14154.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux de3d854cdcfa 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d428061 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15047/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15047/testReport/ | | Max. process+thread count | 334 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15047/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583161#comment-16583161 ] Aaron Fabbri commented on HADOOP-14154: --- Nice work [~gabor.bota]. v5 patch looks good. I'm making one cosmetic change to add some parenthesis and attaching v6 patch for the precommit tests. {noformat} < + changed = changed || !dirMeta.isAuthoritative() && isAuthoritative; --- > + changed = changed || (!dirMeta.isAuthoritative() && > isAuthoritative);{noformat} Ran through the integration tests on us west 2 successfully. Will commit this evening. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, HADOOP-14154.005.patch, > HADOOP-14154.006.patch, all-logs.txt, perf-eval-v1.diff, > run-dir-perf-itest-v2.sh, run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581479#comment-16581479 ] Gabor Bota commented on HADOOP-14154: - Thank you for the review and the idea for dealing with the issue in {{dirListingUnion()}}! I've uploaded my v5 patch which includes the fix for it. You can find the measurements in all-logs.txt. Performance tests : * with perf-eval-v1.diff and run-dir-perf-itest-v2.sh * on aws instance in us-west-2 Usual integration tests ran on eu-west-1 without unknown issues. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, HADOOP-14154.005.patch, > all-logs.txt, perf-eval-v1.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580248#comment-16580248 ] Aaron Fabbri commented on HADOOP-14154: --- Ah, interesting. The existing code appears to work ok with LocalMetadataStore. Maybe it doesn't get {{changed == true}} in DynamoDBMetadataStore because that implementation always creates ancestor directories all the way up the tree? Just a theory. It seems like the boolean {{changed}} in {{dirListingUnion()}} should also take into account the auth bit. It is passed a param {{dirMeta}} which is the last dir listing fetched from the MetadataStore, I believe. If {{dirMeta.isAuthoritative == false}}, but {{isAuthoritative}} parameter is true, we should probably set {{changed = true}} to force a write of the auth bit for that directory. The next time that directory is listed it should skip the extra write since both {{dirMeta.isAuthoritative}} and {{isAuthoritative}} will be true. What do you think? > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579915#comment-16579915 ] Gabor Bota commented on HADOOP-14154: - I've debugged this, and the issue is that we never set the {{dirMeta.setAuthoritative(true);}} anywhere else, and the implementation populates every directory with every entry before {{S3Guard#dirListingUnion}} runs. This causes {{changed}} flag inside {{dirListingUnion}} to be false all the time during the test - the listing will be the same and the {{dirMeta}} will stay isAuthoritative=false. What do you think about this [~fabbri]? Do we need to change that? > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579617#comment-16579617 ] Gabor Bota commented on HADOOP-14154: - Now, the thing for me to debug (or even create a follow-up jira) is that why {{changed}} is false in {{S3Guard#dirListingUnion}} when a new listing is needed to add to the metadatastore as authoritative? I will try to find an answer for it, but do you think that this could be addressed in a new issue [~fabbri]? > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579585#comment-16579585 ] Gabor Bota commented on HADOOP-14154: - I've debugged the test you used in the {{run-dir-perf-itest}}, and found out that {code:java} dirMeta.setAuthoritative(true); // This is the full directory contents ms.put(dirMeta); {code} that actually calls {{put(DirListingMetadata meta)}} which will store the authoritative bit is *never called* during the test. It's fairly easy to find out just by running a covarage test locally. This means you will get the same results regardless of using auth mode or not - there will be no dirListings stored authoritative. After my mod in S3Guard the {{put(DirListingMetadata meta)}} is called for each directory, but only 49 times, so the number of directory listings we do. That's why the first {{treewalk}} is costly, and the others are not. There are no more calls to {{S3Guard#dirListingUnion}}, just in the first iteration of {{treeWalk}}. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579034#comment-16579034 ] Aaron Fabbri commented on HADOOP-14154: --- Interesting results. The fact that you had to make that code change suggests something is not working properly. The change you added to the perf-eval-v2.diff should make things slower, not faster, if it is implemented correctly. The idea of that logic is that is only writes the dir listing back to the MetadataStore if it is different than what we already got from MetadataStore#listChildren(). Your change should mean that it is *always* written back which we would expect to be slower (time) and more expensive ($). Probably need to do more debugging to figure out what is happening. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578007#comment-16578007 ] Gabor Bota commented on HADOOP-14154: - I've uploaded the following: * perf-eval-v2.diff: contains an additional change on top of v1 that will cause an update on the metadata dir listing. This will not happen in perf-eval-v1.diff so the measurements will not show the decreased time for directory listing queries. * run-dir-perf-itest-v2.sh: corrected the v1 script, so we run a test without authoritative directory listings in LocalMetadataStore * all-logs-v2.txt: measurement results with the v2 diff and itest script. In the results you can see a decreased time in directory listings when {{Using metadata store DynamoDBMetadataStore{region=us-west-2, tableName=gaborbota-oregon}, authoritative=true}} is used. The first query takes longer, but then the cached entries can be reached from dynamo. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576498#comment-16576498 ] genericqa commented on HADOOP-14154: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 27s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12935142/HADOOP-14154.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9539b2cf9d55 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0a71bf1 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/15013/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15013/testReport/ | | Max. process+thread count | 333 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15013/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was autom
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576407#comment-16576407 ] Gabor Bota commented on HADOOP-14154: - * I've uploaded HADOOP-14154.004.patch with the tests for backward compatibility check for the translation ({{testIsAuthoritativeCompatibilityItemToPathMetadata}} and {{testIsAuthoritativeCompatibilityPathMetadataToItem}}). * Added the modification to the md docs, that {{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore}} is capable for authoritative bit support. * Tested against eu-west-1, and got known issues (testDestroyNoBucket). I'll upload performance test results from an instance running on aws soon. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, perf-eval-v1.diff, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574200#comment-16574200 ] Aaron Fabbri commented on HADOOP-14154: --- Attached a tiny patch and script (perf-eval-v1.diff and run-dir-perf-test.sh) I used to take some performance measurements today. I ran it on my laptop from home. Results may look better actually running in AWS since the WAN latency goes away (thus dynamo latency speedup is larger), if you want to try that. This also made me realize some metrics around (1) hit rate for dir listings (when we skip s3 list) and (2) hit rate for listStatus() writeback (it skips writing listing back to dynamo if nothing changed). We could do this in a future JIRA. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, perf-eval-v1.diff, run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572676#comment-16572676 ] Aaron Fabbri commented on HADOOP-14154: --- v3 patch looks good. I ran through all the integration tests with -Ddynamo -Ds3guard and they looked good. Running with -Dauth now. Would be good to have some sort of benchmark results that show a performance benefit. I tried using {{ITestS3ADirectoryPerformance}} but didn't see a big difference. This could be because any speedups in listStatus() are being lost when we write back the contents of the directory at the end of listStatus() in {{org.apache.hadoop.fs.s3a.s3guard.S3Guard#dirListingUnion}}: {noformat} if (changed && isAuthoritative) { dirMeta.setAuthoritative(true); // This is the full directory contents ms.put(dirMeta); } return dirMetaToStatuses(dirMeta); } {noformat} We could modify that test to do each treewalk multiple times, which would allow the caching (skipping S3 on listStatus()) to have a benefit, since each listing would happen more than once. Anyways, I think this is just about ready to be committed. Left to do are: - Documentation (mark as unsupported/experimental until we have better TTL expiry and more testing with it) - Nice to have: benchmarks showing a benefit (when I prototyped this before TCP-DS partitioned benchmark showed a good speedup, but it was a bit of a pain to set up). > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571116#comment-16571116 ] Aaron Fabbri commented on HADOOP-14154: --- Thanks for the patch. Running through some tests.. should have some feedback in another day or so. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568673#comment-16568673 ] genericqa commented on HADOOP-14154: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 21s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 33s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 44s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12934302/HADOOP-14154.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 398e54e79ed1 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2b18bb4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14990/testReport/ | | Max. process+thread count | 336 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14990/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > >
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568606#comment-16568606 ] Gabor Bota commented on HADOOP-14154: - Thanks for the review [~fabbri]! Uploaded patch v3 with the fixes. I will write about my testing ideas and add .md documentation with the v4 patch. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568356#comment-16568356 ] genericqa commented on HADOOP-14154: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 47s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 29s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 61m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-aws | | | org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata defines equals but not hashCode At DDBPathMetadata.java:hashCode At DDBPathMetadata.java:[line 70] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12934273/HADOOP-14154.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b49018f3844d 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 3426f40 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/14989/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14989/testReport/ | | Max. process+thread count | 336 (vs. ulimit of 1) | | modules | C: hadoop-
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567775#comment-16567775 ] Aaron Fabbri commented on HADOOP-14154: --- Thanks for the v2 patch. The DDBPathMetadata stuff turned out pretty good. Was thinking we could probably reduce garbage a bit with some additional constructors for that class, but actually I think JVM escape analysis will handle most of those extra allocations on the stack. The lists of metadata are harder to deal with but I think your code looks fine here. Some inline comments... {noformat} @@ -840,21 +861,36 @@ public void prune(long modTime, String keyPrefix) throws IOException { new ArrayList<>(S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT); int delay = conf.getInt(S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY, S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_DEFAULT); + Set parentPathSet = new HashSet<>(); for (Item item : expiredFiles(modTime, keyPrefix)) { -PathMetadata md = PathMetadataDynamoDBTranslation +DDBPathMetadata md = PathMetadataDynamoDBTranslation .itemToPathMetadata(item, username); Path path = md.getFileStatus().getPath(); deletionBatch.add(path); + +// add parent path of what we remove +Path parentPath = path.getParent(); +parentPathSet.add(parentPath); {noformat} What if parentPath is root dir? I think we want a null check here. {noformat} + private void removeAuthoritativeDirFlag(Set pathSet) { +Set metas = pathSet.stream().map(path -> { + try { +DDBPathMetadata ddbPathMetadata = get(path); +if(ddbPathMetadata == null) { + return null; +} +LOG.debug("Setting false isAuthoritativeDir on {}", ddbPathMetadata); +ddbPathMetadata.setAuthoritativeDir(false); +return ddbPathMetadata; + } catch (IOException e) { +String msg = String.format("IOException while getting PathMetadata " ++ "on path: %s.", path); +LOG.error(msg, e); +return null; + } +}).filter(Objects::nonNull).collect(Collectors.toSet()); {noformat} I like that the stream keeps running if one of the paths fail, but should we also save a reference to the IOException and then throw it at the end of the function? That way we keep working even if we get a failure, but the failure still gets propagated to the caller. {noformat} --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java @@ -727,6 +727,13 @@ public void testPruneUnsetsAuthoritative() throws Exception { new FileStatus(0, false, 0, 0, time + 1, strToPath(freshFile)), Tristate.FALSE, false)); +// set parent dir as authoritative +if (!allowMissing()) { + DirListingMetadata parentDirMd = ms.listChildren(strToPath(parentDir)); + parentDirMd.setAuthoritative(true); + ms.put(parentDirMd); +} {noformat} Looks like you found a bug in the existing test case? Nice work. I was wondering if we want an integration test to confirm forward and backward compatibility with the schema change (old S3a works with new schema rows and vice versa). Not sure how we'd implement that though. Would probably need a separate copy of a couple of the PathMetadataDynamoDBTranslation functions with the old logic in them (or better yet, a boolean flag to select old behavior w/o the read/write of the auth flag), and then use those in the DDB integration test to confirm it all works. I'm not sure what this buys us in terms of regression testing though--so I could see the argument for manual testing. What do you think? > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567527#comment-16567527 ] genericqa commented on HADOOP-14154: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 22s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 22s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 48s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-aws | | | org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata doesn't override PathMetadata.equals(Object) At DDBPathMetadata.java:At DDBPathMetadata.java:[line 1] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12934174/HADOOP-14154.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 16309d718418 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 889df6f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/14981/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14981/testReport/ | | Max. process+thread count | 336 (vs. ulimit of 1) | | modules | C: hado
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567473#comment-16567473 ] Gabor Bota commented on HADOOP-14154: - The patch still does not include the docs. I'll upload another patch with the docs once the implementation got +1. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567469#comment-16567469 ] Gabor Bota commented on HADOOP-14154: - Thanks for the review [~fabbri], and for the idea of DDBPathMetadata. Uploaded my v2 patch (HADOOP-14154.002.patch) where I've fixed all the issues you mentioned. For the usual testing part: Tested against eu-west-1, and got some issues, but seems unrelated/known: {noformat} [ERROR] Failures: [ERROR] ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListLocatedStatusFiltering:499->AbstractContractGetFileStatusTest.verifyListStatus:534->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 length of listStatus(s3a://cloudera-dev-gabor-ireland/fork-0003/test, org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest$AllPathsFilter@5beca598 ) expected:<2> but was:<3> [ERROR] Errors: [ERROR] ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile:840 ? UnsupportedOperation [ERROR] ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110 ? IllegalArgument [ERROR] ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110 ? IllegalArgument {noformat} > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564734#comment-16564734 ] Aaron Fabbri commented on HADOOP-14154: --- Ok.. I actually looked at the right patch today ;). This looks pretty good overall. I'm excited to see this coming together. Inline comments: {noformat} @@ -518,12 +520,19 @@ public DirListingMetadata listChildren(final Path path) throws IOException { PathMetadata meta = itemToPathMetadata(item, username); metas.add(meta); } + + PathMetadata dirPathMeta = get(path); + boolean isAuthoritative = false; + if(dirPathMeta != null) { +isAuthoritative = dirPathMeta.isAuthoritativeDir(); + } {noformat} We should try to avoid the change to the public API (PathMetadata) I would create a new class (POJO) DDBPathMetadata which includes all the fields in PathMetadata plus the isAuthoritative flag. This way we don't have to change the public MetadataStore / PathMetadata interface. DDBPathMetadata can have a constructor which takes a PathMetadata + isAuth flag for when you are writing to dynamo. It can also have a method that creates (transforms to) a PathMetadata for when you are reading. More ideas below... {noformat} if (deletionBatch.size() == S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT) { Thread.sleep(delay); processBatchWriteRequest(pathToKey(deletionBatch), null); + + // set authoritative false for each pruned dir listing + removeAuthoritativeDirFlag(parentPathSet); + parentPathSet.clear(); + deletionBatch.clear(); } } if (deletionBatch.size() > 0) { Thread.sleep(delay); processBatchWriteRequest(pathToKey(deletionBatch), null); + +// set authoritative false for each pruned dir listing +removeAuthoritativeDirFlag(parentPathSet); +parentPathSet.clear(); {noformat} looks good. You should get test coverage from existing MetadataStoreTestBase#testPruneUnsetsAuthoritative case. {noformat} + private void removeAuthoritativeDirFlag(Set pathSet) { +pathSet.forEach(pp -> { + removeAuthoritativeDirFlag(pp); +}); + } + + private void removeAuthoritativeDirFlag(Path path){ +try { + DirListingMetadata dlm = listChildren(path); {noformat} looks correct but could be optimized. We are getting the whole listing just to clear the flag. I would probably have internal (private) getters/setters for loading / storing a single DDBPathMetadata from dynamo. Then you can just deal with the single row, instead of getting/putting the whole listing to set a flag. {noformat} + if(dlm == null){ {noformat} nit: spacing (space after if and close paren.) {noformat} @@ -1157,7 +1206,7 @@ private static void checkPathMetadata(PathMetadata meta) { map.put(WRITE_CAPACITY, throughput.getWriteCapacityUnits().toString()); map.put(TABLE, desc.toString()); map.put(MetadataStoreCapabilities.PERSISTS_AUTHORITATIVE_BIT, - Boolean.toString(false)); + Boolean.toString(true)); {noformat} Woo hoo! {noformat} - public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir, boolean - isDeleted) { + public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir, + boolean isDeleted) { +this(fileStatus, isEmptyDir, isDeleted, false); + } + + public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir, + boolean isDeleted, boolean isAuthoritativeDir) { {noformat} IMO, this change should go away, instead happening on the dynamo-specific PathMetadata POJO (e.g. DDBPathMetadata) {noformat} -return new PathMetadata(fileStatus, Tristate.UNKNOWN, isDeleted); +return new PathMetadata(fileStatus, Tristate.UNKNOWN, isDeleted, +isAuthoritativeDir); } {noformat} here I'm thinking you could have a new private function, e.g. private DDBPathMetadata itemToDDBPathMetadata() then if you need an actual PathMetadata you can just DDBPathMetadata dpm = itemToDDBPathMetadata(..); ... return dpm.toPathMetadata() {noformat} +// set parent dir as authoritative +if(!allowMissing()) { {noformat} nit: spacing {noformat} + @Test + public void testPrunePreservesAuthoritative() throws Exception { + // assert that parent dir is still authoritative (no removed elements + // during prune) + assertFalse(rootDirMd.isAuthoritative()); + assertFalse(grandParentDirMd.isAuthoritative()); + assertTrue(parentDirMd.isAuthoritative()); +} + } + {noformat} Nice, thank you for adding more test coverage to the base test class. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jir
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563165#comment-16563165 ] Aaron Fabbri commented on HADOOP-14154: --- Yes. you are right. I just thought of this before going to bed–remembered the old patch I had read before was very similar, haha. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563160#comment-16563160 ] Gabor Bota commented on HADOOP-14154: - Hi [~fabbri], I don't have allowAuthoritative in my patch, I think you have reviewed an older version. Please review https://issues.apache.org/jira/secure/attachment/12933674/HADOOP-14154.001.patch > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562984#comment-16562984 ] Aaron Fabbri commented on HADOOP-14154: --- Thanks for the patch. {quote} * Added {{PathMetadata#isAuthoritativeDir}} because {{DynamoDBMetadataStore}} stores {{PathMetadata}} in ddb, so that was a logical choice. {quote} I did not see this in your diff, it may be missing some files? This may be a quick way to get the code working because PathMetadata are translated DDB rows, but I think it confuses the MetadataStore API. I think the API should probably remain as is (you set Authoritative bit on directories, not files). This probably means you need to change the PathMetadataDynamoDBTranslation logic a bit, though. Also the testAuthoritative case you added; should that go in the base class instead of the TestDynamoDBMetadataStore subclass? I think it should. Any MetadataStore should behave the same if it supports persisting the auth bit (they all will now, except for Null of course). There is already some code there, though. See testListChildrenAuthoritative() {noformat} --- hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java +++ hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java @@ -188,6 +188,7 @@ private String tableName; private Configuration conf; private String username; + private boolean allowAuthoritative; {noformat} The MetadataStore just persists the auth bit on directories. The FS (S3A) is what decides which listing is a full/complete/authoritative listing. So, I don't think you need this here. {noformat} @@ -365,7 +374,7 @@ public DirListingMetadata listChildren(Path path) throws IOException { return (metas.isEmpty() && get(path) == null) ? null - : new DirListingMetadata(path, metas, false); + : new DirListingMetadata(path, metas, allowAuthoritative); } catch (AmazonClientException e) { throw translateException("listChildren", path, e); {noformat} A DirListingMetadata is authoritative=true iff (if and only if) the FS set it when it previously called put(). Returning true based on the config is not correct here. That config option "allow authoritative" should only be visible in the FS (S3A) code; it declares that S3A *may* treat MS results as authoritative if the bit is set. The MetadataStore's job is simply to return (persist) what it was told earlier by the FS when it returns a DirListingMetadata. So if MS.put(/some/dir, auth=true) then MS.get(/some/dir) can return with auth=true (assuming a prune() or something else has not invalidated the listing for that dir) but if MS.put(/some/dir, auth=false) then MS.get(/some/dir) must return with auth=false. This behavior is completely independent from the value of fs.s3a.metadatastore.authoritative (that is only used by the FS to skip round trips to S3 for listings) Thanks for digging into this feature--I know it is confusing but I hope this helps some. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562654#comment-16562654 ] genericqa commented on HADOOP-14154: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 29s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 34s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 63m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14154 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12933674/HADOOP-14154.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ebcf9bec5076 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ee53602 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14965/testReport/ | | Max. process+thread count | 301 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14965/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > >
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562582#comment-16562582 ] Gabor Bota commented on HADOOP-14154: - Tested against eu-west-1, and got some issues, but seems unrelated/known: {noformat} [ERROR] Failures: [ERROR] ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListLocatedStatusEmptyDirectory:129->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 listLocatedStatus(test dir): directory count in 4 directories and 0 files expected:<1> but was:<4> [ERROR] ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListStatusFiltering:463->AbstractContractGetFileStatusTest.verifyListStatus:534->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 length of listStatus(s3a://cloudera-dev-gabor-ireland/fork-0007/test, org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest$AllPathsFilter@4f6f98a3 ) expected:<2> but was:<4> [ERROR] Errors: [ERROR] ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile:840 ? UnsupportedOperation [ERROR] ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110 ? IllegalArgument [ERROR] ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110 ? IllegalArgument {noformat} > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562579#comment-16562579 ] Gabor Bota commented on HADOOP-14154: - Uploaded patch v001 for this issue. * I had to fix {{MetadataStoreTestBase#testPruneUnsetsAuthoritative}} because it did not test anything. (had to add a dir which is authoritative, but a child element gets removed so the parent dir loses the flag) * I've added {{testPrunePreservesAuthoritative}} to test that the authoritative flag is preserved when no child elements pruned from the dir. * Added {{PathMetadata#isAuthoritativeDir}} because {{DynamoDBMetadataStore}} stores {{PathMetadata}} in ddb, so that was a logical choice. Big thanks to [~fabbri] for the continuous help, and for the original idea on how to solve this issue. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559535#comment-16559535 ] Gabor Bota commented on HADOOP-14154: - {{ITestDynamoDBMetadataStore#testListChildrenAuthoritative}} is the test for this issue feature which is skipped right now because {{PERSISTS_AUTHORITATIVE_BIT}} is false for ddbms at the moment. This will be changed with this patch, so the {{testListChildrenAuthoritative}} will run. Added HADOOP-15636 as blocking. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558672#comment-16558672 ] Sean Mackrory commented on HADOOP-14154: {quote}I don't think that if there would be any trouble if a client deletes the isAuth bit in case a record gets overwritten. In that case the listing just won't be authoritative.{quote} I think this point is still valid despite tables being shared between clusters & applications. If the bit is deleted, this feature simply degrades to current behavior. So you can mix old and new code, you just don't get all of the benefit of the new code until you're completely on it. There's a possibility that there's a bit that gets deleted while code is still running that read a true bit, but we have a bit of that today anyway. Something might be in the process of being deleted when other running code has already listed it, for instance. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558664#comment-16558664 ] Steve Loughran commented on HADOOP-14154: - bq. Customers often share a single dynamo table between multiple clusters (it is required if they modify the same bucket). +1. Apps get updated at a different rate. You can't stop a remote distcp from an older version uploading stuff to a s3guarded bucket > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558656#comment-16558656 ] Aaron Fabbri commented on HADOOP-14154: --- {quote} As my current understanding using a dynamodb table is exclusive per ddbms, so there should be no case when the same table gets modified by an old and a new metadata store version at the same time. {quote} Customers often share a single dynamo table between multiple clusters (it is required if they modify the same bucket). I would actually expect multiple versions to concurrently access the same table. Seems like we should try to allow it unless there is a good reason to break compatibility. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558201#comment-16558201 ] Gabor Bota commented on HADOOP-14154: - I've uploaded the spec with the fixes. Thanks [~fabbri], [~ste...@apache.org]. [~ste...@apache.org], I don't think that if there would be any trouble if a client deletes the isAuth bit in case a record gets overwritten. In that case the listing just won't be authoritative. As my current understanding using a dynamodb table is exclusive per ddbms, so there should be no case when the same table gets modified by an old and a new metadata store version at the same time. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551453#comment-16551453 ] Steve Loughran commented on HADOOP-14154: - when the patch goes in, this doc needs to be in the s3guard docs as markdown somehow; maybe a whole new doc * spelling of "isAuthoratitave" wrong at the bottom of p1 * compatibility: what happens if old client does a listing and updates a new DB? Will the old value be retained? Presumably its implicit that no is_authoritative column means: !is_authoritative This patch nice as its update strategy is simple: writes from new clients add the column. It does mean that if you run the new client against an old table the old table gets updated, but that's ok, provided the next listing by an old client deletes that isAuth bit. If the bit remained set while things got updated by an old client, you could be in trouble > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551078#comment-16551078 ] Aaron Fabbri commented on HADOOP-14154: --- Sounds good to me. I'll leave the decision how to split this up to you (separate jiras are fine with me). > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550753#comment-16550753 ] Gabor Bota commented on HADOOP-14154: - Thanks [~fabbri] for the review. I will add a new version of the doc with your suggestion to this issue. I don't really know if we should combine this with HADOOP-15621 in the same jira, and same document. I think the a good idea would be to extend this document with the ttl, and upload it to HADOOP-15621. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549960#comment-16549960 ] Aaron Fabbri commented on HADOOP-14154: --- Doc looks great [~gabor.bota]. One minor comment: {quote} Authoritative directory listings (isAuthoritative bit) * Tells if the stored directory listing metadata is complete. * This is set by the internal logic of the metadata store in the DirListingMetadata(org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata). {quote} The last point, I suggest clarifying to: {quote} This is set by the FileSystem client (e.g. s3a) via the DirListingMetadata class {quote} Basically, the MetadataStore only knows what the FS client tells it. We might want to also talk about adding a "entry last written timestamp" column as well while we are at it. I created HADOOP-15621 to describe the need for this. Might be easier to add this to your doc here and we can review all the compatibility concerns together in one place? > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541384#comment-16541384 ] Sean Mackrory commented on HADOOP-14154: +1 on the doc, Gabor. This sort of relates to HADOOP-14335 in that it is a schema change. Gabor and I had a bit of a discussion about that one the other day (I'm in the middle of posting a summary of that discussion to that JIRA). Despite being a schema change, I don't think there's a need to update the schema version because the version is essentially used as a compatibility version only. This is a compatible change if new code assumes that anything missing the is_authoritative bit is essentially NOT authoritative (i.e. it defaults to false). Old code won't look at the flag, and the effect will simply be the same as before we entered this change. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540271#comment-16540271 ] Gabor Bota commented on HADOOP-14154: - I've added a specification for the solution to this issue. I'm planning to use the text inside the document to do a merge what is already in HADOOP-15400 description to resolve that doc issue - so to clarify which authoritative means what. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > - > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org