[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646822#comment-16646822 ] Steve Loughran commented on HADOOP-15428: - linking to HADOOP-14109 which also has some autocreate issues. proposed: info, destroy are both non-creating; init and destroy must have a URI or FS > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708785#comment-16708785 ] Gabor Bota commented on HADOOP-15428: - submitted patch v001, tested against eu-west-1. Issues: * org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster#testWithMiniCluster (known) * ITestS3GuardToolDynamoDB#testPruneCommandCLI (added to [S3 flakyness collecting spreadsheet| https://docs.google.com/spreadsheets/d/1Z9dkg5yC7Hu7VQ5G2Wz40hG1pb0DhLuLjJ3PZe7WL3Q/edit?usp=sharing) > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-15428.001.patch > > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708912#comment-16708912 ] Hadoop QA commented on HADOOP-15428: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 49s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 55s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 35s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 66m 56s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-15428 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12950566/HADOOP-15428.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2f2333f5f244 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ff31313 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15606/testReport/ | | Max. process+thread count | 309 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15606/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > -
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713101#comment-16713101 ] Sean Mackrory commented on HADOOP-15428: +1 on the patch, with the caveat that I'm gonna change the usage text from "Require S3Guard to be disabled" to "Force S3Guard to be disabled", since this does actually proceed with modified behavior instead of failing if the conditions aren't met. > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-15428.001.patch > > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715829#comment-16715829 ] Hudson commented on HADOOP-15428: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15585 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15585/]) HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set (mackrorysd: rev 3ff8580f2289664a54f369b500ad77f31a07c326) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15428.001.patch > > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718503#comment-16718503 ] Gabor Bota commented on HADOOP-15428: - What about this as a relnote: If -unguarded flag passed to `hadoop s3guard bucket-info`, it will proceed with S3Guard disabled instead of failing if S3Guard is not already disabled. > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15428.001.patch > > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[ https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719117#comment-16719117 ] Sean Mackrory commented on HADOOP-15428: Yeah that's more direct. Tweaked it a bit and updated. I'm happy with it, so I'll resolve. Feel free to discuss if there's more feedback... > s3guard bucket-info will create s3guard table if FS is set to do this > automatically > --- > > Key: HADOOP-15428 > URL: https://issues.apache.org/jira/browse/HADOOP-15428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15428.001.patch > > > If you call hadoop s3guard bucket-info on a bucket where the fs is set to > create a s3guard table on demand, then the DDB table is automatically > created. As a result > the {{bucket-info -unguarded}} option cannot be used, and the call has > significant side effects (i.e. it can run up bills) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org