[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=479116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479116 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 04/Sep/20 13:58 Start Date: 04/Sep/20 13:58 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2254: URL: https://github.com/apache/hadoop/pull/2254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479116) Time Spent: 2h 10m (was: 2h) > improve s3guard markers command line tool > - > > Key: HADOOP-17227 > URL: https://issues.apache.org/jira/browse/HADOOP-17227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > The s3guard markers audit -expected N command is meant to verify that the > marker count is N, but > * it isn't verified if the #of markers is 0, so you can't use it to assert > that markers have been created > * it doesn't work for tests where you expect a minimum number of markers. > It's essentially setting a max #of markers > Proposed: explicit -min, -max args to declare a specific range of values -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=478761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478761 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 03/Sep/20 18:49 Start Date: 03/Sep/20 18:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-686692132 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 10s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 42s | trunk passed | | +1 :green_heart: | shadedclient | 15m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 3s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 36s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 22s | hadoop-tools/hadoop-aws: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11) | | +1 :green_heart: | mvnsite | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 47s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. | | | | 71m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2254 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 6007ceecd302 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 5c158157739 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/5/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=478588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478588 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 03/Sep/20 14:07 Start Date: 03/Sep/20 14:07 Worklog Time Spent: 10m Work Description: bgaborg commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-686516851 Nice improvement Steve, LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 478588) Time Spent: 1h 50m (was: 1h 40m) > improve s3guard markers command line tool > - > > Key: HADOOP-17227 > URL: https://issues.apache.org/jira/browse/HADOOP-17227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > The s3guard markers audit -expected N command is meant to verify that the > marker count is N, but > * it isn't verified if the #of markers is 0, so you can't use it to assert > that markers have been created > * it doesn't work for tests where you expect a minimum number of markers. > It's essentially setting a max #of markers > Proposed: explicit -min, -max args to declare a specific range of values -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=478558&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478558 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 03/Sep/20 13:31 Start Date: 03/Sep/20 13:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-686493179 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 2s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 42s | trunk passed | | +1 :green_heart: | shadedclient | 15m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 2s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 29s | the patch passed | | -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-aws: The patch generated 8 new + 11 unchanged - 0 fixed = 19 total (was 11) | | +1 :green_heart: | mvnsite | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 30s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 72m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2254 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 0ced8146f717 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 5e12dc51f06 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/4/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=478519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478519 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 03/Sep/20 12:16 Start Date: 03/Sep/20 12:16 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-682802215 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 478519) Time Spent: 1.5h (was: 1h 20m) > improve s3guard markers command line tool > - > > Key: HADOOP-17227 > URL: https://issues.apache.org/jira/browse/HADOOP-17227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The s3guard markers audit -expected N command is meant to verify that the > marker count is N, but > * it isn't verified if the #of markers is 0, so you can't use it to assert > that markers have been created > * it doesn't work for tests where you expect a minimum number of markers. > It's essentially setting a max #of markers > Proposed: explicit -min, -max args to declare a specific range of values -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=478064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478064 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 02/Sep/20 18:47 Start Date: 02/Sep/20 18:47 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-685928379 + for testing, it'd be good if we could count objects -minobjects, -maxobjects, which would be for all objects, markers included, under a path. Helps verify rename &c, even when s3guard is enabled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 478064) Time Spent: 1h 20m (was: 1h 10m) > improve s3guard markers command line tool > - > > Key: HADOOP-17227 > URL: https://issues.apache.org/jira/browse/HADOOP-17227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > The s3guard markers audit -expected N command is meant to verify that the > marker count is N, but > * it isn't verified if the #of markers is 0, so you can't use it to assert > that markers have been created > * it doesn't work for tests where you expect a minimum number of markers. > It's essentially setting a max #of markers > Proposed: explicit -min, -max args to declare a specific range of values -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[
https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=477980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477980
]
ASF GitHub Bot logged work on HADOOP-17227:
---
Author: ASF GitHub Bot
Created on: 02/Sep/20 17:08
Start Date: 02/Sep/20 17:08
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2254:
URL: https://github.com/apache/hadoop/pull/2254#issuecomment-685873527
aah, the builder API raises checkstyles
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:762:
public static final String PURPOSE = "destroy the Metadata Store including
its": Line is longer than 80 characters (found 83). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:933:
public ScanArgsBuilder withSourceFS(final FileSystem sourceFS) {:58:
'sourceFS' hides a field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:939:
public ScanArgsBuilder withPath(final Path path) {:48: 'path' hides a
field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:945:
public ScanArgsBuilder withDoPurge(final boolean doPurge) {:54: 'doPurge'
hides a field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:951:
public ScanArgsBuilder withMinMarkerCount(final int minMarkerCount) {:57:
'minMarkerCount' hides a field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:957:
public ScanArgsBuilder withMaxMarkerCount(final int maxMarkerCount) {:57:
'maxMarkerCount' hides a field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:963:
public ScanArgsBuilder withLimit(final int limit) {:48: 'limit' hides a
field. [HiddenField]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:969:
public ScanArgsBuilder withNonAuth(final boolean nonAuth) {:54: 'nonAuth'
hides a field. [HiddenField]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 477980)
Time Spent: 1h 10m (was: 1h)
> improve s3guard markers command line tool
> -
>
> Key: HADOOP-17227
> URL: https://issues.apache.org/jira/browse/HADOOP-17227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The s3guard markers audit -expected N command is meant to verify that the
> marker count is N, but
> * it isn't verified if the #of markers is 0, so you can't use it to assert
> that markers have been created
> * it doesn't work for tests where you expect a minimum number of markers.
> It's essentially setting a max #of markers
> Proposed: explicit -min, -max args to declare a specific range of values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=477966&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477966 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 02/Sep/20 16:36 Start Date: 02/Sep/20 16:36 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-68582 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 28s | trunk passed | | +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 41s | trunk passed | | +1 :green_heart: | shadedclient | 15m 29s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 3s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 37s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 8 new + 11 unchanged - 0 fixed = 19 total (was 11) | | +1 :green_heart: | mvnsite | 0m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 101m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2254 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 7b6e7f0adea4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0207f5cf461 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/3/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[
https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=476215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476215
]
ASF GitHub Bot logged work on HADOOP-17227:
---
Author: ASF GitHub Bot
Created on: 30/Aug/20 10:32
Start Date: 30/Aug/20 10:32
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2254:
URL: https://github.com/apache/hadoop/pull/2254#issuecomment-683403665
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:28:import
java.time.Duration;:8: Unused import - java.time.Duration. [UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java:763:
public static final String PURPOSE = "destroy the Metadata Store including
its contents": Line is longer than 80 characters (found 92). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:332:
String.format("Argument for %s is not a number: %s", option,
value));: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:435:
/**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:622:
private ScanResult failScan(ScanResult result, int code, String message,
Object...args) {: Line is longer than 80 characters (found 91). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476215)
Time Spent: 50m (was: 40m)
> improve s3guard markers command line tool
> -
>
> Key: HADOOP-17227
> URL: https://issues.apache.org/jira/browse/HADOOP-17227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The s3guard markers audit -expected N command is meant to verify that the
> marker count is N, but
> * it isn't verified if the #of markers is 0, so you can't use it to assert
> that markers have been created
> * it doesn't work for tests where you expect a minimum number of markers.
> It's essentially setting a max #of markers
> Proposed: explicit -min, -max args to declare a specific range of values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=475882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475882 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 28/Aug/20 16:08 Start Date: 28/Aug/20 16:08 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-682802215 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 28m 33s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 30s | trunk passed | | +1 :green_heart: | mvnsite | 0m 43s | trunk passed | | +1 :green_heart: | shadedclient | 15m 15s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 5s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 29s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 6 new + 11 unchanged - 0 fixed = 17 total (was 11) | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 30s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 70m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2254 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 4053d53801ae 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / c392d9022a3 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/2/testReport/ | | Max. process+thread count | 431 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[ https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=475851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475851 ] ASF GitHub Bot logged work on HADOOP-17227: --- Author: ASF GitHub Bot Created on: 28/Aug/20 14:58 Start Date: 28/Aug/20 14:58 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2254: URL: https://github.com/apache/hadoop/pull/2254#issuecomment-682217451 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 29m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 48s | trunk passed | | +1 :green_heart: | compile | 0m 46s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 31s | trunk passed | | +1 :green_heart: | mvnsite | 0m 47s | trunk passed | | +1 :green_heart: | shadedclient | 15m 33s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 13s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 38s | the patch passed | | +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 37s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 30s | the patch passed | | -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11) | | +1 :green_heart: | mvnsite | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 32s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 103m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2254 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux f8be76bcca73 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 06793da1001 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2254/1/testReport/ | | Max. process+thread count | 452 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console o
[jira] [Work logged] (HADOOP-17227) improve s3guard markers command line tool
[
https://issues.apache.org/jira/browse/HADOOP-17227?focusedWorklogId=475849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475849
]
ASF GitHub Bot logged work on HADOOP-17227:
---
Author: ASF GitHub Bot
Created on: 28/Aug/20 14:55
Start Date: 28/Aug/20 14:55
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2254:
URL: https://github.com/apache/hadoop/pull/2254#issuecomment-682660636
bucket info gets improved too, based on working on the CLI.
```
bin/hadoop s3guard bucket-info s3a://stevel-london/
2020-08-28 15:45:02,624 [main] INFO impl.DirectoryPolicyImpl
(DirectoryPolicyImpl.java:getDirectoryPolicy(193)) - Directory markers will be
kept on authoritative paths
Filesystem s3a://stevel-london
Location: eu-west-2
Filesystem s3a://stevel-london is using S3Guard with store
DynamoDBMetadataStore{region=eu-west-2, tableName=stevel-london,
tableArn=arn:aws:dynamodb:eu-west-2:152813717728:table/stevel-london}
Authoritative Metadata Store: fs.s3a.metadatastore.authoritative=false
Authoritative Path: fs.s3a.authoritative.path=/tables
Qualified Authoritative Paths:
s3a://stevel-london/tables/
Metadata time to live: (set in fs.s3a.metadatastore.metadata.ttl) =
00:15:00.000
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:eu-west-2:152813717728:table/stevel-london
billing-mode=per-request
description=S3Guard metadata store in DynamoDB
name=stevel-london
persist.authoritative.bit=true
read-capacity=0
region=eu-west-2
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100
MILLISECONDS)
size=38538
sse=DISABLED
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName: stevel-london,KeySchema:
[{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType:
RANGE}],TableStatus: ACTIVE,CreationDateTime: Mon Mar 16 20:21:32 GMT
2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits:
0,WriteCapacityUnits: 0},TableSizeBytes: 38538,ItemCount: 300,TableArn:
arn:aws:dynamodb:eu-west-2:152813717728:table/stevel-london,TableId:
422878eb-823c-4071-826a-3746b7c8fd18,BillingModeSummary: {BillingMode:
PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 16 20:21:32 GMT
2020},}
write-capacity=0
S3A Client
Signing Algorithm: fs.s3a.signing-algorithm=(unset)
Endpoint: fs.s3a.endpoint=s3.eu-west-2.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
Change Detection Source: fs.s3a.change.detection.source=etag
Change Detection Mode: fs.s3a.change.detection.mode=server
S3A Committers
The "magic" committer is supported in the filesystem
S3A Committer factory class:
mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
S3A Committer name: fs.s3a.committer.name=directory
Cluster filesystem staging directory:
fs.s3a.committer.staging.tmp.path=tmp/staging
Local filesystem buffer directory:
fs.s3a.buffer.dir=/tmp/hadoop-stevel/s3a
File conflict resolution: fs.s3a.committer.staging.conflict-mode=append
Security
Delegation token support is disabled
Security
The directory marker policy is "authoritative"
Available Policies: delete, keep, authoritative
Authoritative paths: fs.s3a.authoritative.path=/tables
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 475849)
Time Spent: 20m (was: 10m)
> improve s3guard markers command line tool
> -
>
> Key: HADOOP-17227
> URL: https://issues.apache.org/jira/browse/HADOOP-17227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The s3guard markers audit -expected N command is meant to verify that the
> marker count is N, but
> * it isn't verified if the #of markers is 0, so you can't use it to assert
> that markers have been created
> * it doesn't work for tests where you expect a minimum number of markers.
> It's essentially setting
