[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=765442=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765442 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 03/May/22 13:29 Start Date: 03/May/22 13:29 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1116099593 merged to trunk, albeit with the commit message partially cut. reverted that and recommitted with the full text. test with branch-3.3 in progress Issue Time Tracking --- Worklog Id: (was: 765442) Time Spent: 4h 10m (was: 4h) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=765439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765439 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 03/May/22 13:26 Start Date: 03/May/22 13:26 Worklog Time Spent: 10m Work Description: steveloughran merged PR #4140: URL: https://github.com/apache/hadoop/pull/4140 Issue Time Tracking --- Worklog Id: (was: 765439) Time Spent: 4h (was: 3h 50m) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=764136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764136 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 29/Apr/22 10:25 Start Date: 29/Apr/22 10:25 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1113150246 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 41s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 39m 27s | | trunk passed | | +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 46s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 0s | | trunk passed | | +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 35s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 22m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | | the patch passed | | +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 40s | | the patch passed | | +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 37s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 24s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 41s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 17s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 52s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 34s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. | | | | 98m 49s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/12/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle xml | | uname | Linux b8109ee29d83 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 6302c5b9e54e55153f3afa895c8dc4d032025407 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/12/testReport/ | | Max.
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=764126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764126 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 29/Apr/22 09:59 Start Date: 29/Apr/22 09:59 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1113130188 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 52s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 39m 34s | | trunk passed | | +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 36s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 47s | | trunk passed | | +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 26s | | trunk passed | | +1 :green_heart: | shadedclient | 22m 20s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 22m 39s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 35s | | the patch passed | | +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 40s | | the patch passed | | +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 31s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 20s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 37s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 14s | | the patch passed | | +1 :green_heart: | shadedclient | 22m 35s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 24s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. | | | | 99m 13s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux db4abd57055b 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / cb19dff096688c442e76cba672376eb7cb72161a | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/10/testReport/ | | Max. process+thread count | 690 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U:
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=764086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764086 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 29/Apr/22 08:56 Start Date: 29/Apr/22 08:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1113072418 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 43s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | -1 :x: | mvninstall | 0m 33s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. | | -1 :x: | compile | 0m 33s | [/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt) | hadoop-aws in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. | | -1 :x: | compile | 0m 33s | [/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-aws in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. | | -0 :warning: | checkstyle | 0m 30s | [/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt) | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 32s | [/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in trunk failed. | | -1 :x: | javadoc | 0m 32s | [/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt) | hadoop-aws in trunk failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. | | -1 :x: | javadoc | 0m 32s | [/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-aws in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. | | -1 :x: | spotbugs | 0m 32s | [/branch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in trunk failed. | | +1 :green_heart: | shadedclient | 3m 46s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 4m 19s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 33s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. | | -1 :x: | compile | 0m 32s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/11/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04. | | -1 :x: | javac | 0m 32s |
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=764078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764078 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 29/Apr/22 08:40 Start Date: 29/Apr/22 08:40 Worklog Time Spent: 10m Work Description: dannycjones commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1113037864 Thanks @steveloughran, I've added a note to `testing.md`. I am hoping that if we get all public datasets into `PublicDatasetTestUtils`, we can slim down the notes in `testing.md` and instead point the tester to review `PublicDatasetTestUtils` to understand which configurations to update. For now, I have not tried to move the CSV Scale tests but I'd like to see them moved in a future patch. Issue Time Tracking --- Worklog Id: (was: 764078) Time Spent: 3h 20m (was: 3h 10m) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=764077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764077 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 29/Apr/22 08:38 Start Date: 29/Apr/22 08:38 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4140: URL: https://github.com/apache/hadoop/pull/4140#discussion_r861581105 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java: ## @@ -102,14 +88,30 @@ public void testRequesterPaysDisabledFails() throws Throwable { } } - private Path getRequesterPaysPath(Configuration conf) { -String requesterPaysFile = -conf.getTrimmed(KEY_REQUESTER_PAYS_FILE, DEFAULT_REQUESTER_PAYS_FILE); -S3ATestUtils.assume( -"Empty test property: " + KEY_REQUESTER_PAYS_FILE, -!requesterPaysFile.isEmpty() -); -return new Path(requesterPaysFile); + /** + * Use this after creating the file system, as this is when bucket-specific + * overrides are applied. + * @param conf Hadoop configuration from FS to mutate + * @param requesterPaysEnabled Indicate if requester pays be on or off + */ + private static void updateConf( + Configuration conf, + boolean requesterPaysEnabled Review Comment: makes sense, will note for future. i removed this code now since the change was unnecessary and was making the patch longer. (I had some confusion around the bucket conf propagation when i wrote it) Issue Time Tracking --- Worklog Id: (was: 764077) Time Spent: 3h 10m (was: 3h) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763813 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 28/Apr/22 18:42 Start Date: 28/Apr/22 18:42 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1112541391 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 35s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 44m 31s | | trunk passed | | +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 50s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 2s | | trunk passed | | +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 31s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 43s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 22m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 39s | | the patch passed | | +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 45s | | the patch passed | | +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 36s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 25s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 43s | | the patch passed | | +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 14s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 45s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 29s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 103m 52s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux 6a40a55d820d 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 73eeb50311a3e06deaafe4b7ceb1fbb72107f538 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/9/testReport/ | | Max. process+thread count | 624 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U:
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763782 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 28/Apr/22 18:16 Start Date: 28/Apr/22 18:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1112520064 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 49s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 38m 46s | | trunk passed | | +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 36s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 49s | | trunk passed | | +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 22s | | trunk passed | | +1 :green_heart: | shadedclient | 22m 42s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 23m 1s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 35s | | the patch passed | | +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 42s | | the patch passed | | +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 31s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/8/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | mvnsite | 0m 39s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 8s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 51s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 21s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. | | | | 97m 2s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux 341d2ef8742d 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 364ce9eba20743aad75689f92cd14e0517ce0a5e | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763727 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 28/Apr/22 17:18 Start Date: 28/Apr/22 17:18 Worklog Time Spent: 10m Work Description: dannycjones commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1112461995 Tested latest patch against `eu-west-1`, all OK Issue Time Tracking --- Worklog Id: (was: 763727) Time Spent: 2h 40m (was: 2.5h) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763726 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 28/Apr/22 17:17 Start Date: 28/Apr/22 17:17 Worklog Time Spent: 10m Work Description: steveloughran commented on code in PR #4140: URL: https://github.com/apache/hadoop/pull/4140#discussion_r861131956 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/PublicDatasetTestUtils.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; Review Comment: can you put in the sub package .test ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java: ## @@ -18,6 +18,8 @@ package org.apache.hadoop.fs.s3a; +import org.apache.hadoop.conf.Configuration; Review Comment: nit: import should go below the java. one ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java: ## @@ -102,14 +88,30 @@ public void testRequesterPaysDisabledFails() throws Throwable { } } - private Path getRequesterPaysPath(Configuration conf) { -String requesterPaysFile = -conf.getTrimmed(KEY_REQUESTER_PAYS_FILE, DEFAULT_REQUESTER_PAYS_FILE); -S3ATestUtils.assume( -"Empty test property: " + KEY_REQUESTER_PAYS_FILE, -!requesterPaysFile.isEmpty() -); -return new Path(requesterPaysFile); + /** + * Use this after creating the file system, as this is when bucket-specific + * overrides are applied. + * @param conf Hadoop configuration from FS to mutate + * @param requesterPaysEnabled Indicate if requester pays be on or off + */ + private static void updateConf( + Configuration conf, + boolean requesterPaysEnabled Review Comment: nit: we tend to pull that ) onto the same line as the last paraem Issue Time Tracking --- Worklog Id: (was: 763726) Time Spent: 2.5h (was: 2h 20m) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763040 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 27/Apr/22 16:43 Start Date: 27/Apr/22 16:43 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-223043 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 48s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 39m 56s | | trunk passed | | +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 34s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | | trunk passed | | +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 18s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 1s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 20m 20s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 38s | | the patch passed | | +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 29s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 20s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 36s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 10s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 30s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 23s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. | | | | 94m 42s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux 593ad08f3277 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 1692c71204dcbb340d35ed561991cf680864885a | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/7/testReport/ | | Max. process+thread count | 546 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U:
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763023 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 27/Apr/22 16:20 Start Date: 27/Apr/22 16:20 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-199931 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 45s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 38m 45s | | trunk passed | | +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 52s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 2s | | trunk passed | | +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 35s | | trunk passed | | +1 :green_heart: | shadedclient | 21m 2s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 21m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 42s | | the patch passed | | +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 44s | | the patch passed | | +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 39s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 27s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | mvnsite | 0m 43s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 13s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 24s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 42s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. | | | | 98m 5s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux 0e78edaed65a 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 79710a1d5e3946c9092d1826c72a13a1dbf4 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762940 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 27/Apr/22 14:55 Start Date: 27/Apr/22 14:55 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-104577 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 45s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 39m 20s | | trunk passed | | +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 36s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | | trunk passed | | +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 16s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 15s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 20m 35s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 0m 36s | | the patch passed | | +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 30s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | mvnsite | 0m 35s | | the patch passed | | +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 14s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 35s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 24s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. | | | | 94m 45s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/4140 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml | | uname | Linux 3f7569ed9f48 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / e734caabaaa94a387891d721f08f54a342f7b250 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762867 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 27/Apr/22 13:18 Start Date: 27/Apr/22 13:18 Worklog Time Spent: 10m Work Description: dannycjones commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1110990869 Tests passing against `eu-west-1` bucket. Issue Time Tracking --- Worklog Id: (was: 762867) Time Spent: 1h 50m (was: 1h 40m) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762757 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 27/Apr/22 08:49 Start Date: 27/Apr/22 08:49 Worklog Time Spent: 10m Work Description: dannycjones commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1110735715 Integration tests not currently passing with this commit, diving deeper here. Issue Time Tracking --- Worklog Id: (was: 762757) Time Spent: 1h 40m (was: 1.5h) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762417 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 26/Apr/22 16:51 Start Date: 26/Apr/22 16:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on PR #4140: URL: https://github.com/apache/hadoop/pull/4140#issuecomment-1110029320 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 46s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 39m 40s | | trunk passed | | +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 34s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 46s | | trunk passed | | +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 20s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 35s | | the patch passed | | +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | -1 :x: | javac | 0m 38s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 generated 10 new + 17 unchanged - 0 fixed = 27 total (was 17) | | +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | -1 :x: | javac | 0m 32s | [/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 10 new + 17 unchanged - 0 fixed = 27 total (was 17) | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | -0 :warning: | checkstyle | 0m 19s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4140/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | mvnsite | 0m 36s | | the patch passed | | +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 7s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 41s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 19s | | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. | | | | 95m 3s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Work logged] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged
[ https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=762384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-762384 ] ASF GitHub Bot logged work on HADOOP-18168: --- Author: ASF GitHub Bot Created on: 26/Apr/22 15:16 Start Date: 26/Apr/22 15:16 Worklog Time Spent: 10m Work Description: dannycjones commented on code in PR #4140: URL: https://github.com/apache/hadoop/pull/4140#discussion_r858847636 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java: ## @@ -108,6 +108,21 @@ */ String DEFAULT_REQUESTER_PAYS_FILE = "s3a://usgs-landsat/collection02/catalog.json"; + /** + * Configuration key for an existing bucket with many objects: {@value}. + * + * This is used for tests depending on buckets with a large number of keys. + */ + String KEY_BUCKET_WITH_MANY_OBJECTS + = TEST_FS_S3A + "bucket-with-many-objects"; + + /** + * Default bucket for when {@value KEY_BUCKET_WITH_MANY_OBJECTS} is not set: + * {@value}. + */ + String DEFAULT_BUCKET_MANY_OBJECTS + = "s3a://commoncrawl/"; Review Comment: migrating to `usgs-landsat` https://registry.opendata.aws/usgs-landsat/index.html Issue Time Tracking --- Worklog Id: (was: 762384) Time Spent: 1h 20m (was: 1h 10m) > ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket > content purged > --- > > Key: HADOOP-18168 > URL: https://issues.apache.org/jira/browse/HADOOP-18168 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Daniel Carl Jones >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was > meant to stop after the first page of results is finishing because there > aren't so many objects there. > > first visible sign of the landsat-pds cleanup > now we have requester pays, we could do this against another store with > stability promises, e.g common crawl. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org