[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.1.0
[
https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=529919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529919
]
ASF GitHub Bot logged work on HADOOP-17452:
---
Author: ASF GitHub Bot
Created on: 01/Jan/21 01:47
Start Date: 01/Jan/21 01:47
Worklog Time Spent: 10m
Work Description: wangyum commented on pull request #2582:
URL: https://github.com/apache/hadoop/pull/2582#issuecomment-753239565
@aajisaka
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 529919)
Time Spent: 0.5h (was: 20m)
> Upgrade guice to 4.1.0
> --
>
> Key: HADOOP-17452
> URL: https://issues.apache.org/jira/browse/HADOOP-17452
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Upgrade guice to 4.1.0 to fix compatibility issue:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
> » at
> com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258)
> » at
> com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
> » at
> com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
> » at
> org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
> » at
> org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
> » at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
> » at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at
> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
> » at
> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
> » at com.google.inject.Guice.createInjector(Guice.java:96)
> » at com.google.inject.Guice.createInjector(Guice.java:73)
> » at com.google.inject.Guice.createInjector(Guice.java:62)
> » at
> org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
> » at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
> » at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
> » at org.apache.druid.cli.Main.main(Main.java:113)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] wangyum commented on pull request #2582: HADOOP-17452. Upgrade Guice to 4.1.0
wangyum commented on pull request #2582: URL: https://github.com/apache/hadoop/pull/2582#issuecomment-753239565 @aajisaka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.1.0
[
https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=529918&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529918
]
ASF GitHub Bot logged work on HADOOP-17452:
---
Author: ASF GitHub Bot
Created on: 01/Jan/21 01:36
Start Date: 01/Jan/21 01:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2582:
URL: https://github.com/apache/hadoop/pull/2582#issuecomment-753238524
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 1s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 70m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2582 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux ebe2254660e7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 617af28e807 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
[GitHub] [hadoop] hadoop-yetus commented on pull request #2582: HADOOP-17452. Upgrade Guice to 4.1.0
hadoop-yetus commented on pull request #2582:
URL: https://github.com/apache/hadoop/pull/2582#issuecomment-753238524
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 1s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 70m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2582 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux ebe2254660e7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 617af28e807 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2582/1/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17452) Upgrade guice to 4.1.0
[
https://issues.apache.org/jira/browse/HADOOP-17452?focusedWorklogId=529916&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529916
]
ASF GitHub Bot logged work on HADOOP-17452:
---
Author: ASF GitHub Bot
Created on: 01/Jan/21 00:23
Start Date: 01/Jan/21 00:23
Worklog Time Spent: 10m
Work Description: wangyum opened a new pull request #2582:
URL: https://github.com/apache/hadoop/pull/2582
Upgrade Guice to 4.1.0 to fix compatibility issue with other projects, e.g.:
Druid:
```
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
» at
com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
» at
com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283)
» at
com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258)
» at
com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
» at
com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
» at
org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
» at
org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
» at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at
com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at
com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at
com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
» at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
» at com.google.inject.Guice.createInjector(Guice.java:96)
» at com.google.inject.Guice.createInjector(Guice.java:73)
» at com.google.inject.Guice.createInjector(Guice.java:62)
» at
org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
» at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
» at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
» at org.apache.druid.cli.Main.main(Main.java:113)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 529916)
Remaining Estimate: 0h
Time Spent: 10m
> Upgrade guice to 4.1.0
> --
>
> Key: HADOOP-17452
> URL: https://issues.apache.org/jira/browse/HADOOP-17452
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Yuming Wang
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Upgrade guice to 4.1.0 to fix compatibility issue:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
> » at
> com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258)
> » at
> com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
> » at
> com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
> » at
> org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
> » at
> org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
> » at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements
[jira] [Updated] (HADOOP-17452) Upgrade guice to 4.1.0
[
https://issues.apache.org/jira/browse/HADOOP-17452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17452:
Labels: pull-request-available (was: )
> Upgrade guice to 4.1.0
> --
>
> Key: HADOOP-17452
> URL: https://issues.apache.org/jira/browse/HADOOP-17452
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Upgrade guice to 4.1.0 to fix compatibility issue:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
> » at
> com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283)
> » at
> com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258)
> » at
> com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
> » at
> com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
> » at
> org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
> » at
> org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
> » at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
> » at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
> » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
> » at com.google.inject.spi.Elements.getElements(Elements.java:110)
> » at
> com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
> » at
> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
> » at com.google.inject.Guice.createInjector(Guice.java:96)
> » at com.google.inject.Guice.createInjector(Guice.java:73)
> » at com.google.inject.Guice.createInjector(Guice.java:62)
> » at
> org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
> » at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
> » at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
> » at org.apache.druid.cli.Main.main(Main.java:113)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] wangyum opened a new pull request #2582: HADOOP-17452. Upgrade Guice to 4.1.0
wangyum opened a new pull request #2582: URL: https://github.com/apache/hadoop/pull/2582 Upgrade Guice to 4.1.0 to fix compatibility issue with other projects, e.g.: Druid: ``` Exception in thread "main" java.lang.NoSuchMethodError: com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType; » at com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202) » at com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283) » at com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258) » at com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178) » at com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150) » at org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115) » at org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121) » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) » at com.google.inject.spi.Elements.getElements(Elements.java:110) » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177) » at com.google.inject.AbstractModule.configure(AbstractModule.java:62) » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) » at com.google.inject.spi.Elements.getElements(Elements.java:110) » at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177) » at com.google.inject.AbstractModule.configure(AbstractModule.java:62) » at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) » at com.google.inject.spi.Elements.getElements(Elements.java:110) » at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) » at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) » at com.google.inject.Guice.createInjector(Guice.java:96) » at com.google.inject.Guice.createInjector(Guice.java:73) » at com.google.inject.Guice.createInjector(Guice.java:62) » at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431) » at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69) » at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58) » at org.apache.druid.cli.Main.main(Main.java:113) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17452) Upgrade guice to 4.1.0
Yuming Wang created HADOOP-17452:
Summary: Upgrade guice to 4.1.0
Key: HADOOP-17452
URL: https://issues.apache.org/jira/browse/HADOOP-17452
Project: Hadoop Common
Issue Type: Improvement
Reporter: Yuming Wang
Upgrade guice to 4.1.0 to fix compatibility issue:
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.inject.util.Types.collectionOf(Ljava/lang/reflect/Type;)Ljava/lang/reflect/ParameterizedType;
» at
com.google.inject.multibindings.Multibinder.collectionOfProvidersOf(Multibinder.java:202)
» at
com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:283)
» at
com.google.inject.multibindings.Multibinder$RealMultibinder.(Multibinder.java:258)
» at
com.google.inject.multibindings.Multibinder.newRealSetBinder(Multibinder.java:178)
» at
com.google.inject.multibindings.Multibinder.newSetBinder(Multibinder.java:150)
» at
org.apache.druid.guice.LifecycleModule.getEagerBinder(LifecycleModule.java:115)
» at org.apache.druid.guice.LifecycleModule.configure(LifecycleModule.java:121)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)
» at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
» at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
» at com.google.inject.spi.Elements.getElements(Elements.java:110)
» at
com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
» at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
» at com.google.inject.Guice.createInjector(Guice.java:96)
» at com.google.inject.Guice.createInjector(Guice.java:73)
» at com.google.inject.Guice.createInjector(Guice.java:62)
» at
org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
» at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
» at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
» at org.apache.druid.cli.Main.main(Main.java:113)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16830) Add Public IOStatistics API
[ https://issues.apache.org/jira/browse/HADOOP-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257106#comment-17257106 ] Steve Loughran commented on HADOOP-16830: - OK: core hadoop-common and S3A changes are all in. made it in 2020 by 117 minutes UTC! > Add Public IOStatistics API > --- > > Key: HADOOP-16830 > URL: https://issues.apache.org/jira/browse/HADOOP-16830 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 9h 40m > Remaining Estimate: 0h > > Applications like to collect the statistics which specific operations take, > by collecting exactly those operations done during the execution of FS API > calls by their individual worker threads, and returning these to their job > driver > * S3A has a statistics API for some streams, but it's a non-standard one; > Impala &c can't use it > * FileSystem storage statistics are public, but as they aren't cross-thread, > they don't aggregate properly > Proposed > # A new IOStatistics interface to serve up statistics > # S3A to implement > # other stores to follow > # Pass-through from the usual wrapper classes (FS data input/output streams) > It's hard to think about how best to offer an API for operation context > stats, and how to actually implement. > ThreadLocal isn't enough because the helper threads need to update on the > thread local value of the instigator > My Initial PoC doesn't address that issue, but it shows what I'm thinking of -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-17451: --- Assignee: Steve Loughran > intermittent failure of S3A huge file upload tests: count of bytes uploaded > == 0 > > > Key: HADOOP-17451 > URL: https://issues.apache.org/jira/browse/HADOOP-17451 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Intermittent failure of ITestHuge* upload tests, when doing parallel test > runs. > The count of bytes uploaded through StorageStatistics isn't updated. Maybe > the expected counter isn't updated, and somehow in a parallel run with > recycled FS instances/set up directory structure this surfaces the way it > doesn't in a single test run. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257105#comment-17257105 ] Steve Loughran commented on HADOOP-17451: - came with HADOOP-17271 and the S3A statistics rework. That is meant to be better with HADOOP-16133 fixed, but clearly there's been a regression. which surfaces with multipart uploads. proposed * fix the regression * large upload tests to include probes for IOStats and thread local upload stats. I think its time to cut out some of the ITestHuge test cases and just have some unit tests for the buffer mechanisms plus one live test. Separate patch tho' > intermittent failure of S3A huge file upload tests: count of bytes uploaded > == 0 > > > Key: HADOOP-17451 > URL: https://issues.apache.org/jira/browse/HADOOP-17451 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > Intermittent failure of ITestHuge* upload tests, when doing parallel test > runs. > The count of bytes uploaded through StorageStatistics isn't updated. Maybe > the expected counter isn't updated, and somehow in a parallel run with > recycled FS instances/set up directory structure this surfaces the way it > doesn't in a single test run. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0
Steve Loughran created HADOOP-17451: --- Summary: intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0 Key: HADOOP-17451 URL: https://issues.apache.org/jira/browse/HADOOP-17451 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Intermittent failure of ITestHuge* upload tests, when doing parallel test runs. The count of bytes uploaded through StorageStatistics isn't updated. Maybe the expected counter isn't updated, and somehow in a parallel run with recycled FS instances/set up directory structure this surfaces the way it doesn't in a single test run. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17271: Description: S3A to rework statistics with * API + Implementation split of the interfaces used by subcomponents when reporting stats * S3A Instrumentation to implement all the interfaces * streams, etc to all implement IOStatisticsSources and serve to callers * Add some tracking of durations of remote requests * RemoteIterator iterators returned from list* calls return list stats for callers to collect; toString will print them too was: S3A to rework statistics with * API + Implementation split of the interfaces used by subcomponents when reporting stats * S3A Instrumentation to implement all the interfaces * streams, etc to all implement IOStatisticsSources and serve to callers * Add some tracking of durations of remote requests > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 11.5h > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests > * RemoteIterator iterators returned from list* calls return list stats for > callers to collect; toString will print them too -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17271. - Fix Version/s: 3.4.0 Resolution: Fixed Merged in to trunk -will go into branch-3.3 once stabilized > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 11.5h > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529898&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529898 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 21:55 Start Date: 31/Dec/20 21:55 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2580: URL: https://github.com/apache/hadoop/pull/2580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529898) Time Spent: 11.5h (was: 11h 20m) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11.5h > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
steveloughran merged pull request #2580: URL: https://github.com/apache/hadoop/pull/2580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529897 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 21:54 Start Date: 31/Dec/20 21:54 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753214369 Again, this is yetus playing up; local builds and tests are fine. I am seeing an intermittent test failure on parallel runs of ITestHugeFile uploads, with a counter not incrementing, but this doesn't surface standalone. Will file and fix separately This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529897) Time Spent: 11h 20m (was: 11h 10m) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11h 20m > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753214369 Again, this is yetus playing up; local builds and tests are fine. I am seeing an intermittent test failure on parallel runs of ITestHugeFile uploads, with a counter not incrementing, but this doesn't surface standalone. Will file and fix separately This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
goiri commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r550685412
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,496 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+import static org.junit.Assert.assertTrue;
+
+public class DistributedShellBaseTest {
+
+ protected final static String APPMASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final int NUM_NMS = 1;
+ private static final float DEFAULT_TIMELINE_VERSION = 1.0f;
+ private static final String TIMELINE_AUX_SERVICE_NAME = "timeline_collector";
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private final String yarnClientTimeout =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private final String[] commonArgs = {
+ "--jar",
+ APPMASTER_JAR,
+ "--timeout",
+ yarnClientTimeout,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+
+ protected MiniYARNCluster yarnCluster;
+ protected YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+ protected float getTimelineVersion() {
+return DEFAULT_TIMELINE_VERSION;
+ }
+
+ public String getTimelineV2StorageDir() {
+return timelineV2StorageDir;
+ }
+
+ public void setTimelineV2Stor
[GitHub] [hadoop] hadoop-yetus commented on pull request #2581: YARN-10553. split TestDistributedShell
hadoop-yetus commented on pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#issuecomment-753090950
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 57s | | trunk passed |
| +1 :green_heart: | compile | 20m 9s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 15s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 57s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 55s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 52s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 19s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt)
| hadoop-yarn-applications-distributedshell in the patch failed. |
| -1 :x: | compile | 10m 11s |
[/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.
|
| -1 :x: | javac | 10m 11s |
[/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.
|
| -1 :x: | compile | 8m 56s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -1 :x: | javac | 8m 56s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 2m 20s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 6 new + 5 unchanged - 12 fixed = 11 total (was
17) |
| -1 :x: | mvnsite | 0m 26s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/1/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt)
| hadoop-yarn-applications-distributedshell in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 5s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | findbugs | 0m 25s |
[/patc
[jira] [Work started] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17271 started by Steve Loughran. --- > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11h 10m > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529874
]
ASF GitHub Bot logged work on HADOOP-17271:
---
Author: ASF GitHub Bot
Created on: 31/Dec/20 18:46
Start Date: 31/Dec/20 18:46
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753025232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 32m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt)
| The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 25s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 1m 16s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 2m 29s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 24s |
[/branch-findbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.
[jira] [Commented] (HADOOP-16947) Stale record should be remove when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257076#comment-17257076
]
Hadoop QA commented on HADOOP-16947:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
26s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
15s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 17s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
18s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
16s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/137/artifact/out/diff-checkstyle-root.txt{color}
| {color:orange} root: The patch generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/137/artifact/out/whitespace-eol.txt{color}
| {color
[GitHub] [hadoop] hadoop-yetus commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753025232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 32m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt)
| The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 25s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 1m 16s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 2m 29s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 24s |
[/branch-findbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/3/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529871
]
ASF GitHub Bot logged work on HADOOP-17271:
---
Author: ASF GitHub Bot
Created on: 31/Dec/20 18:25
Start Date: 31/Dec/20 18:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753021806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 15m 36s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt)
| The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 51s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 1m 47s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 26s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 3m 7s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 26s |
[/branch-findbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.
[GitHub] [hadoop] hadoop-yetus commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753021806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 15m 36s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt)
| The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 51s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 1m 47s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 26s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 3m 7s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 26s |
[/branch-findbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529867 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 17:22 Start Date: 31/Dec/20 17:22 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753010028 latest patches fix up the style/findbugs warnings. I've also replicated and fixed the test failure gabor reported. I am seeing a failure on huge uploads and putbyte counts, but I want to do that later just to get this one in by the end of 2020. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529867) Time Spent: 10h 50m (was: 10h 40m) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h 50m > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-753010028 latest patches fix up the style/findbugs warnings. I've also replicated and fixed the test failure gabor reported. I am seeing a failure on huge uploads and putbyte counts, but I want to do that later just to get this one in by the end of 2020. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on pull request #2567: HDFS-15740. Add x-platform utilities
GauthamBanasandra commented on pull request #2567: URL: https://github.com/apache/hadoop/pull/2567#issuecomment-753009651 @aajisaka could you please review my PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529859 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 17:01 Start Date: 31/Dec/20 17:01 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2553: URL: https://github.com/apache/hadoop/pull/2553#issuecomment-753006191 Merging in as #2577 #2579 #2580; closing this one to avoid confusion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529859) Time Spent: 10.5h (was: 10h 20m) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10.5h > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529860 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 17:01 Start Date: 31/Dec/20 17:01 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #2553: URL: https://github.com/apache/hadoop/pull/2553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529860) Time Spent: 10h 40m (was: 10.5h) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h 40m > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2553: HADOOP-17271. S3A to support IOStatistics
steveloughran closed pull request #2553: URL: https://github.com/apache/hadoop/pull/2553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
steveloughran commented on pull request #2553: URL: https://github.com/apache/hadoop/pull/2553#issuecomment-753006191 Merging in as #2577 #2579 #2580; closing this one to avoid confusion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein opened a new pull request #2581: YARN-10553. split TestDistributedShell
amahussein opened a new pull request #2581: URL: https://github.com/apache/hadoop/pull/2581 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2579: MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics.
steveloughran merged pull request #2579: URL: https://github.com/apache/hadoop/pull/2579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529836
]
ASF GitHub Bot logged work on HADOOP-17271:
---
Author: ASF GitHub Bot
Created on: 31/Dec/20 15:47
Start Date: 31/Dec/20 15:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-752991931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 11s | | trunk passed |
| +1 :green_heart: | compile | 20m 15s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 33s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 14s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 38s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 20m 40s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 40s | |
root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2040 unchanged - 21
fixed = 2040 total (was 2061) |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 19m 51s | |
root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1935 unchanged -
21 fixed = 1935 total (was 1956) |
| -0 :warning: | checkstyle | 2m 40s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 130 unchanged - 22 fixed = 135 total (was
152) |
| +1 :green_heart: | mvnsite | 1m 47s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 30s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javadoc | 0m 44s |
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 1 new +
88 unchanged - 0 fixed = 89 total (was 88) |
| -1 :x: | findbugs | 1m 25s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total
(was 0) |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 11s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 1m 43s | | hadoop-aws in the patch passed.
|
| +1 :gr
[GitHub] [hadoop] hadoop-yetus commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
hadoop-yetus commented on pull request #2580:
URL: https://github.com/apache/hadoop/pull/2580#issuecomment-752991931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 45 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 11s | | trunk passed |
| +1 :green_heart: | compile | 20m 15s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 33s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 14s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 38s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 20m 40s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 40s | |
root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2040 unchanged - 21
fixed = 2040 total (was 2061) |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 19m 51s | |
root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1935 unchanged -
21 fixed = 1935 total (was 1956) |
| -0 :warning: | checkstyle | 2m 40s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 5 new + 130 unchanged - 22 fixed = 135 total (was
152) |
| +1 :green_heart: | mvnsite | 1m 47s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 30s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javadoc | 0m 44s |
[/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 1 new +
88 unchanged - 0 fixed = 89 total (was 88) |
| -1 :x: | findbugs | 1m 25s |
[/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2580/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total
(was 0) |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 11s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 1m 43s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 205m 38s | | |
| Reason | Tests |
|---:|:--|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Potentially ambiguous invocation of either an outer or inherited
method
org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics()
in
org.apache.hadoop.fs.s3a.S3AIns
[jira] [Commented] (HADOOP-16947) Stale record should be remove when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257005#comment-17257005
]
Haibin Huang commented on HADOOP-16947:
---
Update the patch to fix checkstyle. Hi [~elgoiri] [~ayushtkn] [~hexiaoqiao]
[~weichiu], can you take a look at this. Now if a node has a problem at network
card, it will be reported as slow peer to namenode by other reporting node, and
we remove this slow peer node out of cluster, but the SlowPeersReport will not
remove this node, because the reporting nodes always update their SumAndCount
deque with old data. If we want to use SlowPeersReport to do a real-time
monitor, it will confuse us. So we need a timestamp to define when the old data
will be invalid.
org.apache.hadoop.metrics2.lib.MutableRollingAverages#rollOverAvgs
{code:java}
final SumAndCount sumAndCount = new SumAndCount(
rate.lastStat().total(),
rate.lastStat().numSamples());
{code}
> Stale record should be remove when generating SlowPeersReport
> -
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HADOOP-16947-005.patch,
> HADOOP-16947-006.patch, HADOOP-16947-007.patch, HDFS-14783,
> HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2579: MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics.
hadoop-yetus commented on pull request #2579:
URL: https://github.com/apache/hadoop/pull/2579#issuecomment-752959475
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 16s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 16s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 14s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 58s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 16s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 58s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 83m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2579 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8776cfff27cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 99d08a19ba5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/testReport/ |
| Max. process+thread count | 1614 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond
[jira] [Updated] (HADOOP-16947) Stale record should be remove when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HADOOP-16947:
--
Attachment: HADOOP-16947-007.patch
> Stale record should be remove when generating SlowPeersReport
> -
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HADOOP-16947-005.patch,
> HADOOP-16947-006.patch, HADOOP-16947-007.patch, HDFS-14783,
> HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529784&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529784 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 12:25 Start Date: 31/Dec/20 12:25 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-752944899 * This is the S3A part of #2577 (including the MR patch pending its merge) * already approved by gabor * rerunning the tests before the final merge Gabor has reported a test failure but I can't replicate it. I'm going to merge this in and then add a followup stabilisation patch for that and any other inevitable surprises -no CP to branch-3.3 until that is in This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529784) Time Spent: 10h 10m (was: 10h) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h 10m > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
steveloughran commented on pull request #2580: URL: https://github.com/apache/hadoop/pull/2580#issuecomment-752944899 * This is the S3A part of #2577 (including the MR patch pending its merge) * already approved by gabor * rerunning the tests before the final merge Gabor has reported a test failure but I can't replicate it. I'm going to merge this in and then add a followup stabilisation patch for that and any other inevitable surprises -no CP to branch-3.3 until that is in This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17271) S3A statistics to support IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=529783&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529783 ] ASF GitHub Bot logged work on HADOOP-17271: --- Author: ASF GitHub Bot Created on: 31/Dec/20 12:20 Start Date: 31/Dec/20 12:20 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #2580: URL: https://github.com/apache/hadoop/pull/2580 This is a major rework of the S3A Statistics collection to * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529783) Time Spent: 10h (was: 9h 50m) > S3A statistics to support IOStatistics > -- > > Key: HADOOP-17271 > URL: https://issues.apache.org/jira/browse/HADOOP-17271 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h > Remaining Estimate: 0h > > S3A to rework statistics with > * API + Implementation split of the interfaces used by subcomponents when > reporting stats > * S3A Instrumentation to implement all the interfaces > * streams, etc to all implement IOStatisticsSources and serve to callers > * Add some tracking of durations of remote requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #2580: HADOOP-17271. S3A connector to support IOStatistics.
steveloughran opened a new pull request #2580: URL: https://github.com/apache/hadoop/pull/2580 This is a major rework of the S3A Statistics collection to * Move from direct references of S3AInstrumention statistics collectors to interface/implementation classes in new packages. * Ubiquitous support of IOStatistics, including: S3AFileSystem, input and output streams, RemoteIterator instances provided in list calls. * Adoption of new statistic names from hadoop-common Regarding statistic collection, as well as all existing statistics, the connector now records min/max/mean durations of HTTP GET and HEAD requests, and those of LIST operations. Contributed by Steve Loughran. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #2579: MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics.
steveloughran opened a new pull request #2579: URL: https://github.com/apache/hadoop/pull/2579 Part of HADOOP-16830 IOStatistics Support. Contributed by Steve Loughran. # Tests are in hadoop-aws so excluded here # full PR already reviewed by @bgaborg in #2553; this is the feature split up by module for better isolation/backport This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17450) hadoop-common to add IOStatistics API
[ https://issues.apache.org/jira/browse/HADOOP-17450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17450. - Fix Version/s: 3.4.0 Resolution: Fixed merged to trunk will backport once the ready-to-merge followup prs are in and all looks good > hadoop-common to add IOStatistics API > - > > Key: HADOOP-17450 > URL: https://issues.apache.org/jira/browse/HADOOP-17450 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Add the iostatistics API to hadoop-common with > * public interfaces for querying statistics > * serializable snapshot > * logging support > and implementation support for filesystems -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17450) hadoop-common to add IOStatistics API
[ https://issues.apache.org/jira/browse/HADOOP-17450?focusedWorklogId=529774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529774 ] ASF GitHub Bot logged work on HADOOP-17450: --- Author: ASF GitHub Bot Created on: 31/Dec/20 11:52 Start Date: 31/Dec/20 11:52 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2577: URL: https://github.com/apache/hadoop/pull/2577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529774) Time Spent: 50m (was: 40m) > hadoop-common to add IOStatistics API > - > > Key: HADOOP-17450 > URL: https://issues.apache.org/jira/browse/HADOOP-17450 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Add the iostatistics API to hadoop-common with > * public interfaces for querying statistics > * serializable snapshot > * logging support > and implementation support for filesystems -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2577: HADOOP-17450. Add Public IOStatistics API.
steveloughran merged pull request #2577: URL: https://github.com/apache/hadoop/pull/2577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17450) hadoop-common to add IOStatistics API
[ https://issues.apache.org/jira/browse/HADOOP-17450?focusedWorklogId=529771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529771 ] ASF GitHub Bot logged work on HADOOP-17450: --- Author: ASF GitHub Bot Created on: 31/Dec/20 11:36 Start Date: 31/Dec/20 11:36 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2577: URL: https://github.com/apache/hadoop/pull/2577#issuecomment-752932940 This is an out of memory error. as I only changed javadocs since the last successful build, I'm going to do the build and javadoc audit locally, and if good: finish off that merge *into trunk only*. Then I'll followup with the later changes, again, trunk, until I'm happy to do the CP into branch-3.3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529771) Time Spent: 40m (was: 0.5h) > hadoop-common to add IOStatistics API > - > > Key: HADOOP-17450 > URL: https://issues.apache.org/jira/browse/HADOOP-17450 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Add the iostatistics API to hadoop-common with > * public interfaces for querying statistics > * serializable snapshot > * logging support > and implementation support for filesystems -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2577: HADOOP-17450. Add Public IOStatistics API.
steveloughran commented on pull request #2577: URL: https://github.com/apache/hadoop/pull/2577#issuecomment-752932940 This is an out of memory error. as I only changed javadocs since the last successful build, I'm going to do the build and javadoc audit locally, and if good: finish off that merge *into trunk only*. Then I'll followup with the later changes, again, trunk, until I'm happy to do the CP into branch-3.3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
