[jira] [Commented] (YARN-7009) TestNMClient.testNMClientNoCleanupOnStop is flaky by design
[
https://issues.apache.org/jira/browse/YARN-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130021#comment-16130021
]
Hadoop QA commented on YARN-7009:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 136 unchanged - 17 fixed = 139 total (was 153) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 39s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
26s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Dead store to sum in
org.apache.hadoop.yarn.state.DebugStateMachineDecoratorImpl.doTransition(Enum,
Object) At
DebugStateMachineDecoratorImpl.java:org.apache.hadoop.yarn.state.DebugStateMachineDecoratorImpl.doTransition(Enum,
Object) At DebugStateMachineDecoratorImpl.java:[line 56] |
| Timed out junit tests |
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7009 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882267/YARN-7009.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checksty
[jira] [Assigned] (YARN-7034) DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor
[ https://issues.apache.org/jira/browse/YARN-7034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-7034: Assignee: Miklos Szegedi > DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client > environment variables to container-executor > - > > Key: YARN-7034 > URL: https://issues.apache.org/jira/browse/YARN-7034 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Critical > Attachments: YARN-7034.000.patch > > > This behavior is unnecessary since there is nothing that is used from the > environment right now. One option is to whitelist these variables before > passing them. Are there any known use cases for this to justify? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7034) DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor
[ https://issues.apache.org/jira/browse/YARN-7034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-7034: - Attachment: YARN-7034.000.patch I created a minimalist patch to start the discussion. Are there any use cases for passing a whitelisted list? > DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client > environment variables to container-executor > - > > Key: YARN-7034 > URL: https://issues.apache.org/jira/browse/YARN-7034 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Priority: Critical > Attachments: YARN-7034.000.patch > > > This behavior is unnecessary since there is nothing that is used from the > environment right now. One option is to whitelist these variables before > passing them. Are there any known use cases for this to justify? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7032) [ATSv2] NPE while starting hbase co-processor
[
https://issues.apache.org/jira/browse/YARN-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129996#comment-16129996
]
Vrushali C commented on YARN-7032:
--
Hmm, interesting. What version of HBase are you testing with?
> [ATSv2] NPE while starting hbase co-processor
> -
>
> Key: YARN-7032
> URL: https://issues.apache.org/jira/browse/YARN-7032
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>
> It is seen randomly that hbase co-processor fails to start with NPE. But
> again starting RegionServer, able to succeed in starting RS.
> {noformat}
> 2017-08-17 05:53:13,535 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> threw java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.Tag.fromList(Tag.java:187)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor.prePut(FlowRunCoprocessor.java:102)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:885)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7034) DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor
Miklos Szegedi created YARN-7034: Summary: DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor Key: YARN-7034 URL: https://issues.apache.org/jira/browse/YARN-7034 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Miklos Szegedi Priority: Critical This behavior is unnecessary since there is nothing that is used from the environment right now. One option is to whitelist these variables before passing them. Are there any known use cases for this to justify? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
[ https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129994#comment-16129994 ] Devaraj K commented on YARN-5764: - I've created YARN-7033 to move the common logic from YARN-6620 to handle the recovery of assigned resources. > NUMA awareness support for launching containers > --- > > Key: YARN-5764 > URL: https://issues.apache.org/jira/browse/YARN-5764 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, yarn >Reporter: Olasoji >Assignee: Devaraj K > Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance > Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch, > YARN-5764-v3.patch > > > The purpose of this feature is to improve Hadoop performance by minimizing > costly remote memory accesses on non SMP systems. Yarn containers, on launch, > will be pinned to a specific NUMA node and all subsequent memory allocations > will be served by the same node, reducing remote memory accesses. The current > default behavior is to spread memory across all NUMA nodes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7033) Add support for NM Recovery of assigned resources(GPU's, NUMA, FPGA's) to container
Devaraj K created YARN-7033: --- Summary: Add support for NM Recovery of assigned resources(GPU's, NUMA, FPGA's) to container Key: YARN-7033 URL: https://issues.apache.org/jira/browse/YARN-7033 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Devaraj K Assignee: Devaraj K This JIRA adds the common logic to store the assigned resources to container such as GPU's(YARN-6620), NUMA(YARN-5764) and FPGA's(YARN-5983) etc. and recover upon restart of NM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-7028:
-
Attachment: YARN-7028-YARN-5355.0001.patch
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
> Attachments: YARN-7028-YARN-5355.0001.patch
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-7028:
-
Attachment: (was: YARN-6323-YARN-5355.0001.patch)
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7032) [ATSv2] NPE while starting hbase co-processor
[
https://issues.apache.org/jira/browse/YARN-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129976#comment-16129976
]
Rohith Sharma K S commented on YARN-7032:
-
Full stack trace is
{noformat}
2017-08-17 05:53:13,535 ERROR
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
coprocessor.CoprocessorHost: The coprocessor
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
threw java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.hbase.Tag.fromList(Tag.java:187)
at
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor.prePut(FlowRunCoprocessor.java:102)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:885)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:3036)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3011)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2957)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:750)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:710)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2137)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32393)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
498516-98899-01-06.hwx.site,16020,1502949162490: The coprocessor
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
threw java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.hbase.Tag.fromList(Tag.java:187)
at
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor.prePut(FlowRunCoprocessor.java:102)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:885)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:3036)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3011)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2957)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:750)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:710)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2137)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32393)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
2017-08-17 05:53:13,536 FATAL
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
regionserver.HRegionServer: RegionServer abort: loaded coprocessors are:
[org.apache.hadoop.hbase.security.access.AccessController,
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor,
org.apache.hadoop.hbase.security.token.TokenProvider,
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
{noformat}
> [ATSv2] NPE while starting hbase co-processor
> -
>
>
[jira] [Created] (YARN-7032) [ATSv2] NPE while starting hbase co-processor
Rohith Sharma K S created YARN-7032:
---
Summary: [ATSv2] NPE while starting hbase co-processor
Key: YARN-7032
URL: https://issues.apache.org/jira/browse/YARN-7032
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
It is seen randomly that hbase co-processor fails to start with NPE. But again
starting RegionServer, able to succeed in starting RS.
{noformat}
2017-08-17 05:53:13,535 ERROR
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
coprocessor.CoprocessorHost: The coprocessor
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
threw java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.hbase.Tag.fromList(Tag.java:187)
at
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor.prePut(FlowRunCoprocessor.java:102)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:885)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7031) Support distributed node attributes
[ https://issues.apache.org/jira/browse/YARN-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-7031: --- Component/s: (was: api) (was: client) (was: capacityscheduler) > Support distributed node attributes > --- > > Key: YARN-7031 > URL: https://issues.apache.org/jira/browse/YARN-7031 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Allow nodemanagers to push its attributes to RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129968#comment-16129968
]
Hadoop QA commented on YARN-7028:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
16s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-7028 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882264/YARN-6323-YARN-5355.0001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7a72681e1127 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 315ff9b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16952/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16952/testReport/ |
| modules | C: hadoop-yarn-project/ha
[jira] [Updated] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3409: Component/s: (was: capacityscheduler) RM > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6257) CapacityScheduler REST API produces incorrect JSON - JSON object operationsInfo contains deplicate key
[
https://issues.apache.org/jira/browse/YARN-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129955#comment-16129955
]
Tao Yang commented on YARN-6257:
Thanks [~sunilg] and [~leftnoteasy]. it makes sense to me. I will update the
document and upload a new patch for review.
> CapacityScheduler REST API produces incorrect JSON - JSON object
> operationsInfo contains deplicate key
> --
>
> Key: YARN-6257
> URL: https://issues.apache.org/jira/browse/YARN-6257
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Minor
> Attachments: YARN-6257.001.patch
>
>
> In response string of CapacityScheduler REST API,
> scheduler/schedulerInfo/health/operationsInfo have duplicate key 'entry' as a
> JSON object :
> {code}
> "operationsInfo":{
>
> "entry":{"key":"last-preemption","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}},
>
> "entry":{"key":"last-reservation","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}},
>
> "entry":{"key":"last-allocation","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}},
>
> "entry":{"key":"last-release","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}}
> }
> {code}
> To solve this problem, I suppose the type of operationsInfo field in
> CapacitySchedulerHealthInfo class should be converted from Map to List.
> After convert to List, The operationsInfo string will be:
> {code}
> "operationInfos":[
>
> {"operation":"last-allocation","nodeId":"N/A","containerId":"N/A","queue":"N/A"},
>
> {"operation":"last-release","nodeId":"N/A","containerId":"N/A","queue":"N/A"},
>
> {"operation":"last-preemption","nodeId":"N/A","containerId":"N/A","queue":"N/A"},
>
> {"operation":"last-reservation","nodeId":"N/A","containerId":"N/A","queue":"N/A"}
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7031) Support distributed node attributes
Bibin A Chundatt created YARN-7031: -- Summary: Support distributed node attributes Key: YARN-7031 URL: https://issues.apache.org/jira/browse/YARN-7031 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Allow nodemanagers to push its attributes to RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7009) TestNMClient.testNMClientNoCleanupOnStop is flaky by design
[
https://issues.apache.org/jira/browse/YARN-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129937#comment-16129937
]
Miklos Szegedi commented on YARN-7009:
--
The exception above was caused by the fact that we are always in the RUNNING
state, so we just jump to reinitialize after restart. I have also seen another
instance, when the container finished on a busy machine early waiting
indefinitely. I solved this by increasing the sleep time to an arbitrarily
large number. We stop the container anyways, plus we have the timeout.
There is another issue with the pid file. If it is not created within 2
seconds, this code may skip the reinitialize keeping the container in the
REINITIALIZE state forever.
processId may be null:
{code}
// get process id from pid file if available
// else if shell is still active, get it from the shell
String processId = null;
if (pidFilePath != null) {
processId = getContainerPid(pidFilePath);
}
// kill process
if (processId != null) {
{code}
Fixing this however needs significant refactoring, so another Jira I think. The
only thing I could do in this jira is to increase the
DEFAULT_NM_PROCESS_KILL_WAIT_MS timeout to 1. What do you think?
> TestNMClient.testNMClientNoCleanupOnStop is flaky by design
> ---
>
> Key: YARN-7009
> URL: https://issues.apache.org/jira/browse/YARN-7009
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7009.000.patch
>
>
> The sleeps to wait for a transition to reinit and than back to running is not
> long enough, it can miss the reinit event.
> {code}
> java.lang.AssertionError: Exception is not expected:
> org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform RE_INIT on
> [container_1502735389852_0001_01_01]. Current state is [REINITIALIZING,
> isReInitializing=true].
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitializeOrLocalizeCheck(ContainerManagerImpl.java:1772)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1697)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1668)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.reInitializeContainer(ContainerManagementProtocolPBServiceImpl.java:214)
> at
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:237)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testReInitializeContainer(TestNMClient.java:567)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testContainerManagement(TestNMClient.java:405)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClientNoCleanupOnStop(TestNMClient.java:214)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform
> RE_INIT on [container_1502735389852_0001_01_01]. Current state is
> [REINITIALIZING, isReInitializing=true].
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitiali
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129929#comment-16129929
]
Naganarasimha G R commented on YARN-6240:
-
Thanks [~bibinchundatt] for pointing it out ! Had forgotten about it will
cherry pick that patch !
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch,
> YARN-6240-branch-2.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7009) TestNMClient.testNMClientNoCleanupOnStop is flaky by design
[
https://issues.apache.org/jira/browse/YARN-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-7009:
-
Attachment: YARN-7009.000.patch
> TestNMClient.testNMClientNoCleanupOnStop is flaky by design
> ---
>
> Key: YARN-7009
> URL: https://issues.apache.org/jira/browse/YARN-7009
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7009.000.patch
>
>
> The sleeps to wait for a transition to reinit and than back to running is not
> long enough, it can miss the reinit event.
> {code}
> java.lang.AssertionError: Exception is not expected:
> org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform RE_INIT on
> [container_1502735389852_0001_01_01]. Current state is [REINITIALIZING,
> isReInitializing=true].
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitializeOrLocalizeCheck(ContainerManagerImpl.java:1772)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1697)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1668)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.reInitializeContainer(ContainerManagementProtocolPBServiceImpl.java:214)
> at
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:237)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testReInitializeContainer(TestNMClient.java:567)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testContainerManagement(TestNMClient.java:405)
> at
> org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClientNoCleanupOnStop(TestNMClient.java:214)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform
> RE_INIT on [container_1502735389852_0001_01_01]. Current state is
> [REINITIALIZING, isReInitializing=true].
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitializeOrLocalizeCheck(ContainerManagerImpl.java:1772)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1697)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1668)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.reInitializeContainer(ContainerManagementProtocolPBServiceImpl.java:214)
> at
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:237)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.d
[jira] [Comment Edited] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129912#comment-16129912 ] Junping Du edited comment on YARN-7011 at 8/17/17 4:56 AM: --- I am not sure if everyone would agree that it is by design at least I am not convinced yet. I tried to understand why we give up this old behavior and what trade-off we get in here as this significantly change the old behavior that work well in production. We shouldn't assume that this is a new major branch/release so we just break everything for free (or just marking as incompatible, and put a line of document). We should also ask why we break this for and also challenge if it deserves to break it. Otherwise, deploying/upgrading Hadoop 3 will be a nightmare for many of Hadoop users. Let's keep it open for discussion before we make an agreement. I also want to hear what others want to say. was (Author: djp): I am not sure if everyone would agree that it is by design at least I am not. I would like to hear what others to say about this - this significantly change the old behavior that work well in production. Let's keep it open for discussion before we make any agreement. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[
https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt reassigned YARN-7007:
--
Assignee: Lingfeng Su
> NPE in RM while using YarnClientImpl.getApplications() ot get all applications
> --
>
> Key: YARN-7007
> URL: https://issues.apache.org/jira/browse/YARN-7007
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Lingfeng Su
>Assignee: Lingfeng Su
> Labels: patch
> Attachments: YARN-7007.001.patch
>
>
> {code:java}
> java.lang.NullPointerException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254)
> at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy161.getApplications(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456)
> {code}
> When I use YarnClient.getApplications() to get all applications of RM,
> Occasionally, it throw a NPE problem.
> {code:java}
> RMAppAttempt currentAttempt = rmContext.getRMApps()
>.get(attemptId.getApplicationId()).getCurrentAppAttempt();
> {code}
> if the application id is not in ConcurrentMap
> getRMApps(), it may throw NPE problem.
> Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this
> problem ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129912#comment-16129912 ] Junping Du commented on YARN-7011: -- I am not sure if everyone would agree that it is by design at least I am not. I would like to hear what others to say about this - this significantly change the old behavior that work well in production. Let's keep it open for discussion before we make any agreement. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129908#comment-16129908
]
Hadoop QA commented on YARN-6240:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-6240 does not apply to branch-2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6240 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882263/YARN-6240-branch-2.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16951/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch,
> YARN-6240-branch-2.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-7028:
-
Attachment: YARN-6323-YARN-5355.0001.patch
Uploading patch 0001. This is a combined patch from HADOOP-14386 and
HADOOP-14382.
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
> Attachments: YARN-6323-YARN-5355.0001.patch
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reopened YARN-7011: -- > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-7011: - Priority: Blocker (was: Trivial) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C reassigned YARN-7028:
Assignee: Vrushali C
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129900#comment-16129900
]
Bibin A Chundatt commented on YARN-6240:
[~naganarasimha...@apache.org]
Seems YARN-6327 is not available in branch-2 which is already available in
trunk.
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch,
> YARN-6240-branch-2.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-6240:
Attachment: YARN-6240-branch-2.002.patch
attached patch for branch-2
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch,
> YARN-6240-branch-2.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129899#comment-16129899
]
Vrushali C commented on YARN-7028:
--
Ah, thanks [~haibo.chen] . Our last rebase to trunk was around April end and so
we picked up the upgrade to guava 21 but not the roll back. Looks like we would
also need HADOOP-14382 rebased onto YARN-5355 branch.
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7023) Incorrect ReservationId.compareTo() implementation
[
https://issues.apache.org/jira/browse/YARN-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129898#comment-16129898
]
Akira Ajisaka commented on YARN-7023:
-
I like {{Long.compare(getId(), other.getId())}} because it is more simple.
Would you replace it?
> Incorrect ReservationId.compareTo() implementation
> --
>
> Key: YARN-7023
> URL: https://issues.apache.org/jira/browse/YARN-7023
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: YARN-7023.patch
>
>
> Obviously this is a typo and the second getId() call should be rewritten as
> 'other.getId()'
> +Lines 94-102+
> {code:java}
> @Override
> public int compareTo(ReservationId other) {
> if (this.getClusterTimestamp() - other.getClusterTimestamp() == 0) {
> return getId() > getId() ? 1 : getId() < getId() ? -1 : 0;
> } else {
> return this.getClusterTimestamp() > other.getClusterTimestamp() ? 1
> : this.getClusterTimestamp() < other.getClusterTimestamp() ? -1 : 0;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129857#comment-16129857
]
Hadoop QA commented on YARN-6240:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6240 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6240 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882260/YARN-6240.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16950/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-6240:
Attachment: YARN-6240.002.patch
Simple issue , my bad did not identify it!
[~bibinchundatt]/ [~sunilg], Can one of you look at the attached patch &
review?
Existing test case is already covering the scenario hence have not attached any
test case
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch, YARN-6240.002.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6839) [YARN-3368] Be able to view application / container logs on the new YARN UI.
[ https://issues.apache.org/jira/browse/YARN-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Ding reassigned YARN-6839: - Assignee: (was: Da Ding) > [YARN-3368] Be able to view application / container logs on the new YARN UI. > > > Key: YARN-6839 > URL: https://issues.apache.org/jira/browse/YARN-6839 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > Currently viewing application/container logs will be redirected to the old > UI, we should leverage the new UI capabilities to make a better log-viewing > experiences. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7030) Additional performance improvements of YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7030: - Attachment: YARN-7030.YARN-3926.001.patch Attached patch, with this patch, I see about 3-4% performance improvements. It looks like getResourceTypesArray call is actually expensive than we expected, the latest patch uses a volatile int to store array length instead of getting array every time. This patch is on top of YARN-6610. > Additional performance improvements of YARN-3926 branch > --- > > Key: YARN-7030 > URL: https://issues.apache.org/jira/browse/YARN-7030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7030.YARN-3926.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6839) [YARN-3368] Be able to view application / container logs on the new YARN UI.
[ https://issues.apache.org/jira/browse/YARN-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Ding reassigned YARN-6839: - Assignee: Da Ding > [YARN-3368] Be able to view application / container logs on the new YARN UI. > > > Key: YARN-6839 > URL: https://issues.apache.org/jira/browse/YARN-6839 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Da Ding > > Currently viewing application/container logs will be redirected to the old > UI, we should leverage the new UI capabilities to make a better log-viewing > experiences. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129835#comment-16129835
]
Hadoop QA commented on YARN-6610:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
5 unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4510 unchanged - 5 fixed = 4510 total (was 4515) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882255/YARN-6610.YARN-3926.perf-wangda-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6faec6dcf63f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / f25bc50 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16949/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16949/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16949/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YAR
[jira] [Created] (YARN-7030) Additional performance improvements of YARN-3926 branch
Wangda Tan created YARN-7030: Summary: Additional performance improvements of YARN-3926 branch Key: YARN-7030 URL: https://issues.apache.org/jira/browse/YARN-7030 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7025) Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129828#comment-16129828 ] Miklos Szegedi commented on YARN-7025: -- [~ebadger], YARN-6757 updated the use of NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, so that it can be hardcoded now to /sys/fs/cgroup. Please consider using it directly, or probably better through CGroupsHandlerImp. It is better to configure cgroups only at a single location for all users. > Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime > --- > > Key: YARN-7025 > URL: https://issues.apache.org/jira/browse/YARN-7025 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Badger >Assignee: Eric Badger > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6610:
-
Attachment: YARN-6610.YARN-3926.perf-wangda-002.patch
Updated patch (wangda-002)
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.YARN-3926.perf-rebased.patch,
> YARN-6610.YARN-3926.perf-wangda-001.patch,
> YARN-6610.YARN-3926.perf-wangda-002.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6789) new api to get all supported resources from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6789: - Attachment: YARN-6789-YARN-3926.003.patch Attached ver.3 patch, [~sunilg] plz review. > new api to get all supported resources from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-6789-YARN-3926.001.patch, > YARN-6789-YARN-3926.002_incomplete_.patch, YARN-6789-YARN-3926.003.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129796#comment-16129796 ] Allen Wittenauer commented on YARN-7011: I have, however, opened HADOOP-14781 to signify that we should clarify the documentation around this. (Because setting HADOOP_CONF_DIR in hadoop-env.sh isn't productive in any version of Hadoop.) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3661) Basic Federation UI
[
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129783#comment-16129783
]
Hadoop QA commented on YARN-3661:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 44 new + 0 unchanged - 0 fixed = 44 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router
generated 12 new + 0 unchanged - 0 fixed = 12 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
|
| | Possible null pointer dereference of rmApps in
org.apache.hadoop.yarn.server.router.webapp.AppsBlock.initAppsInfo()
Dereferenced at AppsBlock.java:rmApps in
org.apache.hadoop.yarn.server.router.webapp.AppsBlock.initAppsInfo()
Dereferenced at AppsBlock.java:[line 169] |
| | Possible null pointer dereference of rmNodes in
org.apache.hadoop.yarn.server.router.webapp.NodesBlock.initNodesInfo()
Dereferenced at NodesBlock.java:rmNodes in
org.apache.hadoop.yarn.server.router.webapp.NodesBlock.initNodesInfo()
Dereferenced at NodesBlock.java:[line 132] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.yarn.server.router.webapp.ResourceManagersBlock.render(HtmlBlock$Block)
At ResourceManagersBlock.java:is not thrown in
org.apache.hadoop.yarn.server.router.webapp.ResourceManagersBlock.render(HtmlBlock$Block)
At ResourceManagersBlock.java:[line 90] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-3661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/atta
[jira] [Resolved] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved YARN-7011. Resolution: Won't Fix Closing as not a bug. Working as designed. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-7011: --- Priority: Trivial (was: Blocker) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129777#comment-16129777 ] Allen Wittenauer commented on YARN-7011: It's not a regression since the patch that changed the behavior was specifically marked as an incompatible change. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-7011: - Target Version/s: 3.0.0-beta1 > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129757#comment-16129757 ] Junping Du commented on YARN-7011: -- bq. There's the problem then. Remove it. Setting HADOOP_CONF_DIR in *-env.sh effectively resets the config directory back to wherever it is set. It acts as an override to the --config option in 3.x. Why setting HADOOP_CONF_DIR in env is supposed to override --config specified in yarn or hadoop CLI? This will be a regression since 2.x as existing Hadoop user (and cluster management tools) will get affected. Set back to blocker for 3.0 beta. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-7011: - Priority: Blocker (was: Trivial) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-3661: -- Attachment: YARN-3661-000.patch First proposal for the patch. > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129744#comment-16129744
]
Yuqi Wang commented on YARN-6959:
-
[~jianhe]
Seems the UT failures are not caused by my patch, please check.
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.8.0, 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.005.patch, YARN-6959-branch-2.7.005.patch,
> YARN-6959-branch-2.7.006.patch, YARN-6959-branch-2.8.001.patch,
> YARN-6959.yarn_nm.log.zip, YARN-6959.yarn_rm.log.zip
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7029) Add more UTs in yarn-native-services
[
https://issues.apache.org/jira/browse/YARN-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129719#comment-16129719
]
Hadoop QA commented on YARN-7029:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
55s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
in yarn-native-services has 7 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
The patch generated 48 new + 117 unchanged - 7 fixed = 165 total (was 124)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 8s{color}
| {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.yarn.service.TestYarnNativeServices
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7029 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882238/YARN-7029.yarn-native-services.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 201d8fd5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 164c0c4 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16947/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core-warnings.html
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/16947/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN
[jira] [Resolved] (YARN-6796) Add unit test for NM to launch OPPORTUNISTIC container for overallocation
[ https://issues.apache.org/jira/browse/YARN-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved YARN-6796. -- Resolution: Duplicate > Add unit test for NM to launch OPPORTUNISTIC container for overallocation > - > > Key: YARN-6796 > URL: https://issues.apache.org/jira/browse/YARN-6796 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.
[
https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129676#comment-16129676
]
Hadoop QA commented on YARN-6323:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
9s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
49s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
2s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6323 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882225/YARN-6323-YARN-5355.0001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 65410ea93454 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 315ff9b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16945/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16945/testReport/ |
| modules | C: hadoop-yarn-project/h
[jira] [Resolved] (YARN-6843) Add unit test for NM to preempt OPPORTUNISTIC containers under high utilization
[ https://issues.apache.org/jira/browse/YARN-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved YARN-6843. -- Resolution: Duplicate Unit tests will be added as part of YARN-6672 > Add unit test for NM to preempt OPPORTUNISTIC containers under high > utilization > --- > > Key: YARN-6843 > URL: https://issues.apache.org/jira/browse/YARN-6843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6796) Add unit test for NM to launch OPPORTUNISTIC container for overallocation
[ https://issues.apache.org/jira/browse/YARN-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129675#comment-16129675 ] Haibo Chen commented on YARN-6796: -- This will be added as part of YARN-6675 > Add unit test for NM to launch OPPORTUNISTIC container for overallocation > - > > Key: YARN-6796 > URL: https://issues.apache.org/jira/browse/YARN-6796 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6876) Create an abstract log writer for extendability
[
https://issues.apache.org/jira/browse/YARN-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129668#comment-16129668
]
Hadoop QA commented on YARN-6876:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
11s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 85
unchanged - 7 fixed = 85 total (was 92) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 21 new + 517 unchanged - 10 fixed = 538 total (was 527) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 19s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 14s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 21s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.webapp.TestWebApp |
| | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| Timed out junit tests |
org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6876 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882209/YARN-6876-t
[jira] [Updated] (YARN-7029) Add more UTs in yarn-native-services
[ https://issues.apache.org/jira/browse/YARN-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7029: -- Attachment: YARN-7029.yarn-native-services.01.patch > Add more UTs in yarn-native-services > > > Key: YARN-7029 > URL: https://issues.apache.org/jira/browse/YARN-7029 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7029.yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-7011: --- Priority: Trivial (was: Blocker) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-7011: --- Fix Version/s: (was: 3.0.0-beta1) > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Trivial > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6789) new api to get all supported resources from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6789: - Attachment: YARN-6789-YARN-3926.002_incomplete_.patch Attached ver.2 patch, not completed yet, it needs more logic to properly handle initialize from proto in releases which doesn't include YARN-3926. > new api to get all supported resources from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-6789-YARN-3926.001.patch, > YARN-6789-YARN-3926.002_incomplete_.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7029) Add more UTs in yarn-native-services
Jian He created YARN-7029: - Summary: Add more UTs in yarn-native-services Key: YARN-7029 URL: https://issues.apache.org/jira/browse/YARN-7029 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129636#comment-16129636
]
Haibo Chen commented on YARN-7028:
--
[~vrushalic], HADOOP-14386 has rolled back guava version from 21 to 11.0, and
it is missing from YARN-5355. I think backport this into YARN-5355 will fix it.
FYI, HADOOP-14284 (shade guava everywhere) is a blocker for guava update, we
should be fine provided the shading includes server modules as well.
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[
https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129635#comment-16129635
]
Hadoop QA commented on YARN-6134:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 45s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| | hadoop.yarn.server.TestContainerManagerSecurity |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882215/YARN-6134-YARN-5355.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e7c357682dd4 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129634#comment-16129634 ] Allen Wittenauer commented on YARN-7011: It's effectively a bug in 2.x and lower. Let's break this down to it's components: etc/hadoop/x has a working config that includes HADOOP_CONF_DIR definition that points to etc/hadoop user copies etc/hadoop to tmp/config and changes something else. HADOOP_CONF_DIR remains etc/hadoop. That's a problem because HADOOP_CONF_DIR isn't actually etc/hadoop anymore This is especially problematic for 3rd parties. They might read hadoop-env.sh directly and not have the logic in 2.x's hadoop-config.sh to know that the HADOOP_CONF_DIR value is a lie. In 3.x, users either need to a) make sure that they aren't lying to the system by keeping HADOOP_CONF_DIR truthful or b) not set it at all in the env files. > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > Fix For: 3.0.0-beta1 > > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129630#comment-16129630
]
Wangda Tan commented on YARN-6610:
--
Oops, included some debug code, will update patch shortly.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.YARN-3926.perf-rebased.patch,
> YARN-6610.YARN-3926.perf-wangda-001.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129625#comment-16129625
]
Vrushali C commented on YARN-7028:
--
I am thinking the timeline reader at
hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd class path should be set to
include the hbase class path so that guava11 would come in first.
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129622#comment-16129622 ] Sumana Sathish commented on YARN-7011: -- bq. There's the problem then. Remove it. Setting HADOOP_CONF_DIR in *-env.sh effectively resets the config directory back to wherever it is set. It acts as an override to the --config option in 3.x. In 2.x, --config takes precedence over the environment variables and -env.sh. Why did that have to change in 3.x? > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > Fix For: 3.0.0-beta1 > > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129621#comment-16129621
]
Hadoop QA commented on YARN-6610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
54s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
45s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 208 unchanged - 0 fixed = 209 total (was 208) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4510 unchanged - 5 fixed = 4510 total (was 4515) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| |
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator.totalInvoked
isn't final but should be At DominantResourceCalculator.java:be At
DominantResourceCalculator.java:[line 59] |
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882204/YARN-6610.YARN-3926.per
[jira] [Updated] (YARN-7028) Timeline reader guava issue
[
https://issues.apache.org/jira/browse/YARN-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-7028:
-
Labels: yarn-5355-merge-blocker (was: )
> Timeline reader guava issue
> ---
>
> Key: YARN-7028
> URL: https://issues.apache.org/jira/browse/YARN-7028
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
>
> [~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
> He ran into this issue with timeline reader while testing from YARN-5355 .
> The timeline reader from YARN-5355_branch2 did not have this.
> The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
> hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
> part of HADOOP-10101. The guava version to compile hadoop with was made
> configurable in HADOOP-14380
> The HBase guava client side fix was made in HBASE-14963. Will be good to have
> a new stable hbase release for this fix.
> {code}
> 2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
> reader.TimelineReaderWebServices: Error while processing REST request
> org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
> tried to access method com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> Caused by: java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 66 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[
https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129619#comment-16129619
]
Hadoop QA commented on YARN-6134:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
18s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 3s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882215/YARN-6134-YARN-5355.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b408b0b13887 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 /
[jira] [Created] (YARN-7028) Timeline reader guava issue
Vrushali C created YARN-7028:
Summary: Timeline reader guava issue
Key: YARN-7028
URL: https://issues.apache.org/jira/browse/YARN-7028
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vrushali C
Fix For: YARN-5355
[~agresch] pinged me about a guava incompatibility issue today. cc [~jlowe]
He ran into this issue with timeline reader while testing from YARN-5355 . The
timeline reader from YARN-5355_branch2 did not have this.
The guava version in YARN_5355_branch2 is guava 11. Guava 11 is also the
hbase-compatible-guava-version. Hadoop trunk (or YARN-5355) added guava 21 as
part of HADOOP-10101. The guava version to compile hadoop with was made
configurable in HADOOP-14380
The HBase guava client side fix was made in HBASE-14963. Will be good to have a
new stable hbase release for this fix.
{code}
2017-08-16 15:32:09,488 ERROR [qtp743778731-22]
reader.TimelineReaderWebServices: Error while processing REST request
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
tried to access method com.google.common.base.Stopwatch.()V from class
org.apache.hadoop.hbase.zookeeper.MetaTableLocator
Caused by: java.lang.IllegalAccessError: tried to access method
com.google.common.base.Stopwatch.()V from class
org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1357)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
... 66 more
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7013: - Attachment: YARN-7013.001.patch Attached ver.001 patch to run Jenkins. > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7013.001.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6361) FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues
[
https://issues.apache.org/jira/browse/YARN-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129592#comment-16129592
]
Yufei Gu commented on YARN-6361:
Hi [~daemon], thanks for the patch. Let's talk about the idea first.
{quote}
Avoid frequent ordering, and it's pointless and a waste of time to do a
sequence before each assign container. Because, after each assignment, the
whole child nodes of the queue are basically staying in order.
{quote}
Agreed basically, but it's the right thing to do until we find a solution to
either avoid sorting every time and guarantee fairness between queues.
{quote}
And we don't really need to ensure that all of our fair shares is guaranteed,
after all, even though we do a sort of order before each of the container's
assignment because the FSQueue#demand is updated in the last time the
FairScheduler# update cycle.
So the value of demand is not real time, which also leads to the fact that we
are not strictly and fairly shared.
So, we can sort all the queues at the FairScheduler#update cycle, and we now
have a default of 0.5 s per update cycle, which is worth doing.
{quote}
We should ensure fairness as possible as we can. Demand doesn't play a big role
in {{FairShareComparator}}. It only matters while resource usage is less than
min share. In addition, applications like MR don't change their demands
frequently. In most cases, there are only 2 times for a job, which is asking
resources for all Mapper and all Reducer.
The community is working on the global scheduling, which will mitigate this
performance issue dramatically in different ways.
BTW, your second point is legit. YARN-4090 is quiet for a while, you are
welcome to contribute if you want.
> FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big
> queues
>
>
> Key: YARN-6361
> URL: https://issues.apache.org/jira/browse/YARN-6361
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: YunFan Zhou
> Attachments: dispatcherthread.png, threads.png, YARN-6361.001.patch
>
>
> FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy.
> Most of the time is spent in FairShareComparator.compare. We could improve
> this by doing the calculations outside the sort loop {{(O\(n\))}} and we
> sorted by a fixed number inside instead {{O(n*log\(n\))}}. This could be an
> performance issue when there are huge number of applications in a single
> queue. The attachments shows the performance impact when there are 10k
> applications in one queue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.
[ https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-6323: - Attachment: YARN-6323-YARN-5355.0001.patch Uploading rebased patch 001. > Rolling upgrade/config change is broken on timeline v2. > > > Key: YARN-6323 > URL: https://issues.apache.org/jira/browse/YARN-6323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6323.001.patch, YARN-6323-YARN-5355.0001.patch > > > Found this issue when deploying on real clusters. If there are apps running > when we enable timeline v2 (with work preserving restart enabled), node > managers will fail to start due to missing app context data. We should > probably assign some default names to these "left over" apps. I believe it's > suboptimal to let users clean up the whole cluster before enabling timeline > v2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7020) TestAMRMProxy#testAMRMProxyTokenRenewal is flakey
[
https://issues.apache.org/jira/browse/YARN-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129576#comment-16129576
]
Robert Kanter commented on YARN-7020:
-
Thanks Jason!
> TestAMRMProxy#testAMRMProxyTokenRenewal is flakey
> -
>
> Key: YARN-7020
> URL: https://issues.apache.org/jira/browse/YARN-7020
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-7020.001.patch
>
>
> {{TestAMRMProxy#testAMRMProxyTokenRenewal}} is flakey. It infrequently fails
> with:
> {noformat}
> testAMRMProxyTokenRenewal(org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy)
> Time elapsed: 19.036 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException:
> Application attempt appattempt_1502837054903_0001_01 doesn't exist in
> ApplicationMasterService cache.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:355)
> at
> org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor$3.allocate(DefaultRequestInterceptor.java:224)
> at
> org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor.allocate(DefaultRequestInterceptor.java:135)
> at
> org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService.allocate(AMRMProxyService.java:279)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1490)
> at org.apache.hadoop.ipc.Client.call(Client.java:1436)
> at org.apache.hadoop.ipc.Client.call(Client.java:1346)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy90.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy91.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy.testAMRMProxyTokenRenewal(TestAMRMProxy.java:190)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Attachment: YARN-6134-YARN-5355.03.patch > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Attachment: (was: YARN-6134-YARN-5355.03.patch) > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Attachment: YARN-6134-YARN-5355.03.patch > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Attachment: (was: YARN-6134-YARN-5355.03.patch) > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Attachment: YARN-6134-YARN-5355.03.patch > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7027) Log aggregation finish time should get logged for trouble shooting.
Junping Du created YARN-7027: Summary: Log aggregation finish time should get logged for trouble shooting. Key: YARN-7027 URL: https://issues.apache.org/jira/browse/YARN-7027 Project: Hadoop YARN Issue Type: Sub-task Components: log-aggregation Reporter: Junping Du Assignee: Junping Du Now, RM track application log aggregation status in RMApp and the status change is triggered by NM heartbeat with log aggregation report. For each node's log aggregation status change from in-progress (NOT_START,RUNNING,RUNNING_WITH_FAILURE) to final status (SUCCEEDED,FAILED, TIMEOUT), it trigger an aggregation for log aggregation status: updateLogAggregationStatus(). The whole progress is log less and we cannot trace the log aggregation problem (delay of log aggregation, etc.) from RM (or NM) log. We should add more log here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7024) Fix issues on recovery in LevelDB store
[
https://issues.apache.org/jira/browse/YARN-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129506#comment-16129506
]
Hadoop QA commented on YARN-7024:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5734 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} YARN-5734 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} YARN-5734 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7024 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882194/YARN-7024-YARN-5734.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aede84dbeb70 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5734 / 96063d5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16939/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16939/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix issues on recovery in LevelDB store
> ---
>
> Key: YARN-7024
> URL: https://issues.apache.org/jira/browse/YARN-7024
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7024-YARN-5734.001.patch
>
[jira] [Updated] (YARN-6876) Create an abstract log writer for extendability
[ https://issues.apache.org/jira/browse/YARN-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6876: Attachment: YARN-6876-trunk.004.patch > Create an abstract log writer for extendability > --- > > Key: YARN-6876 > URL: https://issues.apache.org/jira/browse/YARN-6876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6876-branch-2.001.patch, YARN-6876-trunk.001.patch, > YARN-6876-trunk.002.patch, YARN-6876-trunk.003.patch, > YARN-6876-trunk.004.patch > > > Currently, TFile log writer is used to aggregate log in YARN. We need to add > an abstract layer, and pick up the correct log writer based on the > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129497#comment-16129497
]
Jason Lowe commented on YARN-6988:
--
+1 lgtm. I'll commit this tomorrow if there are no objections.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-6988.001.patch, YARN-6988.002.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6610:
-
Attachment: YARN-6610.YARN-3926.perf-wangda-001.patch
Investigated more, it looks like original perf patch has some typos inside
{{calculateShares2}}, I tried to run all CapacityScheduler tests with the old
perf patch, it causes many failures.
With attached patch {{YARN-6610.YARN-3926.perf-wangda-001.patch}}, it has 2
failures, but looks unrelated. (Ran unit test w/o the patch, it fails as well).
[~templedf], it looks like existing tests in TestResourceCalculator are not
enough to capture the issue, I tried to add some (testCompare2) but it looks
not enough as well. If you have any cycles, could you help to add more unit
tests?
I also tried to run testUserLimitThroughput again, the latest patch shows very
similar result.
Since this is blocker of YARN-3926 merge, and it doesn't show performance
regression, I plan to commit this patch to branch by end of today if no
opposite opinions.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.YARN-3926.perf-rebased.patch,
> YARN-6610.YARN-3926.perf-wangda-001.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129450#comment-16129450 ] Wangda Tan commented on YARN-6852: -- If no opposite opinions, I will commit the patch by Friday. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, > YARN-6852.009.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129437#comment-16129437
]
Hadoop QA commented on YARN-6610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
9s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4510 unchanged - 5 fixed = 4510 total (was 4515) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882196/YARN-6610.YARN-3926.006-rebased.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4ec4ced9f4fe 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / f25bc50 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/16940/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16940/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16940/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
>
[jira] [Commented] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129396#comment-16129396
]
Vrushali C commented on YARN-5355:
--
Thanks [~andrew.wang]
We will make sure to update the release notes to this jira in the next few days.
I have noted down in our documentation jira YARN-6047 regarding the release
note changes in hadoop-project/src/site/markdown/index.md.
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Attachments: Timeline Service v2_ Ideas for Next Steps.pdf,
> YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6047) Documentation updates for TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129384#comment-16129384 ] Vrushali C commented on YARN-6047: -- Note: we should update the release notes at hadoop-project/src/site/markdown/index.md.vm. The alpha4 release notes say "an early preview (alpha 1)", so we should at least update this to say alpha 2. > Documentation updates for TimelineService v2 > > > Key: YARN-6047 > URL: https://issues.apache.org/jira/browse/YARN-6047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6047-YARN-5355.001.patch, > YARN-6047-YARN-5355.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129381#comment-16129381
]
Wangda Tan commented on YARN-6610:
--
Investigated more, it looks like after the perf patch, most user limit
calculation becomes no-op, I'm trying to figure out what happened..
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.YARN-3926.perf-rebased.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[
https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129375#comment-16129375
]
Hadoop QA commented on YARN-6852:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882190/YARN-6852.009.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux ffca14155929 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / de462da |
| Default Java | 1.8.0_144 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16938/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16938/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-6223] Native code changes to support isolate GPU devices by using
> CGroups
> ---
>
> Key: YARN-6852
> URL: https://issues.apache.org/jira/browse/YARN-6852
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6852.001.patch, YARN-6852.002.patch,
> YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch,
> YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch,
> YARN-6852.009.patch
>
>
> This JIRA plan to add support of:
> 1) Isolation in CGroups. (native side).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6610:
-
Attachment: YARN-6610.YARN-3926.006-rebased.patch
YARN-6610.YARN-3926.perf-rebased.patch
Attached rebased patch (with rebased suffix)
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.YARN-3926.perf-rebased.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7024) Fix issues on recovery in LevelDB store
[ https://issues.apache.org/jira/browse/YARN-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129373#comment-16129373 ] Jonathan Hung commented on YARN-7024: - Attached 001 patch which fixes the two issues. > Fix issues on recovery in LevelDB store > --- > > Key: YARN-7024 > URL: https://issues.apache.org/jira/browse/YARN-7024 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7024-YARN-5734.001.patch > > > On recovery, we get the pending mutations not yet committed to store. Upon > confirming them, we remove them in memory while iterating, which may cause > issues in the future. > Also, when getting the pending mutations which were logged but not committed > on recovery, we don't update the txnId in memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7024) Fix issues on recovery in LevelDB store
[ https://issues.apache.org/jira/browse/YARN-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7024: Attachment: YARN-7024-YARN-5734.001.patch > Fix issues on recovery in LevelDB store > --- > > Key: YARN-7024 > URL: https://issues.apache.org/jira/browse/YARN-7024 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7024-YARN-5734.001.patch > > > On recovery, we get the pending mutations not yet committed to store. Upon > confirming them, we remove them in memory while iterating, which may cause > issues in the future. > Also, when getting the pending mutations which were logged but not committed > on recovery, we don't update the txnId in memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129366#comment-16129366
]
Wangda Tan commented on YARN-6610:
--
[~templedf],
Thanks for updating patch, I did some test and got some amazing results:
By executing test: {{mvn test
-Dtest=TestCapacityScheduler#testUserLimitThroughput
-DRunUserLimitThroughput=true}}
For throughput (higher is better)
1) YARN-3926 w/o the patch: 50505.05
2) trunk: 57142.855
3) 3926 + 6610 (perf): {color:red}606060.6 (!!){color}
4) 3926 + 6610 (006): 46403.71
Since YARN-3926 is rebased to trunk, I did some rebases while doing the test as
well. The number looks great, I just hope we didn't accidentally introduce some
issues. I will do some other verifications as well.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch,
> YARN-6610.YARN-3926.perf.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129351#comment-16129351 ] Varun Saxena edited comment on YARN-6134 at 8/16/17 8:22 PM: - Sorry, actually timer is set again on regeneration as well. We call generateTokenAndSetTimers while regenerating the token, which will set the required timers. Let me see if we can refactor to ensure only one runnable runs. was (Author: varun_saxena): Sorry, actually timer is set again on regeneration as well. We call generateTokenAndSetTimers while regenerating the token, which will set the required timers. Anyways I think we can refactor to ensure only one runnable runs. > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129351#comment-16129351 ] Varun Saxena commented on YARN-6134: Sorry, actually timer is set again on regeneration as well. We call generateTokenAndSetTimers while regenerating the token, which will set the required timers. Anyways I think we can refactor to ensure only one runnable runs. > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129345#comment-16129345 ] Varun Saxena commented on YARN-6134: Thanks [~jianhe] for the review. bq. In regenerateToken, looks like the regeneration of token is only scheduled for the first time, it is not rescheduled for the next time? Oops. Seems I missed that in the patch because of refactoring. Will fix it. bq. Instead of scheduling two runnable separately, we can do schedule regeneration of token in the else case below? Yeah this can be done. Will add the logs and update the patch shortly. > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7026) Fair scheduler docs should explain what happens when no placement rules are specified
Daniel Templeton created YARN-7026: -- Summary: Fair scheduler docs should explain what happens when no placement rules are specified Key: YARN-7026 URL: https://issues.apache.org/jira/browse/YARN-7026 Project: Hadoop YARN Issue Type: Improvement Components: docs Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
