[jira] [Updated] (YARN-2784) Yarn project module names in POM needs to consistent acros hadoop project
[ https://issues.apache.org/jira/browse/YARN-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated YARN-2784: - Summary: Yarn project module names in POM needs to consistent acros hadoop project (was: Yarn project module names in POM needs to consitent acros hadoop project) > Yarn project module names in POM needs to consistent acros hadoop project > - > > Key: YARN-2784 > URL: https://issues.apache.org/jira/browse/YARN-2784 > Project: Hadoop YARN > Issue Type: Improvement > Components: scripts >Reporter: Rohith >Assignee: Rohith >Priority: Minor > > All yarn and mapreduce pom.xml has project name has > hadoop-mapreduce/hadoop-yarn. This can be made consistent acros Hadoop > projects build like 'Apache Hadoop Yarn ' and 'Apache Hadoop > MapReduce ". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2778) YARN node CLI should display labels on returned node reports
[
https://issues.apache.org/jira/browse/YARN-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191493#comment-14191493
]
Hadoop QA commented on YARN-2778:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678330/YARN-2778-20141030-1.patch
against trunk revision e0233c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5651//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5651//console
This message is automatically generated.
> YARN node CLI should display labels on returned node reports
>
>
> Key: YARN-2778
> URL: https://issues.apache.org/jira/browse/YARN-2778
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-2778-20141030-1.patch
>
>
> When user run "yarn node -list ..", it should print labels of this node as
> well.
> Upon changes of YARN-2698, this patch can get labels of nodes from
> ResourceManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2784) Yarn project module names in POM needs to consitent acros hadoop project
Rohith created YARN-2784: Summary: Yarn project module names in POM needs to consitent acros hadoop project Key: YARN-2784 URL: https://issues.apache.org/jira/browse/YARN-2784 Project: Hadoop YARN Issue Type: Improvement Components: scripts Reporter: Rohith Assignee: Rohith Priority: Minor All yarn and mapreduce pom.xml has project name has hadoop-mapreduce/hadoop-yarn. This can be made consistent acros Hadoop projects build like 'Apache Hadoop Yarn ' and 'Apache Hadoop MapReduce ". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2778) YARN node CLI should display labels on returned node reports
[ https://issues.apache.org/jira/browse/YARN-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191482#comment-14191482 ] Vinod Kumar Vavilapalli commented on YARN-2778: --- I forgot one missing piece of code: bin/yarn.cmd also needs to be updated.. > YARN node CLI should display labels on returned node reports > > > Key: YARN-2778 > URL: https://issues.apache.org/jira/browse/YARN-2778 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-2778-20141030-1.patch > > > When user run "yarn node -list ..", it should print labels of this node as > well. > Upon changes of YARN-2698, this patch can get labels of nodes from > ResourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191456#comment-14191456 ] Hudson commented on YARN-2779: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6406 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6406/]) YARN-2779. Fixed ResourceManager to not require delegation tokens for communicating with Timeline Service. Contributed by Zhijie Shen. (vinodkv: rev d1828d94435eca21761b0ba8458f9de2f125d012) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/CHANGES.txt > SystemMetricsPublisher can use Kerberos directly instead of timeline DT > --- > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2779.1.patch > > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191442#comment-14191442 ] Vinod Kumar Vavilapalli commented on YARN-2779: --- Straight forward patch. Looks good, tx for the testing. +1, checking this in. > SystemMetricsPublisher can use Kerberos directly instead of timeline DT > --- > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2779.1.patch > > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNodesToLabelsRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodeLabelsResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNodesToLabelsResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/GetClusterNodeLabelsResponse.java > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191417#comment-14191417 ] Vinod Kumar Vavilapalli commented on YARN-2698: --- This looks good, +1. Checking this in. > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2775) There is no close method in NMWebServices#getLogs()
[ https://issues.apache.org/jira/browse/YARN-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191411#comment-14191411 ] Ravi Prakash commented on YARN-2775: Hi skrho! Thanks for the contribution. It seems the JAX-RS spec doesn't say anything about either closing or not closing the OutputStream. Have you seen these objects accumulate on the heap? Are you sure they are never reaped? > There is no close method in NMWebServices#getLogs() > --- > > Key: YARN-2775 > URL: https://issues.apache.org/jira/browse/YARN-2775 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: skrho >Priority: Minor > Attachments: YARN-2775_001.patch > > > If getLogs method is called, fileInputStream object is accumulated in > memory.. > Because fileinputStream object is not closed.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2634) Test failure for TestClientRMTokens
[
https://issues.apache.org/jira/browse/YARN-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191337#comment-14191337
]
Yang Yifan commented on YARN-2634:
--
Face the same problem.
> Test failure for TestClientRMTokens
> ---
>
> Key: YARN-2634
> URL: https://issues.apache.org/jira/browse/YARN-2634
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Junping Du
>Assignee: Jian He
>Priority: Blocker
>
> The test get failed as below:
> {noformat}
> ---
> Test set: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
> ---
> Tests run: 6, Failures: 3, Errors: 2, Skipped: 0, Time elapsed: 60.184 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
> testShortCircuitRenewCancelDifferentHostSamePort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 22.693 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:319)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelDifferentHostSamePort(TestClientRMTokens.java:272)
> testShortCircuitRenewCancelDifferentHostDifferentPort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 20.087 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:319)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelDifferentHostDifferentPort(TestClientRMTokens.java:283)
> testShortCircuitRenewCancel(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 0.031 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.getRmClient(RMDelegationTokenIdentifier.java:148)
> at
> org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.renew(RMDelegationTokenIdentifier.java:101)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:309)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancel(TestClientRMTokens.java:241)
> testShortCircuitRenewCancelSameHostDifferentPort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 0.061 sec <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:319)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelSameHostDifferentPort(TestClientRMTokens.java:261)
> testShortCircuitRenewCancelWildcardAddress(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 0.07 sec <<< ERROR!
> java.lang.NullPointerException: null
> at org.apache.hadoop.net.NetUtils.isLocalAddress(NetUtils.java:684)
> at
> org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.getRmClient(RMDelegationTokenIdentifier.java:149)
>
>
>1,1 Top
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[ https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191308#comment-14191308 ] Zhijie Shen commented on YARN-2770: --- The two test failures are not related, and happen on other Jiras, too: file two tickets for them - YARN-2782 an YARN-2783. > Timeline delegation tokens need to be automatically renewed by the RM > - > > Key: YARN-2770 > URL: https://issues.apache.org/jira/browse/YARN-2770 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.5.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2770.1.patch, YARN-2770.2.patch > > > YarnClient will automatically grab a timeline DT for the application and pass > it to the app AM. Now the timeline DT renew is still dummy. If an app is > running for more than 24h (default DT expiry time), the app AM is no longer > able to use the expired DT to communicate with the timeline server. Since RM > will cache the credentials of each app, and renew the DTs for the running > app. We should provider renew hooks similar to what HDFS DT has for RM, and > set RM user as the renewer when grabbing the timeline DT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2783) TestApplicationClientProtocolOnHA
Zhijie Shen created YARN-2783:
-
Summary: TestApplicationClientProtocolOnHA
Key: YARN-2783
URL: https://issues.apache.org/jira/browse/YARN-2783
Project: Hadoop YARN
Issue Type: Test
Reporter: Zhijie Shen
{code}
Running org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
Tests run: 17, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 147.881 sec
<<< FAILURE! - in
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
testGetContainersOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
Time elapsed: 12.928 sec <<< ERROR!
java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149 to
asf905.gq1.ygridcore.net:28032 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy17.getContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getContainers(ApplicationClientProtocolPBClientImpl.java:400)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy18.getContainers(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:639)
at
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetContainersOnHA(TestApplicationClientProtocolOnHA.java:154)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2782) TestResourceTrackerOnHA fails on trunk
Zhijie Shen created YARN-2782:
-
Summary: TestResourceTrackerOnHA fails on trunk
Key: YARN-2782
URL: https://issues.apache.org/jira/browse/YARN-2782
Project: Hadoop YARN
Issue Type: Test
Reporter: Zhijie Shen
{code}
Running org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.684 sec <<<
FAILURE! - in org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
testResourceTrackerOnHA(org.apache.hadoop.yarn.client.TestResourceTrackerOnHA)
Time elapsed: 12.518 sec <<< ERROR!
java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149 to
asf905.gq1.ygridcore.net:28031 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy87.registerNodeManager(Unknown Source)
at
org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy88.registerNodeManager(Unknown Source)
at
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA.testResourceTrackerOnHA(TestResourceTrackerOnHA.java:64)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[
https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191224#comment-14191224
]
Hadoop QA commented on YARN-2770:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678376/YARN-2770.2.patch
against trunk revision e1f7d65.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5650//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5650//console
This message is automatically generated.
> Timeline delegation tokens need to be automatically renewed by the RM
> -
>
> Key: YARN-2770
> URL: https://issues.apache.org/jira/browse/YARN-2770
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.5.0
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Critical
> Attachments: YARN-2770.1.patch, YARN-2770.2.patch
>
>
> YarnClient will automatically grab a timeline DT for the application and pass
> it to the app AM. Now the timeline DT renew is still dummy. If an app is
> running for more than 24h (default DT expiry time), the app AM is no longer
> able to use the expired DT to communicate with the timeline server. Since RM
> will cache the credentials of each app, and renew the DTs for the running
> app. We should provider renew hooks similar to what HDFS DT has for RM, and
> set RM user as the renewer when grabbing the timeline DT.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191177#comment-14191177 ] Zhijie Shen commented on YARN-2779: --- I've verified it in a secure cluster, and SystemMetricsPublisher works fine with kerberos directly. > SystemMetricsPublisher can use Kerberos directly instead of timeline DT > --- > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2779.1.patch > > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[ https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2770: -- Attachment: YARN-2770.2.patch > Timeline delegation tokens need to be automatically renewed by the RM > - > > Key: YARN-2770 > URL: https://issues.apache.org/jira/browse/YARN-2770 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.5.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2770.1.patch, YARN-2770.2.patch > > > YarnClient will automatically grab a timeline DT for the application and pass > it to the app AM. Now the timeline DT renew is still dummy. If an app is > running for more than 24h (default DT expiry time), the app AM is no longer > able to use the expired DT to communicate with the timeline server. Since RM > will cache the credentials of each app, and renew the DTs for the running > app. We should provider renew hooks similar to what HDFS DT has for RM, and > set RM user as the renewer when grabbing the timeline DT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[ https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191174#comment-14191174 ] Zhijie Shen commented on YARN-2770: --- bq. SecurityUtil#getServerPrincipal may be useful. bq. Let's make sure the renewer name mangling imitates MR JobClient, it is easy to get this wrong. I think we should use HadoopKerberosName#getShortName (AbstractDelegationTokenSecretManager is using it as well) and RM_Principal (which should be there in secure mode) to get the RM daemon user, and HadoopKerberosName will automatically handle auth_to_local if we need to map the auth name to the real operating system name. bq. It'll be great to also test separately that renewal can work fine when https is enabled. I've verified it will work with SSL. BTW, SystemMetricsPublisher works fine with SSL too. To make it work, we must make sure RM have seen the proper configuration for SSL and the truststore. bq. the same DelegationTokenAuthenticatedURL is instantiated multiple times, is it possible to store it as a variable ? It's probably okay to reuse DelegationTokenAuthenticatedURL. However, I'd like to construct one for each request to isolate the possible resource sharing, preventing introducing potential bugs. Actually Jersey client also construct a new URL for each request. It won't be a big overhead, as it doesn't deeply construct something. bq. similarly for the timeline client instantiation. I'm not sure, but guess you're talking about TokenRenewer. Actually I'm following the way that RMDelegationTokenIdentifier does. If we don't construct the client per call, we need to make it a service, and have separate stage for init/start and stop. It may complex the change. Please let me know if you want this change. bq. We may replace the token after renew is really succeeded. According to the design of DelegationTokenAuthenticatedURL, I need to put the DT into the current DelegationTokenAuthenticatedURL.Token, which will be fetched internally to do the corresponding operations. So to renew a given DT, I need to set DT there. However, if it already cached there, the client can skip the set step. Otherwise, I've addressed the remaining comments. Thanks Jian and Vinod! > Timeline delegation tokens need to be automatically renewed by the RM > - > > Key: YARN-2770 > URL: https://issues.apache.org/jira/browse/YARN-2770 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.5.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2770.1.patch > > > YarnClient will automatically grab a timeline DT for the application and pass > it to the app AM. Now the timeline DT renew is still dummy. If an app is > running for more than 24h (default DT expiry time), the app AM is no longer > able to use the expired DT to communicate with the timeline server. Since RM > will cache the credentials of each app, and renew the DTs for the running > app. We should provider renew hooks similar to what HDFS DT has for RM, and > set RM user as the renewer when grabbing the timeline DT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2753) Fix potential issues and code clean up for *NodeLabelsManager
[ https://issues.apache.org/jira/browse/YARN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191173#comment-14191173 ] zhihai xu commented on YARN-2753: - Hi [~xgong], since [~leftnoteasy] agreed to my patch(YARN-2756 is discussed separately) and also the patch passed Hadoop QA, Could you review/commit the patch? thanks zhihai > Fix potential issues and code clean up for *NodeLabelsManager > - > > Key: YARN-2753 > URL: https://issues.apache.org/jira/browse/YARN-2753 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: YARN-2753.000.patch, YARN-2753.001.patch, > YARN-2753.002.patch, YARN-2753.003.patch, YARN-2753.004.patch, > YARN-2753.005.patch > > > Issues include: > * CommonNodeLabelsManager#addToCluserNodeLabels should not change the value > in labelCollections if the key already exists otherwise the Label.resource > will be changed(reset). > * potential NPE(NullPointerException) in checkRemoveLabelsFromNode of > CommonNodeLabelsManager. > ** because when a Node is created, Node.labels can be null. > ** In this case, nm.labels; may be null. So we need check originalLabels not > null before use it(originalLabels.containsAll). > * addToCluserNodeLabels should be protected by writeLock in > RMNodeLabelsManager.java. because we should protect labelCollections in > RMNodeLabelsManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[
https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191144#comment-14191144
]
Hadoop QA commented on YARN-2779:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678351/YARN-2779.1.patch
against trunk revision 81fe8e4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5646//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5646//console
This message is automatically generated.
> SystemMetricsPublisher can use Kerberos directly instead of timeline DT
> ---
>
> Key: YARN-2779
> URL: https://issues.apache.org/jira/browse/YARN-2779
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, timelineserver
>Affects Versions: 2.6.0
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Critical
> Attachments: YARN-2779.1.patch
>
>
> SystemMetricsPublisher is going to grab a timeline DT in secure mode as well.
> The timeline DT will expiry after 24h. No DT renewer will handle renewing
> work for SystemMetricsPublisher, but this has to been handled by itself. In
> addition, SystemMetricsPublisher should cancel the timeline DT when it is
> stopped, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[
https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191135#comment-14191135
]
Hadoop QA commented on YARN-2186:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678363/YARN-2186-trunk-v6.patch
against trunk revision 348bfb7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5648//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5648//console
This message is automatically generated.
> Node Manager uploader service for cache manager
> ---
>
> Key: YARN-2186
> URL: https://issues.apache.org/jira/browse/YARN-2186
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch,
> YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch,
> YARN-2186-trunk-v6.patch
>
>
> Implement the node manager uploader service for the cache manager. This
> service is responsible for communicating with the node manager when it
> uploads resources to the shared cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2383) Add ability to renew ClientToAMToken
[
https://issues.apache.org/jira/browse/YARN-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191131#comment-14191131
]
Hadoop QA commented on YARN-2383:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662875/YARN-2383.preview.3.2.patch
against trunk revision a9331fe.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5649//console
This message is automatically generated.
> Add ability to renew ClientToAMToken
>
>
> Key: YARN-2383
> URL: https://issues.apache.org/jira/browse/YARN-2383
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications, resourcemanager
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-2383.preview.1.patch, YARN-2383.preview.2.patch,
> YARN-2383.preview.3.1.patch, YARN-2383.preview.3.2.patch,
> YARN-2383.preview.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2383) Add ability to renew ClientToAMToken
[ https://issues.apache.org/jira/browse/YARN-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191121#comment-14191121 ] Anubhav Dhoot commented on YARN-2383: - Hi [~xgong] can you please rebase the patch > Add ability to renew ClientToAMToken > > > Key: YARN-2383 > URL: https://issues.apache.org/jira/browse/YARN-2383 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-2383.preview.1.patch, YARN-2383.preview.2.patch, > YARN-2383.preview.3.1.patch, YARN-2383.preview.3.2.patch, > YARN-2383.preview.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[
https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191105#comment-14191105
]
Hadoop QA commented on YARN-2186:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678352/YARN-2186-trunk-v5.patch
against trunk revision 81fe8e4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5647//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5647//console
This message is automatically generated.
> Node Manager uploader service for cache manager
> ---
>
> Key: YARN-2186
> URL: https://issues.apache.org/jira/browse/YARN-2186
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch,
> YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch,
> YARN-2186-trunk-v6.patch
>
>
> Implement the node manager uploader service for the cache manager. This
> service is responsible for communicating with the node manager when it
> uploads resources to the shared cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[
https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191104#comment-14191104
]
Hadoop QA commented on YARN-2698:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678275/YARN-2698-20141030-1.patch
against trunk revision 5e3f428.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5644//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5644//console
This message is automatically generated.
> Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of
> AdminService
> ---
>
> Key: YARN-2698
> URL: https://issues.apache.org/jira/browse/YARN-2698
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch,
> YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch,
> YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch
>
>
> YARN AdminService should have write API only, for other read APIs, they
> should be located at RM ClientService. Include,
> 1) getClusterNodeLabels
> 2) getNodeToLabels
> 3) getNodeReport should contains labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2186) Node Manager uploader service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-2186: -- Attachment: YARN-2186-trunk-v6.patch v.6 patch posted. > Node Manager uploader service for cache manager > --- > > Key: YARN-2186 > URL: https://issues.apache.org/jira/browse/YARN-2186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch, > YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch, > YARN-2186-trunk-v6.patch > > > Implement the node manager uploader service for the cache manager. This > service is responsible for communicating with the node manager when it > uploads resources to the shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2771) DistributedShell's DSConstants are badly named
[
https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191089#comment-14191089
]
Hadoop QA commented on YARN-2771:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678350/YARN-2771.3.patch
against trunk revision 81fe8e4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5645//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5645//console
This message is automatically generated.
> DistributedShell's DSConstants are badly named
> --
>
> Key: YARN-2771
> URL: https://issues.apache.org/jira/browse/YARN-2771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Zhijie Shen
> Attachments: YARN-2771.1.patch, YARN-2771.2.patch, YARN-2771.3.patch
>
>
> I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of
> DISTRIBUTEDSHELLTIMELINEDOMAIN).
> DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to
> be DISTRIBUTED_SHELL_TIMELINE_DOMAIN?
> For the old envs, we can just add new envs that point to the old-one and
> deprecate the old ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191074#comment-14191074 ] Sangjin Lee commented on YARN-2186: --- Got it. I'll update the patch shortly. Thanks! > Node Manager uploader service for cache manager > --- > > Key: YARN-2186 > URL: https://issues.apache.org/jira/browse/YARN-2186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch, > YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch > > > Implement the node manager uploader service for the cache manager. This > service is responsible for communicating with the node manager when it > uploads resources to the shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191070#comment-14191070 ] Karthik Kambatla commented on YARN-2186: The response field of canUpload to uploadable from accepted. > Node Manager uploader service for cache manager > --- > > Key: YARN-2186 > URL: https://issues.apache.org/jira/browse/YARN-2186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch, > YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch > > > Implement the node manager uploader service for the cache manager. This > service is responsible for communicating with the node manager when it > uploads resources to the shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[
https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191066#comment-14191066
]
Sangjin Lee commented on YARN-2186:
---
{quote}
Thanks for filing the follow up JIRAs. If you don't mind, can we annotate the
TODOs in code with those JIRAs. e.g. // TODO (YARN-xyz):
{quote}
I added the JIRA ids for two, but I missed one. I'll fix it.
{quote}
yarn_server_common_service_protos.proto: Should we rename the upload response
content to uploadable?
{quote}
I'm not exactly sure which one you're referring to. We have 4 types:
SCMUploaderNotifyRequest/Response, and SCMUploaderCanUploadRequest/Response.
Are you talking about renaming SCMUploaderCanUploadRequest/Response to
SCMUploaderUploableRequest/Response, or renaming the response field from
accepted to uploadable?
> Node Manager uploader service for cache manager
> ---
>
> Key: YARN-2186
> URL: https://issues.apache.org/jira/browse/YARN-2186
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch,
> YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch
>
>
> Implement the node manager uploader service for the cache manager. This
> service is responsible for communicating with the node manager when it
> uploads resources to the shared cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2186) Node Manager uploader service for cache manager
[
https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191062#comment-14191062
]
Karthik Kambatla commented on YARN-2186:
Thanks Sangjin.
I barely skimmed through the updated patch. Couple of nits before I go through
a detailed review:
# yarn_server_common_service_protos.proto: Should we rename the upload response
content to uploadable?
# Thanks for filing the follow up JIRAs. If you don't mind, can we annotate the
TODOs in code with those JIRAs. e.g. {{// TODO (YARN-xyz):}}
> Node Manager uploader service for cache manager
> ---
>
> Key: YARN-2186
> URL: https://issues.apache.org/jira/browse/YARN-2186
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch,
> YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch
>
>
> Implement the node manager uploader service for the cache manager. This
> service is responsible for communicating with the node manager when it
> uploads resources to the shared cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2186) Node Manager uploader service for cache manager
[ https://issues.apache.org/jira/browse/YARN-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-2186: -- Attachment: YARN-2186-trunk-v5.patch v.5 patch posted. It addresses all the review comments from Karthik. To see the diff better, see https://github.com/ctrezzo/hadoop/compare/trunk...sharedcache-4-YARN-2186-uploader To see the changes between v.4 and v.5, https://github.com/ctrezzo/hadoop/commit/ce090b328505b6acf36f0419a99feeb8ec0cff44 Most of the changes revolve around renaming the protobuf types and the classes. I ended up using "SCMUploader" as the prefix for most of them. I considered "SharedCacheUploader" but it turns out it's bit too verbose, and makes other class names very long. Thanks! > Node Manager uploader service for cache manager > --- > > Key: YARN-2186 > URL: https://issues.apache.org/jira/browse/YARN-2186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2186-trunk-v1.patch, YARN-2186-trunk-v2.patch, > YARN-2186-trunk-v3.patch, YARN-2186-trunk-v4.patch, YARN-2186-trunk-v5.patch > > > Implement the node manager uploader service for the cache manager. This > service is responsible for communicating with the node manager when it > uploads resources to the shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2779: -- Attachment: YARN-2779.1.patch Upload a patch to remove the code of getting the timeline DT in the SystemMetricsPublisher > SystemMetricsPublisher can use Kerberos directly instead of timeline DT > --- > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2779.1.patch > > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2771) DistributedShell's DSConstants are badly named
[ https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2771: -- Attachment: YARN-2771.3.patch Fix the test failure > DistributedShell's DSConstants are badly named > -- > > Key: YARN-2771 > URL: https://issues.apache.org/jira/browse/YARN-2771 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: YARN-2771.1.patch, YARN-2771.2.patch, YARN-2771.3.patch > > > I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of > DISTRIBUTEDSHELLTIMELINEDOMAIN). > DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to > be DISTRIBUTED_SHELL_TIMELINE_DOMAIN? > For the old envs, we can just add new envs that point to the old-one and > deprecate the old ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2778) YARN node CLI should display labels on returned node reports
[ https://issues.apache.org/jira/browse/YARN-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2778: - Description: When user run "yarn node -list ..", it should print labels of this node as well. Upon changes of YARN-2698, this patch can get labels of nodes from ResourceManager > YARN node CLI should display labels on returned node reports > > > Key: YARN-2778 > URL: https://issues.apache.org/jira/browse/YARN-2778 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-2778-20141030-1.patch > > > When user run "yarn node -list ..", it should print labels of this node as > well. > Upon changes of YARN-2698, this patch can get labels of nodes from > ResourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[ https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190998#comment-14190998 ] Vinod Kumar Vavilapalli commented on YARN-2770: --- Quick comments: - Let's make sure the renewer name mangling imitates MR JobClient, it is easy to get this wrong. - It'll be great to also test separately that renewal can work fine when https is enabled. > Timeline delegation tokens need to be automatically renewed by the RM > - > > Key: YARN-2770 > URL: https://issues.apache.org/jira/browse/YARN-2770 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 2.5.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-2770.1.patch > > > YarnClient will automatically grab a timeline DT for the application and pass > it to the app AM. Now the timeline DT renew is still dummy. If an app is > running for more than 24h (default DT expiry time), the app AM is no longer > able to use the expired DT to communicate with the timeline server. Since RM > will cache the credentials of each app, and renew the DTs for the running > app. We should provider renew hooks similar to what HDFS DT has for RM, and > set RM user as the renewer when grabbing the timeline DT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[
https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190971#comment-14190971
]
Hudson commented on YARN-2677:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6399 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6399/])
YARN-2677 registry punycoding of usernames doesn't fix all usernames to be
DNS-valid (stevel) (stevel: rev 81fe8e414748161f537e6902021d63928f8635f1)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/operations/TestRegistryOperations.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistryOperationsService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/client/binding/TestRegistryOperationUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java
> registry punycoding of usernames doesn't fix all usernames to be DNS-valid
> --
>
> Key: YARN-2677
> URL: https://issues.apache.org/jira/browse/YARN-2677
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.6.0
>
> Attachments: YARN-2677-001.patch, YARN-2677-002.patch
>
>
> The registry has a restriction "DNS-valid names only" to retain the future
> option of DNS exporting of the registry.
> to handle complex usernames, it punycodes the username first, using Java's
> {{java.net.IDN}} class.
> This turns out to only map high unicode-> ASCII, and does nothing for
> ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names
> (e.g. with an underscore in them) from being able to register
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2779) SystemMetricsPublisher can use Kerberos directly instead of timeline DT
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2779: -- Summary: SystemMetricsPublisher can use Kerberos directly instead of timeline DT (was: SystemMetricsPublisher needs to renew and cancel timeline DT too) > SystemMetricsPublisher can use Kerberos directly instead of timeline DT > --- > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2779) SystemMetricsPublisher needs to renew and cancel timeline DT too
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190955#comment-14190955 ] Zhijie Shen commented on YARN-2779: --- [~vinodkv], in the current code base, we're SystemMetricsPublisher to grab a timeline DT to talk to the timeline server in secure mode. That's why we need this Jira to add renew and cancel work. But thinking of this issue again, it should be okay to let RM talk to the timeline server with kerberos directly. As this is the only process, which will not add too much workload to the kerberos server. So on the other hand, let's remove the getting DT logic, and let RM uses kerberos directly. > SystemMetricsPublisher needs to renew and cancel timeline DT too > > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (YARN-2779) SystemMetricsPublisher needs to renew and cancel timeline DT too
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reopened YARN-2779: --- > SystemMetricsPublisher needs to renew and cancel timeline DT too > > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2778) YARN node CLI should display labels on returned node reports
[ https://issues.apache.org/jira/browse/YARN-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2778: - Attachment: YARN-2778-20141030-1.patch > YARN node CLI should display labels on returned node reports > > > Key: YARN-2778 > URL: https://issues.apache.org/jira/browse/YARN-2778 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-2778-20141030-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2730) Only one localizer can run on a NodeManager at a time
[ https://issues.apache.org/jira/browse/YARN-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190947#comment-14190947 ] Siqi Li commented on YARN-2730: --- [~jlowe] Hi Jason, do you have some time to take a look at this patch? > Only one localizer can run on a NodeManager at a time > - > > Key: YARN-2730 > URL: https://issues.apache.org/jira/browse/YARN-2730 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Critical > Attachments: YARN-2730.v1.patch > > > We are seeing that when one of the localizerRunner stuck, the rest of the > localizerRunners are blocked. We should remove the synchronized modifier. > The synchronized modifier appears to have been added by > https://issues.apache.org/jira/browse/MAPREDUCE-3537 > It could be removed if Localizer doesn't depend on current directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190922#comment-14190922 ] Wangda Tan commented on YARN-2698: -- Tried to run tests locally, can be passed, should not related to this change > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2779) SystemMetricsPublisher needs to renew and cancel timeline DT too
[ https://issues.apache.org/jira/browse/YARN-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-2779. --- Resolution: Invalid This doesn't make any sense. RM can simply use kerberos based auth to talk to Timeline service. It doesn't need tokens at all. Closing this as invalid, reopen if you disagree. > SystemMetricsPublisher needs to renew and cancel timeline DT too > > > Key: YARN-2779 > URL: https://issues.apache.org/jira/browse/YARN-2779 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.6.0 >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Critical > > SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. > The timeline DT will expiry after 24h. No DT renewer will handle renewing > work for SystemMetricsPublisher, but this has to been handled by itself. In > addition, SystemMetricsPublisher should cancel the timeline DT when it is > stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[
https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190907#comment-14190907
]
Hadoop QA commented on YARN-2698:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678275/YARN-2698-20141030-1.patch
against trunk revision c2866ac.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5641//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5641//console
This message is automatically generated.
> Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of
> AdminService
> ---
>
> Key: YARN-2698
> URL: https://issues.apache.org/jira/browse/YARN-2698
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch,
> YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch,
> YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch
>
>
> YARN AdminService should have write API only, for other read APIs, they
> should be located at RM ClientService. Include,
> 1) getClusterNodeLabels
> 2) getNodeToLabels
> 3) getNodeReport should contains labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190883#comment-14190883 ] Wangda Tan commented on YARN-2698: -- And please note that, I removed getClusterNodeLabels and getNodeToLabels out from RM Admin CLI, reader interfaces shouldn't exist in RM Admin CLI. Getting labels of running NMs can be done via YARN-2778 Wangda > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2770) Timeline delegation tokens need to be automatically renewed by the RM
[
https://issues.apache.org/jira/browse/YARN-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190855#comment-14190855
]
Jian He commented on YARN-2770:
---
thanks Zhijie ! some comments:
- {{SecurityUtil#getServerPrincipal}} may be useful.
{code}
if (rmPrincipal != null && rmPrincipal.length() > 0) {
renewer = new KerberosName(rmPrincipal).getServiceName();
}
{code}
- We may replace the token after renew is really succeeded.
{code}
if (!timelineDT.equals(token.getDelegationToken())) {
token.setDelegationToken((Token) timelineDT);
}
{code}
- In cancelDelegationToken, why replacing the token. Also rename the
{{renewDTAction}} to {{cacnelDT}}
{code}
// If the timeline DT to renew is different than cached, replace it.
// Token to set every time for retry, because when exception
happens,
// DelegationTokenAuthenticatedURL will reset it to null;
if (!timelineDT.equals(token.getDelegationToken())) {
token.setDelegationToken((Token) timelineDT);
}
{code}
- the same DelegationTokenAuthenticatedURL is instantiated multiple times, is
it possible to store it as a variable ?
{code}
DelegationTokenAuthenticatedURL authUrl =
new DelegationTokenAuthenticatedURL(authenticator,
connConfigurator);
{code}
- similarly for the timeline client instantiation.
{code}
TimelineClient client = TimelineClient.createTimelineClient();
client.init(conf);
client.start();
{code}
> Timeline delegation tokens need to be automatically renewed by the RM
> -
>
> Key: YARN-2770
> URL: https://issues.apache.org/jira/browse/YARN-2770
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.5.0
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Critical
> Attachments: YARN-2770.1.patch
>
>
> YarnClient will automatically grab a timeline DT for the application and pass
> it to the app AM. Now the timeline DT renew is still dummy. If an app is
> running for more than 24h (default DT expiry time), the app AM is no longer
> able to use the expired DT to communicate with the timeline server. Since RM
> will cache the credentials of each app, and renew the DTs for the running
> app. We should provider renew hooks similar to what HDFS DT has for RM, and
> set RM user as the renewer when grabbing the timeline DT.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[
https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated YARN-2677:
Hadoop Flags: Reviewed
+1 for the patch. I verified this on both Mac and Windows. Thanks, Steve!
> registry punycoding of usernames doesn't fix all usernames to be DNS-valid
> --
>
> Key: YARN-2677
> URL: https://issues.apache.org/jira/browse/YARN-2677
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-2677-001.patch, YARN-2677-002.patch
>
>
> The registry has a restriction "DNS-valid names only" to retain the future
> option of DNS exporting of the registry.
> to handle complex usernames, it punycodes the username first, using Java's
> {{java.net.IDN}} class.
> This turns out to only map high unicode-> ASCII, and does nothing for
> ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names
> (e.g. with an underscore in them) from being able to register
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2766) ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
[
https://issues.apache.org/jira/browse/YARN-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190820#comment-14190820
]
Hadoop QA commented on YARN-2766:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678285/YARN-2766.patch
against trunk revision c2866ac.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5643//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5643//console
This message is automatically generated.
> ApplicationHistoryManager is expected to return a sorted list of
> apps/attempts/containers
> --
>
> Key: YARN-2766
> URL: https://issues.apache.org/jira/browse/YARN-2766
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-2766.patch, YARN-2766.patch, YARN-2766.patch,
> YARN-2766.patch
>
>
> {{TestApplicationHistoryClientService.testContainers}} and
> {{TestApplicationHistoryClientService.testApplicationAttempts}} both fail
> because the test assertions are assuming a returned Collection is in a
> certain order. The collection comes from a HashMap, so the order is not
> guaranteed, plus, according to [this
> page|http://docs.oracle.com/javase/8/docs/technotes/guides/collections/changes8.html],
> there are situations where the iteration order of a HashMap will be
> different between Java 7 and 8.
> We should fix the test code to not assume a specific ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2781) support more flexible policy for uploading in shared cache
Sangjin Lee created YARN-2781: - Summary: support more flexible policy for uploading in shared cache Key: YARN-2781 URL: https://issues.apache.org/jira/browse/YARN-2781 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sangjin Lee Today all resources are always uploaded as long as the client wants to upload it. We may want to implement a feature where the shared cache manager can instruct the node managers not to upload under some circumstances. Some examples may be uploading a resource if it is seen more than N number of times. This doesn't need to be included in the first version of the shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne reassigned YARN-2780: Assignee: Eric Payne > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
Koji Noguchi created YARN-2780: -- Summary: Log aggregated resource allocation in rm-appsummary.log Key: YARN-2780 URL: https://issues.apache.org/jira/browse/YARN-2780 Project: Hadoop YARN Issue Type: New Feature Components: resourcemanager Reporter: Koji Noguchi Priority: Minor YARN-415 added useful information about resource usage by applications. Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2766) ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
[
https://issues.apache.org/jira/browse/YARN-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-2766:
Attachment: YARN-2766.patch
Makes sense. The new patch updates ApplicationHistoryManagerOnTimelineStore
instead of the protobuf classes. It replaces the HashMap with a LinkedHashMap,
which does maintain order (the data is loaded from the store into the Map in a
consistent order).
> ApplicationHistoryManager is expected to return a sorted list of
> apps/attempts/containers
> --
>
> Key: YARN-2766
> URL: https://issues.apache.org/jira/browse/YARN-2766
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-2766.patch, YARN-2766.patch, YARN-2766.patch,
> YARN-2766.patch
>
>
> {{TestApplicationHistoryClientService.testContainers}} and
> {{TestApplicationHistoryClientService.testApplicationAttempts}} both fail
> because the test assertions are assuming a returned Collection is in a
> certain order. The collection comes from a HashMap, so the order is not
> guaranteed, plus, according to [this
> page|http://docs.oracle.com/javase/8/docs/technotes/guides/collections/changes8.html],
> there are situations where the iteration order of a HashMap will be
> different between Java 7 and 8.
> We should fix the test code to not assume a specific ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2771) DistributedShell's DSConstants are badly named
[
https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190685#comment-14190685
]
Hadoop QA commented on YARN-2771:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678276/YARN-2771.2.patch
against trunk revision c2866ac.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5642//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5642//console
This message is automatically generated.
> DistributedShell's DSConstants are badly named
> --
>
> Key: YARN-2771
> URL: https://issues.apache.org/jira/browse/YARN-2771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Zhijie Shen
> Attachments: YARN-2771.1.patch, YARN-2771.2.patch
>
>
> I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of
> DISTRIBUTEDSHELLTIMELINEDOMAIN).
> DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to
> be DISTRIBUTED_SHELL_TIMELINE_DOMAIN?
> For the old envs, we can just add new envs that point to the old-one and
> deprecate the old ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2779) SystemMetricsPublisher needs to renew and cancel timeline DT too
Zhijie Shen created YARN-2779: - Summary: SystemMetricsPublisher needs to renew and cancel timeline DT too Key: YARN-2779 URL: https://issues.apache.org/jira/browse/YARN-2779 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, timelineserver Affects Versions: 2.6.0 Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Critical SystemMetricsPublisher is going to grab a timeline DT in secure mode as well. The timeline DT will expiry after 24h. No DT renewer will handle renewing work for SystemMetricsPublisher, but this has to been handled by itself. In addition, SystemMetricsPublisher should cancel the timeline DT when it is stopped, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2771) DistributedShell's DSConstants are badly named
[ https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2771: -- Attachment: YARN-2771.2.patch Just realized I've missed one suggestion from Vinod in the description. I think it makes sense to keep the old constants, and mark them deprecated. In addition. I altered the logic in the AM to accept both new and old evn vars. > DistributedShell's DSConstants are badly named > -- > > Key: YARN-2771 > URL: https://issues.apache.org/jira/browse/YARN-2771 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: YARN-2771.1.patch, YARN-2771.2.patch > > > I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of > DISTRIBUTEDSHELLTIMELINEDOMAIN). > DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to > be DISTRIBUTED_SHELL_TIMELINE_DOMAIN? > For the old envs, we can just add new envs that point to the old-one and > deprecate the old ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2698: - Attachment: YARN-2698-20141030-1.patch Attached patch contains get node reports > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch, YARN-2698-20141030-1.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2772) DistributedShell's timeline related options are not clear
[
https://issues.apache.org/jira/browse/YARN-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190566#comment-14190566
]
Hadoop QA commented on YARN-2772:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678251/YARN-2772.2.patch
against trunk revision b811212.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5640//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5640//console
This message is automatically generated.
> DistributedShell's timeline related options are not clear
> -
>
> Key: YARN-2772
> URL: https://issues.apache.org/jira/browse/YARN-2772
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Zhijie Shen
> Attachments: YARN-2772.1.patch, YARN-2772.2.patch
>
>
> The new options "domain" and "create" options - they are not descriptive at
> all. It is also not clear when view_acls and modify_acls need to be set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[ https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-2729: Attachment: YARN-2729.20141031-1.patch Updated with review comments and test cases > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup > --- > > Key: YARN-2729 > URL: https://issues.apache.org/jira/browse/YARN-2729 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch, > YARN-2729.20141031-1.patch > > > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-2495: Attachment: YARN-2495.20141031-1.patch Updated with fixes for all review comments,testcases and rebasing to the trunk code > Allow admin specify labels from each NM (Distributed configuration) > --- > > Key: YARN-2495 > URL: https://issues.apache.org/jira/browse/YARN-2495 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch, > YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch, > YARN-2495_20141022.1.patch > > > Target of this JIRA is to allow admin specify labels in each NM, this covers > - User can set labels in each NM (by setting yarn-site.xml or using script > suggested by [~aw]) > - NM will send labels to RM via ResourceTracker API > - RM will set labels in NodeLabelManager when NM register/update labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2772) DistributedShell's timeline related options are not clear
[ https://issues.apache.org/jira/browse/YARN-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2772: -- Attachment: YARN-2772.2.patch Make one improvement. If failure happens when creating a domain, the client will reset the domain ID, and make the timeline entities go into DEFAULT. > DistributedShell's timeline related options are not clear > - > > Key: YARN-2772 > URL: https://issues.apache.org/jira/browse/YARN-2772 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: YARN-2772.1.patch, YARN-2772.2.patch > > > The new options "domain" and "create" options - they are not descriptive at > all. It is also not clear when view_acls and modify_acls need to be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2727) In RMAdminCLI usage display, instead of "yarn.node-labels.fs-store.root-dir", "yarn.node-labels.fs-store.uri" is being displayed
[ https://issues.apache.org/jira/browse/YARN-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190473#comment-14190473 ] Naganarasimha G R commented on YARN-2727: - Seems like failure is not due to this patch... Can we rerun the test cases with patch again ? > In RMAdminCLI usage display, instead of "yarn.node-labels.fs-store.root-dir", > "yarn.node-labels.fs-store.uri" is being displayed > > > Key: YARN-2727 > URL: https://issues.apache.org/jira/browse/YARN-2727 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-2727.20141023.1.patch > > > In org.apache.hadoop.yarn.client.cli.RMAdminCLI usage display instead of > "yarn.node-labels.fs-store.root-dir", "yarn.node-labels.fs-store.uri" is > being used > And also some modifications for the description -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2778) YARN node CLI should display labels on returned node reports
Wangda Tan created YARN-2778: Summary: YARN node CLI should display labels on returned node reports Key: YARN-2778 URL: https://issues.apache.org/jira/browse/YARN-2778 Project: Hadoop YARN Issue Type: Sub-task Components: client Reporter: Wangda Tan Assignee: Wangda Tan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2771) DistributedShell's DSConstants are badly named
[ https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190430#comment-14190430 ] Zhijie Shen commented on YARN-2771: --- Constant name changes. No need for test case. > DistributedShell's DSConstants are badly named > -- > > Key: YARN-2771 > URL: https://issues.apache.org/jira/browse/YARN-2771 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Attachments: YARN-2771.1.patch > > > I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of > DISTRIBUTEDSHELLTIMELINEDOMAIN). > DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to > be DISTRIBUTED_SHELL_TIMELINE_DOMAIN? > For the old envs, we can just add new envs that point to the old-one and > deprecate the old ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2698: - Description: YARN AdminService should have write API only, for other read APIs, they should be located at RM ClientService. Include, 1) getClusterNodeLabels 2) getNodeToLabels 3) getNodeReport should contains labels was:YARN RMAdminCLI and AdminService should have write API only, for other read APIs, they should be located at YARNCLI and RMClientService. > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch > > > YARN AdminService should have write API only, for other read APIs, they > should be located at RM ClientService. Include, > 1) getClusterNodeLabels > 2) getNodeToLabels > 3) getNodeReport should contains labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService
[ https://issues.apache.org/jira/browse/YARN-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2698: - Summary: Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of AdminService (was: Move getClusterNodeLabels and getNodeToLabels to YARN CLI instead of RMAdminCLI) > Move getClusterNodeLabels and getNodeToLabels to YarnClient instead of > AdminService > --- > > Key: YARN-2698 > URL: https://issues.apache.org/jira/browse/YARN-2698 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-2698-20141028-1.patch, YARN-2698-20141028-2.patch, > YARN-2698-20141028-3.patch, YARN-2698-20141029-1.patch, > YARN-2698-20141029-2.patch > > > YARN RMAdminCLI and AdminService should have write API only, for other read > APIs, they should be located at YARNCLI and RMClientService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2777) Mark the end of individual log in aggregated log
Ted Yu created YARN-2777:
Summary: Mark the end of individual log in aggregated log
Key: YARN-2777
URL: https://issues.apache.org/jira/browse/YARN-2777
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Ted Yu
Below is snippet of aggregated log showing hbase master log:
{code}
LogType: hbase-hbase-master-ip-172-31-34-167.log
LogUploadTime: 29-Oct-2014 22:31:55
LogLength: 24103045
Log Contents:
Wed Oct 29 15:43:57 UTC 2014 Starting master on ip-172-31-34-167
...
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
at java.lang.Thread.run(Thread.java:745)
LogType: hbase-hbase-master-ip-172-31-34-167.out
{code}
Since logs from various daemons are aggregated in one log file, it would be
desirable to mark the end of one log before starting with the next.
e.g. with such a line:
{code}
End of LogType: hbase-hbase-master-ip-172-31-34-167.log
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-661) NM fails to cleanup local directories for users
[ https://issues.apache.org/jira/browse/YARN-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190220#comment-14190220 ] Hudson commented on YARN-661: - FAILURE: Integrated in Hadoop-trunk-Commit #6393 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6393/]) YARN-2755. NM fails to clean up usercache_DEL_ dirs after YARN-661. Contributed by Siqi Li (jlowe: rev 73e626ad91cd5c06a005068d8432fd16e06fe6a0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/CHANGES.txt > NM fails to cleanup local directories for users > --- > > Key: YARN-661 > URL: https://issues.apache.org/jira/browse/YARN-661 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.1.0-beta, 0.23.8 >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta > > Attachments: YARN-661-20130701.patch, YARN-661-20130708.patch, > YARN-661-20130710.1.patch, YARN-661-20130711.1.patch, > YARN-661-20130712.1.patch, YARN-661-20130715.1.patch, > YARN-661-20130716.1.patch > > > YARN-71 added deletion of local directories on startup, but in practice it > fails to delete the directories because of permission problems. The > top-level usercache directory is owned by the user but is in a directory that > is not writable by the user. Therefore the deletion of the user's usercache > directory, as the user, fails due to lack of permissions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2755) NM fails to clean up usercache_DEL_ dirs after YARN-661
[ https://issues.apache.org/jira/browse/YARN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190221#comment-14190221 ] Hudson commented on YARN-2755: -- FAILURE: Integrated in Hadoop-trunk-Commit #6393 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6393/]) YARN-2755. NM fails to clean up usercache_DEL_ dirs after YARN-661. Contributed by Siqi Li (jlowe: rev 73e626ad91cd5c06a005068d8432fd16e06fe6a0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/CHANGES.txt > NM fails to clean up usercache_DEL_ dirs after YARN-661 > -- > > Key: YARN-2755 > URL: https://issues.apache.org/jira/browse/YARN-2755 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2755.v1.patch, YARN-2755.v2.patch, > YARN-2755.v3.patch, YARN-2755.v4.patch > > > When NM restarts frequently due to some reason, a large number of directories > like these left in /data/disk$num/yarn/local/: > /data/disk1/yarn/local/usercache_DEL_1414372756105 > /data/disk1/yarn/local/usercache_DEL_1413557901696 > /data/disk1/yarn/local/usercache_DEL_1413657004894 > /data/disk1/yarn/local/usercache_DEL_1413675321860 > /data/disk1/yarn/local/usercache_DEL_1414093167936 > /data/disk1/yarn/local/usercache_DEL_1413565841271 > These directories are empty, but take up 100M+ due to the number of them. > There were 38714 on the machine I looked at per data disk. > It appears to be a regression introduced by YARN-661 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2742) FairSchedulerConfiguration should allow extra spaces between value and unit
[
https://issues.apache.org/jira/browse/YARN-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190205#comment-14190205
]
Hudson commented on YARN-2742:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1942 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1942/])
YARN-2742. FairSchedulerConfiguration should allow extra spaces between value
and unit. (Wei Yan via kasha) (kasha: rev
782971ae7a0247bcf5920e10b434b7e0954dd868)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
> FairSchedulerConfiguration should allow extra spaces between value and unit
> ---
>
> Key: YARN-2742
> URL: https://issues.apache.org/jira/browse/YARN-2742
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.4.0
>Reporter: Sangjin Lee
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2742-1.patch, YARN-2742-2.patch
>
>
> FairSchedulerConfiguration is very strict about the number of space
> characters between the value and the unit: 0 or 1 space.
> For example, for values like the following:
> {noformat}
> 4096 mb, 2 vcores
> {noformat}
> (note 2 spaces)
> This above line fails to parse:
> {noformat}
> 2014-10-24 22:56:40,802 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
> Failed to reload fair scheduler config file - will use existing allocations.
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Missing resource: mb
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.findResource(FairSchedulerConfiguration.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.parseResourceConfigValue(FairSchedulerConfiguration.java:231)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:381)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService$1.run(AllocationFileLoaderService.java:117)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190208#comment-14190208
]
Hudson commented on YARN-2712:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1942 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1942/])
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and
headroom checks. (Tsuyoshi Ozawa via kasha) (kasha: rev
179cab81e0bde1af0cba6131f16ff127358a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* hadoop-yarn-project/CHANGES.txt
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2769) Timeline server domain not set correctly when using shell_command on Windows
[
https://issues.apache.org/jira/browse/YARN-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190200#comment-14190200
]
Hudson commented on YARN-2769:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1942 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1942/])
YARN-2769. Fixed the problem that timeline domain is not set in distributed
shell AM when using shell_command on Windows. Contributed by Varun Vasudev.
(zjshen: rev a8c120222047280234c3411ce1c1c9b17f08c851)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Timeline server domain not set correctly when using shell_command on Windows
>
>
> Key: YARN-2769
> URL: https://issues.apache.org/jira/browse/YARN-2769
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Fix For: 2.6.0
>
> Attachments: apache-yarn-2769.0.patch
>
>
> The bug is caught by one of the unit tests which fails.
> {noformat}
> Running
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.661 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.applications.distribut
> testDSShellWithDomain(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 37.366 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[TEST_DOMAIN]> but was:<[DEFAULT]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:290)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithDomain(TestDistributedShell.java:179)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-2776: --- Priority: Major (was: Critical) > In HA mode, can't set ip but hostname to > "yarn.resourcemanager.webapp.address.*" > - > > Key: YARN-2776 > URL: https://issues.apache.org/jira/browse/YARN-2776 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: meiyoula > Attachments: YARN-2766.patch > > > In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, > I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, > but spark own page has a bug when the address turns to yarn address. > But when setting yarn.resourcemanager.webapp.address.* with hostname:port, > the webs both ok. > The error in web: > HTTP ERROR: 500 > Problem accessing /stages/. Reason: > Server Error > The exception in log : > WARN | [qtp542345580-71] | /stages/ | > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) > javax.servlet.ServletException: Could not determine the proxy server for > redirection > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2755) NM fails to clean up usercache_DEL_ dirs after YARN-661
[ https://issues.apache.org/jira/browse/YARN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190172#comment-14190172 ] Jason Lowe commented on YARN-2755: -- +1, committing this. > NM fails to clean up usercache_DEL_ dirs after YARN-661 > -- > > Key: YARN-2755 > URL: https://issues.apache.org/jira/browse/YARN-2755 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Critical > Attachments: YARN-2755.v1.patch, YARN-2755.v2.patch, > YARN-2755.v3.patch, YARN-2755.v4.patch > > > When NM restarts frequently due to some reason, a large number of directories > like these left in /data/disk$num/yarn/local/: > /data/disk1/yarn/local/usercache_DEL_1414372756105 > /data/disk1/yarn/local/usercache_DEL_1413557901696 > /data/disk1/yarn/local/usercache_DEL_1413657004894 > /data/disk1/yarn/local/usercache_DEL_1413675321860 > /data/disk1/yarn/local/usercache_DEL_1414093167936 > /data/disk1/yarn/local/usercache_DEL_1413565841271 > These directories are empty, but take up 100M+ due to the number of them. > There were 38714 on the machine I looked at per data disk. > It appears to be a regression introduced by YARN-661 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[
https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190151#comment-14190151
]
Hadoop QA commented on YARN-2677:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678180/YARN-2677-002.patch
against trunk revision 179cab8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5639//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5639//console
This message is automatically generated.
> registry punycoding of usernames doesn't fix all usernames to be DNS-valid
> --
>
> Key: YARN-2677
> URL: https://issues.apache.org/jira/browse/YARN-2677
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-2677-001.patch, YARN-2677-002.patch
>
>
> The registry has a restriction "DNS-valid names only" to retain the future
> option of DNS exporting of the registry.
> to handle complex usernames, it punycodes the username first, using Java's
> {{java.net.IDN}} class.
> This turns out to only map high unicode-> ASCII, and does nothing for
> ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names
> (e.g. with an underscore in them) from being able to register
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[
https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-2677:
-
Attachment: YARN-2677-002.patch
Updated patch which also converts usernames to lower case (english locale).
This handles names like {{Administrator}} which you can see on windows
> registry punycoding of usernames doesn't fix all usernames to be DNS-valid
> --
>
> Key: YARN-2677
> URL: https://issues.apache.org/jira/browse/YARN-2677
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-2677-001.patch, YARN-2677-002.patch
>
>
> The registry has a restriction "DNS-valid names only" to retain the future
> option of DNS exporting of the registry.
> to handle complex usernames, it punycodes the username first, using Java's
> {{java.net.IDN}} class.
> This turns out to only map high unicode-> ASCII, and does nothing for
> ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names
> (e.g. with an underscore in them) from being able to register
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2769) Timeline server domain not set correctly when using shell_command on Windows
[
https://issues.apache.org/jira/browse/YARN-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190079#comment-14190079
]
Hudson commented on YARN-2769:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1917 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/])
YARN-2769. Fixed the problem that timeline domain is not set in distributed
shell AM when using shell_command on Windows. Contributed by Varun Vasudev.
(zjshen: rev a8c120222047280234c3411ce1c1c9b17f08c851)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Timeline server domain not set correctly when using shell_command on Windows
>
>
> Key: YARN-2769
> URL: https://issues.apache.org/jira/browse/YARN-2769
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Fix For: 2.6.0
>
> Attachments: apache-yarn-2769.0.patch
>
>
> The bug is caught by one of the unit tests which fails.
> {noformat}
> Running
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.661 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.applications.distribut
> testDSShellWithDomain(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 37.366 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[TEST_DOMAIN]> but was:<[DEFAULT]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:290)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithDomain(TestDistributedShell.java:179)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2742) FairSchedulerConfiguration should allow extra spaces between value and unit
[
https://issues.apache.org/jira/browse/YARN-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190084#comment-14190084
]
Hudson commented on YARN-2742:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1917 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/])
YARN-2742. FairSchedulerConfiguration should allow extra spaces between value
and unit. (Wei Yan via kasha) (kasha: rev
782971ae7a0247bcf5920e10b434b7e0954dd868)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
> FairSchedulerConfiguration should allow extra spaces between value and unit
> ---
>
> Key: YARN-2742
> URL: https://issues.apache.org/jira/browse/YARN-2742
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.4.0
>Reporter: Sangjin Lee
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2742-1.patch, YARN-2742-2.patch
>
>
> FairSchedulerConfiguration is very strict about the number of space
> characters between the value and the unit: 0 or 1 space.
> For example, for values like the following:
> {noformat}
> 4096 mb, 2 vcores
> {noformat}
> (note 2 spaces)
> This above line fails to parse:
> {noformat}
> 2014-10-24 22:56:40,802 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
> Failed to reload fair scheduler config file - will use existing allocations.
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Missing resource: mb
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.findResource(FairSchedulerConfiguration.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.parseResourceConfigValue(FairSchedulerConfiguration.java:231)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:381)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService$1.run(AllocationFileLoaderService.java:117)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190087#comment-14190087
]
Hudson commented on YARN-2712:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1917 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/])
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and
headroom checks. (Tsuyoshi Ozawa via kasha) (kasha: rev
179cab81e0bde1af0cba6131f16ff127358a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2742) FairSchedulerConfiguration should allow extra spaces between value and unit
[
https://issues.apache.org/jira/browse/YARN-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189970#comment-14189970
]
Hudson commented on YARN-2742:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #728 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/728/])
YARN-2742. FairSchedulerConfiguration should allow extra spaces between value
and unit. (Wei Yan via kasha) (kasha: rev
782971ae7a0247bcf5920e10b434b7e0954dd868)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* hadoop-yarn-project/CHANGES.txt
> FairSchedulerConfiguration should allow extra spaces between value and unit
> ---
>
> Key: YARN-2742
> URL: https://issues.apache.org/jira/browse/YARN-2742
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.4.0
>Reporter: Sangjin Lee
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2742-1.patch, YARN-2742-2.patch
>
>
> FairSchedulerConfiguration is very strict about the number of space
> characters between the value and the unit: 0 or 1 space.
> For example, for values like the following:
> {noformat}
> 4096 mb, 2 vcores
> {noformat}
> (note 2 spaces)
> This above line fails to parse:
> {noformat}
> 2014-10-24 22:56:40,802 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
> Failed to reload fair scheduler config file - will use existing allocations.
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Missing resource: mb
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.findResource(FairSchedulerConfiguration.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration.parseResourceConfigValue(FairSchedulerConfiguration.java:231)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:381)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService$1.run(AllocationFileLoaderService.java:117)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2769) Timeline server domain not set correctly when using shell_command on Windows
[
https://issues.apache.org/jira/browse/YARN-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189965#comment-14189965
]
Hudson commented on YARN-2769:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #728 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/728/])
YARN-2769. Fixed the problem that timeline domain is not set in distributed
shell AM when using shell_command on Windows. Contributed by Varun Vasudev.
(zjshen: rev a8c120222047280234c3411ce1c1c9b17f08c851)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Timeline server domain not set correctly when using shell_command on Windows
>
>
> Key: YARN-2769
> URL: https://issues.apache.org/jira/browse/YARN-2769
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Fix For: 2.6.0
>
> Attachments: apache-yarn-2769.0.patch
>
>
> The bug is caught by one of the unit tests which fails.
> {noformat}
> Running
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.661 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.applications.distribut
> testDSShellWithDomain(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 37.366 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[TEST_DOMAIN]> but was:<[DEFAULT]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:290)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithDomain(TestDistributedShell.java:179)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189973#comment-14189973
]
Hudson commented on YARN-2712:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #728 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/728/])
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and
headroom checks. (Tsuyoshi Ozawa via kasha) (kasha: rev
179cab81e0bde1af0cba6131f16ff127358a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189934#comment-14189934 ] Tsuyoshi OZAWA commented on YARN-2765: -- Currently we can assume that LeveldbRMStateStore is access from single process, thus failure detection itself is done by EmbeddedElector which depends on ZooKeeper in addition to no support of fencing on LevelDBRMStateStore. It means we need to launch ZooKeeper and it's normal decision to use ZKRMStateStore in this case. Please correct me if I'm wrong. On another front, if we use RockDB as a backend db of timeline server, we don't need to use leveldb and it's good decision to switch the dependency. > Add leveldb-based implementation for RMStateStore > - > > Key: YARN-2765 > URL: https://issues.apache.org/jira/browse/YARN-2765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-2765.patch, YARN-2765v2.patch > > > It would be nice to have a leveldb option to the resourcemanager recovery > store. Leveldb would provide some benefits over the existing filesystem store > such as better support for atomic operations, fewer I/O ops per state update, > and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2762) Provide RMAdminCLI args validation for NodeLabelManager operations
[ https://issues.apache.org/jira/browse/YARN-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated YARN-2762: - Issue Type: Sub-task (was: Improvement) Parent: YARN-2492 > Provide RMAdminCLI args validation for NodeLabelManager operations > -- > > Key: YARN-2762 > URL: https://issues.apache.org/jira/browse/YARN-2762 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Rohith >Assignee: Rohith >Priority: Minor > Attachments: YARN-2762.patch > > > All NodeLabel args validation's are done at server side. The same can be done > at RMAdminCLI so that unnecessary RPC calls can be avoided. > And for the input such as "x,y,,z,", no need to add empty string instead can > be skipped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meiyoula updated YARN-2776: --- Description: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. But when setting yarn.resourcemanager.webapp.address.* with hostname:port, the webs both ok. The error in web: HTTP ERROR: 500 Problem accessing /stages/. Reason: Server Error The exception in log : WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) was: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The error in web: HTTP ERROR: 500 Problem accessing /stages/. Reason: Server Error The exception in log : WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:49
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meiyoula updated YARN-2776: --- Description: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The error in web: HTTP ERROR: 500 Problem accessing /stages/. Reason: Server Error The exception in log : WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) was: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The exception shows like this: WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComp
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meiyoula updated YARN-2776: --- Description: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The exception shows like this: WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) was: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The exception shows like this: WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:6
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meiyoula updated YARN-2776: --- Description: In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. The exception shows like this: WARN | [qtp542345580-71] | /stages/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) was:In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. > In HA mode, can't set ip but hostname to > "yarn.resourcemanager.webapp.address.*" > - > > Key: YARN-2776 > URL: https://issues.apache.org/jira/browse/YARN-2776 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: meiyoula >Priority: Critical > Attachments: YARN-2766.patch > > > In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, > I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, > but spark own page has a bug when the address turns to yarn address. > The exception shows like this: > WARN | [qtp542345580-71] | /stages/ | > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) > javax.servlet.ServletException: Could not determine the proxy server for > redirection > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.e
[jira] [Updated] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meiyoula updated YARN-2776: --- Attachment: YARN-2766.patch > In HA mode, can't set ip but hostname to > "yarn.resourcemanager.webapp.address.*" > - > > Key: YARN-2776 > URL: https://issues.apache.org/jira/browse/YARN-2776 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: meiyoula >Priority: Critical > Attachments: YARN-2766.patch > > > In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, > I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, > but spark own page has a bug when the address turns to yarn address. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
meiyoula created YARN-2776: -- Summary: In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*" Key: YARN-2776 URL: https://issues.apache.org/jira/browse/YARN-2776 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: meiyoula Priority: Critical In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, but spark own page has a bug when the address turns to yarn address. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189835#comment-14189835
]
Tsuyoshi OZAWA commented on YARN-2712:
--
Thanks Anubhav and Karhitk for the reviews.
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2775) There is no close method in NMWebServices#getLogs()
[ https://issues.apache.org/jira/browse/YARN-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] skrho updated YARN-2775: Attachment: YARN-2775_001.patch I added close method.. so fileInputStream object is not accumulated in memory.. How about that? > There is no close method in NMWebServices#getLogs() > --- > > Key: YARN-2775 > URL: https://issues.apache.org/jira/browse/YARN-2775 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: skrho >Priority: Minor > Attachments: YARN-2775_001.patch > > > If getLogs method is called, fileInputStream object is accumulated in > memory.. > Because fileinputStream object is not closed.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2775) There is no close method in NMWebServices#getLogs()
skrho created YARN-2775: --- Summary: There is no close method in NMWebServices#getLogs() Key: YARN-2775 URL: https://issues.apache.org/jira/browse/YARN-2775 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: skrho Priority: Minor If getLogs method is called, fileInputStream object is accumulated in memory.. Because fileinputStream object is not closed.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189754#comment-14189754
]
Hudson commented on YARN-2712:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6392 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6392/])
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and
headroom checks. (Tsuyoshi Ozawa via kasha) (kasha: rev
179cab81e0bde1af0cba6131f16ff127358a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2712) TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-2712:
---
Summary: TestWorkPreservingRMRestart: Augment FS tests with queue and
headroom checks (was: Adding tests about FSQueue and headroom of FairScheduler
to TestWorkPreservingRMRestart)
> TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
>
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2712) Adding tests about FSQueue and headroom of FairScheduler to TestWorkPreservingRMRestart
[
https://issues.apache.org/jira/browse/YARN-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189742#comment-14189742
]
Karthik Kambatla commented on YARN-2712:
LGTM, +1.
> Adding tests about FSQueue and headroom of FairScheduler to
> TestWorkPreservingRMRestart
> ---
>
> Key: YARN-2712
> URL: https://issues.apache.org/jira/browse/YARN-2712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-2712.1.patch, YARN-2712.2.patch
>
>
> TestWorkPreservingRMRestart#testSchedulerRecovery doesn't have test cases
> about FairScheduler partially. We should support them.
> {code}
>// Until YARN-1959 is resolved
>if (scheduler.getClass() != FairScheduler.class) {
> assertEquals(availableResources, schedulerAttempt.getHeadroom());
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2771) DistributedShell's DSConstants are badly named
[
https://issues.apache.org/jira/browse/YARN-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189724#comment-14189724
]
Hadoop QA commented on YARN-2771:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678058/YARN-2771.1.patch
against trunk revision 0126cf1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5638//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5638//console
This message is automatically generated.
> DistributedShell's DSConstants are badly named
> --
>
> Key: YARN-2771
> URL: https://issues.apache.org/jira/browse/YARN-2771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Zhijie Shen
> Attachments: YARN-2771.1.patch
>
>
> I'd rather have underscores (DISTRIBUTED_SHELL_TIMELINE_DOMAIN instead of
> DISTRIBUTEDSHELLTIMELINEDOMAIN).
> DISTRIBUTEDSHELLTIMELINEDOMAIN is added in this release, can we rename it to
> be DISTRIBUTED_SHELL_TIMELINE_DOMAIN?
> For the old envs, we can just add new envs that point to the old-one and
> deprecate the old ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
