[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[
https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755412#comment-13755412
]
Hadoop QA commented on YARN-1065:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600917/YARN-1065.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1814//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1814//console
This message is automatically generated.
> NM should provide AuxillaryService data to the container
>
>
> Key: YARN-1065
> URL: https://issues.apache.org/jira/browse/YARN-1065
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch,
> YARN-1065.4.patch, YARN-1065.5.patch
>
>
> Start container returns auxillary service data to the AM but does not provide
> the same information to the task itself. It could add that information to the
> container env with key=service_name and value=service_data. This allows the
> container to start using the service without having to depend on the AM to
> send the info to it indirectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755382#comment-13755382 ] Xuan Gong commented on YARN-1065: - Thanks for the comments. Expose the ContainerManagerImpl to containerLauncher and containerLaunch, that we can use ContainerManagerImpl.auxiliaryServices.getMetaData() to get all the service data > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch, > YARN-1065.4.patch, YARN-1065.5.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1065: Attachment: YARN-1065.5.patch > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch, > YARN-1065.4.patch, YARN-1065.5.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1090) Job does not get into Pending State
[ https://issues.apache.org/jira/browse/YARN-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755368#comment-13755368 ] Sandy Ryza commented on YARN-1090: -- Ah, ok, makes total sense > Job does not get into Pending State > --- > > Key: YARN-1090 > URL: https://issues.apache.org/jira/browse/YARN-1090 > Project: Hadoop YARN > Issue Type: Bug >Reporter: yeshavora >Assignee: Jian He > Attachments: YARN-1090.patch > > > When there is no resource available to run a job, next job should go in > pending state. RM UI should show next job as pending app and the counter for > the pending app should be incremented. > But Currently. Next job stays in ACCEPTED state and No AM has been assigned > to this job.Though Pending App count is not incremented. > Running 'job status ' shows job state=PREP. > $ mapred job -status job_1377122233385_0002 > 13/08/21 21:59:23 INFO client.RMProxy: Connecting to ResourceManager at > host1/ip1 > Job: job_1377122233385_0002 > Job File: /ABC/.staging/job_1377122233385_0002/job.xml > Job Tracking URL : http://host1:port1/application_1377122233385_0002/ > Uber job : false > Number of maps: 0 > Number of reduces: 0 > map() completion: 0.0 > reduce() completion: 0.0 > Job state: PREP > retired: false > reason for failure: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755334#comment-13755334 ] Bikas Saha commented on YARN-1065: -- We need the service data provided by the Auxillary Service and not the service data sent by the AM in the container launch context. The AM provides application part of service data. In response, the aux service provides the service part of the service data. This service part == ContainerManagerImpl.auxiliaryServices.getMetaData(). That is what we need to set into the env of the container via sanitizeEnv. It would be great if we use matching names like getServiceDataFromEnv() and setServiceDataIntoEnv(). Would be good to prepend something NM specific to the service name before setting in the env. This just helps identify that this was set by the NM. Something like NM_AUX_SERVICE_. So for service abcd, we will add NM_AUX_SERVICE_abcd into the env. > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch, > YARN-1065.4.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1116) Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts
[
https://issues.apache.org/jira/browse/YARN-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755329#comment-13755329
]
Hadoop QA commented on YARN-1116:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600898/YARN-1116.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1813//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1813//console
This message is automatically generated.
> Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts
>
>
> Key: YARN-1116
> URL: https://issues.apache.org/jira/browse/YARN-1116
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-1116.patch
>
>
> The AMRMTokens are now only saved in RMStateStore and not populated back to
> AMRMTokenSecretManager after RM restarts. This is more needed now since
> AMRMToken also becomes used in non-secure env.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755313#comment-13755313 ] Junping Du commented on YARN-771: - Thanks [~bikassaha] for review! > AMRMClient support for resource blacklisting > - > > Key: YARN-771 > URL: https://issues.apache.org/jira/browse/YARN-771 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Junping Du > Fix For: 2.1.1-beta > > Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, > YARN-771-v3.patch, YARN-771-v4.patch > > > After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1116) Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts
[ https://issues.apache.org/jira/browse/YARN-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755305#comment-13755305 ] Jian He commented on YARN-1116: --- Tested on single node cluster, now the AMRMToken can be recognized by RM after it restarts. > Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts > > > Key: YARN-1116 > URL: https://issues.apache.org/jira/browse/YARN-1116 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-1116.patch > > > The AMRMTokens are now only saved in RMStateStore and not populated back to > AMRMTokenSecretManager after RM restarts. This is more needed now since > AMRMToken also becomes used in non-secure env. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[
https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755303#comment-13755303
]
Hadoop QA commented on YARN-1065:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600896/YARN-1065.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1812//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1812//console
This message is automatically generated.
> NM should provide AuxillaryService data to the container
>
>
> Key: YARN-1065
> URL: https://issues.apache.org/jira/browse/YARN-1065
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch,
> YARN-1065.4.patch
>
>
> Start container returns auxillary service data to the AM but does not provide
> the same information to the task itself. It could add that information to the
> container env with key=service_name and value=service_data. This allows the
> container to start using the service without having to depend on the AM to
> send the info to it indirectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1116) Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts
[ https://issues.apache.org/jira/browse/YARN-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1116: -- Attachment: YARN-1116.patch Upload a patch that populates AMRMToken's password back to AMRMTokenSecretManager after RM restarts. > Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts > > > Key: YARN-1116 > URL: https://issues.apache.org/jira/browse/YARN-1116 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-1116.patch > > > The AMRMTokens are now only saved in RMStateStore and not populated back to > AMRMTokenSecretManager after RM restarts. This is more needed now since > AMRMToken also becomes used in non-secure env. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1065: Attachment: YARN-1065.4.patch Fix -1 on findBug Modify the test case to test if environment variable is forwarded using sanitizeEnv > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch, > YARN-1065.4.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1116) Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts
[ https://issues.apache.org/jira/browse/YARN-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755278#comment-13755278 ] Jian He commented on YARN-1116: --- Looks like simply populating the password inside AMRMToken back to AMRMTokenSecretManager after RM restarts is enough. > Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts > > > Key: YARN-1116 > URL: https://issues.apache.org/jira/browse/YARN-1116 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > > The AMRMTokens are now only saved in RMStateStore and not populated back to > AMRMTokenSecretManager after RM restarts. This is more needed now since > AMRMToken also becomes used in non-secure env. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1133) AMRMClient should be able to blacklist nodes, and specify them in AllocateRequest
[ https://issues.apache.org/jira/browse/YARN-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-1133. --- Resolution: Duplicate > AMRMClient should be able to blacklist nodes, and specify them in > AllocateRequest > - > > Key: YARN-1133 > URL: https://issues.apache.org/jira/browse/YARN-1133 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > After YARN-750, YARN scheduler is able to blacklist nodes when scheduling. > AMRMClient should enable this feature to the clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755265#comment-13755265 ] Hudson commented on YARN-1117: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4355 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4355/]) YARN-1117. Improved help messages for "yarn application" and "yarn node" commands. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519117) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java > Improve help message for $ yarn applications and $yarn node > --- > > Key: YARN-1117 > URL: https://issues.apache.org/jira/browse/YARN-1117 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.1.1-beta > > Attachments: YARN-1117.1.patch, YARN-1117.2.patch, YARN-1117.3.patch > > > There is standardization of help message in YARN-1080. It is nice to have > similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755253#comment-13755253 ] Vinod Kumar Vavilapalli commented on YARN-1117: --- Looks good, +1. Checking this in. > Improve help message for $ yarn applications and $yarn node > --- > > Key: YARN-1117 > URL: https://issues.apache.org/jira/browse/YARN-1117 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: YARN-1117.1.patch, YARN-1117.2.patch, YARN-1117.3.patch > > > There is standardization of help message in YARN-1080. It is nice to have > similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1133) AMRMClient should be able to blacklist nodes, and specify them in AllocateRequest
Zhijie Shen created YARN-1133: - Summary: AMRMClient should be able to blacklist nodes, and specify them in AllocateRequest Key: YARN-1133 URL: https://issues.apache.org/jira/browse/YARN-1133 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen After YARN-750, YARN scheduler is able to blacklist nodes when scheduling. AMRMClient should enable this feature to the clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[
https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755031#comment-13755031
]
Hadoop QA commented on YARN-696:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600827/YARN-696.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1807//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1807//console
This message is automatically generated.
> Enable multiple states to to be specified in Resource Manager apps REST call
>
>
> Key: YARN-696
> URL: https://issues.apache.org/jira/browse/YARN-696
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.0.4-alpha
>Reporter: Trevor Lorimer
>Assignee: Trevor Lorimer
> Attachments: YARN-696.diff, YARN-696.diff
>
>
> Within the YARN Resource Manager REST API the GET call which returns all
> Applications can be filtered by a single State query parameter (http:// http address:port>/ws/v1/cluster/apps).
> There are 8 possible states (New, Submitted, Accepted, Running, Finishing,
> Finished, Failed, Killed), if no state parameter is specified all states are
> returned, however if a sub-set of states is required then multiple REST calls
> are required (max. of 7).
> The proposal is to be able to specify multiple states in a single REST call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755236#comment-13755236 ] Timothy St. Clair commented on YARN-888: Our current list of JIRA's can be found here: https://fedoraproject.org/wiki/Changes/Hadoop#Upstream_patch_tracking > clean up POM dependencies > - > > Key: YARN-888 > URL: https://issues.apache.org/jira/browse/YARN-888 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Alejandro Abdelnur >Assignee: Roman Shaposhnik > > Intermediate 'pom' modules define dependencies inherited by leaf modules. > This is causing issues in intellij IDE. > We should normalize the leaf modules like in common, hdfs and tools where all > dependencies are defined in each leaf module and the intermediate 'pom' > module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755235#comment-13755235 ] Hudson commented on YARN-771: - SUCCESS: Integrated in Hadoop-trunk-Commit #4354 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4354/]) YARN-771. AMRMClient support for resource blacklisting (Junping Du via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519107) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceBlacklistRequestPBImpl.java > AMRMClient support for resource blacklisting > - > > Key: YARN-771 > URL: https://issues.apache.org/jira/browse/YARN-771 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Junping Du > Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, > YARN-771-v3.patch, YARN-771-v4.patch > > > After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[
https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755228#comment-13755228
]
Hadoop QA commented on YARN-1065:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600867/YARN-1065.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1811//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1811//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1811//console
This message is automatically generated.
> NM should provide AuxillaryService data to the container
>
>
> Key: YARN-1065
> URL: https://issues.apache.org/jira/browse/YARN-1065
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch
>
>
> Start container returns auxillary service data to the AM but does not provide
> the same information to the task itself. It could add that information to the
> container env with key=service_name and value=service_data. This allows the
> container to start using the service without having to depend on the AM to
> send the info to it indirectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1090) Job does not get into Pending State
[ https://issues.apache.org/jira/browse/YARN-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755227#comment-13755227 ] Jian He commented on YARN-1090: --- bq. My understanding was that an application is "Pending" if no AM has yet been allocated to it Yes, that's correct. What I meant here is about the UI of the Application Queues usage inside which: 'Num Pending Applications' doesn't mean that no AM has been allocated it, and propose to rename it to 'Num non-schedulable applications'. > Job does not get into Pending State > --- > > Key: YARN-1090 > URL: https://issues.apache.org/jira/browse/YARN-1090 > Project: Hadoop YARN > Issue Type: Bug >Reporter: yeshavora >Assignee: Jian He > Attachments: YARN-1090.patch > > > When there is no resource available to run a job, next job should go in > pending state. RM UI should show next job as pending app and the counter for > the pending app should be incremented. > But Currently. Next job stays in ACCEPTED state and No AM has been assigned > to this job.Though Pending App count is not incremented. > Running 'job status ' shows job state=PREP. > $ mapred job -status job_1377122233385_0002 > 13/08/21 21:59:23 INFO client.RMProxy: Connecting to ResourceManager at > host1/ip1 > Job: job_1377122233385_0002 > Job File: /ABC/.staging/job_1377122233385_0002/job.xml > Job Tracking URL : http://host1:port1/application_1377122233385_0002/ > Uber job : false > Number of maps: 0 > Number of reduces: 0 > map() completion: 0.0 > reduce() completion: 0.0 > Job state: PREP > retired: false > reason for failure: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1090) Job does not get into Pending State
[ https://issues.apache.org/jira/browse/YARN-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755219#comment-13755219 ] Sandy Ryza commented on YARN-1090: -- My understanding was that an application is "Pending" if no AM has yet been allocated to it. Can you go a little more into what the issue is with this definition? Also, whatever we decide, is there somewhere where we can document exactly what these mean? > Job does not get into Pending State > --- > > Key: YARN-1090 > URL: https://issues.apache.org/jira/browse/YARN-1090 > Project: Hadoop YARN > Issue Type: Bug >Reporter: yeshavora >Assignee: Jian He > Attachments: YARN-1090.patch > > > When there is no resource available to run a job, next job should go in > pending state. RM UI should show next job as pending app and the counter for > the pending app should be incremented. > But Currently. Next job stays in ACCEPTED state and No AM has been assigned > to this job.Though Pending App count is not incremented. > Running 'job status ' shows job state=PREP. > $ mapred job -status job_1377122233385_0002 > 13/08/21 21:59:23 INFO client.RMProxy: Connecting to ResourceManager at > host1/ip1 > Job: job_1377122233385_0002 > Job File: /ABC/.staging/job_1377122233385_0002/job.xml > Job Tracking URL : http://host1:port1/application_1377122233385_0002/ > Uber job : false > Number of maps: 0 > Number of reduces: 0 > map() completion: 0.0 > reduce() completion: 0.0 > Job state: PREP > retired: false > reason for failure: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1132) QueueMetrics.java has wrong comments
[
https://issues.apache.org/jira/browse/YARN-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755208#comment-13755208
]
Jian He commented on YARN-1132:
---
Hi [~ajisakaa], thanks for reporting this.
This will be fixed as part of YARN-1090, thx!
> QueueMetrics.java has wrong comments
>
>
> Key: YARN-1132
> URL: https://issues.apache.org/jira/browse/YARN-1132
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.1.0-beta
>Reporter: Akira AJISAKA
>Priority: Minor
> Labels: newbie
>
> I found o.a.h.yarn.server.resourcemanager.scheduler.QueueMetrics.java has
> wrong comments
> {code}
> @Metric("# of reserved memory in MB") MutableGaugeInt reservedMB;
> @Metric("# of active users") MutableGaugeInt activeApplications;
> {code}
> they should be fixed as follows:
> {code}
> @Metric("Reserved memory in MB") MutableGaugeInt reservedMB;
> @Metric("# of active applications") MutableGaugeInt activeApplications;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1132) QueueMetrics.java has wrong comments
Akira AJISAKA created YARN-1132:
---
Summary: QueueMetrics.java has wrong comments
Key: YARN-1132
URL: https://issues.apache.org/jira/browse/YARN-1132
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Priority: Minor
I found o.a.h.yarn.server.resourcemanager.scheduler.QueueMetrics.java has wrong
comments
{code}
@Metric("# of reserved memory in MB") MutableGaugeInt reservedMB;
@Metric("# of active users") MutableGaugeInt activeApplications;
{code}
they should be fixed as follows:
{code}
@Metric("Reserved memory in MB") MutableGaugeInt reservedMB;
@Metric("# of active applications") MutableGaugeInt activeApplications;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1131) $ yarn logs should return a message log aggregation is during progress if YARN application is running
Tassapol Athiapinya created YARN-1131:
-
Summary: $ yarn logs should return a message log aggregation is
during progress if YARN application is running
Key: YARN-1131
URL: https://issues.apache.org/jira/browse/YARN-1131
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Priority: Minor
Fix For: 2.1.1-beta
In the case when log aggregation is enabled, if a user submits MapReduce job
and runs $ yarn logs -applicationId while the YARN application is
running, the command will return no message and return user back to shell. It
is nice to tell the user that log aggregation is in progress.
{code}
-bash-4.1$ /usr/bin/yarn logs -applicationId application_1377900193583_0002
-bash-4.1$
{code}
At the same time, if invalid application ID is given, YARN CLI should say that
the application ID is incorrect rather than throwing NoSuchElementException.
{code}
$ /usr/bin/yarn logs -applicationId application_0
Exception in thread "main" java.util.NoSuchElementException
at com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:124)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:119)
at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:110)
at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:255)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Description:
When userlog limit is set with something like this:
{code}
mapred.userlog.limit.kb
2048
The maximum size of user-logs of each task in KB. 0 disables the
cap.
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not
flush to file until the container shutdown and logmanager close all appenders.
If user likes to see the log during task execution, it doesn't support it.
Will propose a patch to add a flush mechanism and also flush the log when task
is done.
was:
When userlog limit is set with something like this:
{code}
mapred.userlog.limit.kb
2048
The maximum size of user-logs of each task in KB. 0 disables the
cap.
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) will not write to log until
the container shutdown and logmanager close all appenders. If user likes to see
the log during task execution, it doesn't support it.
Will propose a patch to add a flush mechanism and also flush the log when task
is done.
> Improve the log flushing for tasks when mapred.userlog.limit.kb is set
> --
>
> Key: YARN-1130
> URL: https://issues.apache.org/jira/browse/YARN-1130
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.5-alpha
>Reporter: Paul Han
>
> When userlog limit is set with something like this:
> {code}
>
> mapred.userlog.limit.kb
> 2048
> The maximum size of user-logs of each task in KB. 0 disables the
> cap.
>
>
> {code}
> the log entry will be truncated randomly for the jobs.
> The log size is left between 1.2MB to 1.6MB.
> Since the log is already limited, avoid the log truncation is crucial for
> user.
> The other issue with the current
> impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
> not flush to file until the container shutdown and logmanager close all
> appenders. If user likes to see the log during task execution, it doesn't
> support it.
> Will propose a patch to add a flush mechanism and also flush the log when
> task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
Paul Han created YARN-1130:
--
Summary: Improve the log flushing for tasks when
mapred.userlog.limit.kb is set
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
When userlog limit is set with something like this:
{code}
mapred.userlog.limit.kb
2048
The maximum size of user-logs of each task in KB. 0 disables the
cap.
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) will not write to log until
the container shutdown and logmanager close all appenders. If user likes to see
the log during task execution, it doesn't support it.
Will propose a patch to add a flush mechanism and also flush the log when task
is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1065: Attachment: YARN-1065.3.patch > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch, YARN-1065.3.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1098) Separate out RM services into "Always On" and "Active"
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1098: --- Attachment: yarn-1098-1.patch Uploading a patch along the lines of my previous commit. The only change is that AdminService is also in the Active services. We should move it to "Always On" once we make it HA-aware; currently, some of the functions don't apply to Standby RM. Also, the patch depends on HADOOP-9918. > Separate out RM services into "Always On" and "Active" > -- > > Key: YARN-1098 > URL: https://issues.apache.org/jira/browse/YARN-1098 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1098-1.patch, yarn-1098-approach.patch, > yarn-1098-approach.patch > > > From discussion on YARN-1027, it makes sense to separate out services that > are stateful and stateless. The stateless services can run perennially > irrespective of whether the RM is in Active/Standby state, while the stateful > services need to be started on transitionToActive() and completely shutdown > on transitionToStandby(). > The external-facing stateless services should respond to the client/AM/NM > requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1127) reservation exchange and excess reservation is not working for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755110#comment-13755110 ] Bikas Saha commented on YARN-1127: -- Then please clarify this in the description or comment. Otherwise it looked like an exact duplicate. So the purpose of this jira is to fix the following situation. 1) NM1 has 2048 capacity in total but only 512 is free. A reservation of 1024 is placed on it 2) NM2 now reports 1024 free space. At this point, the above reservation should be removed from NM1 and container should be assigned to NM2. Step 2 is not happening and this jira intends to fix it. > reservation exchange and excess reservation is not working for capacity > scheduler > - > > Key: YARN-1127 > URL: https://issues.apache.org/jira/browse/YARN-1127 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. > * We support a notion where if say we have 5 nodes with 4 AM and all node > managers have 8GB each and AM 2 GB each. Each AM is requesting 8GB each. Now > to avoid deadlock AM will make an extra reservation. By doing this we would > never hit the deadlock situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[
https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755162#comment-13755162
]
Zhijie Shen commented on YARN-696:
--
The patch is almost good. Here's some minor comments:
1. To compare the strings with case ingored, is it better to use
{code}
+appStates.add(state.trim().toLowerCase());
{code}
and
{code}
+ if (appState.contains(rmapp.getState().toString().toLowerCase())) {
{code}
It may not have much performance gain given a small string collection, but it
should make code more concise.
2. It would be more clear if the test can verify one app is ACCEPTED and the
other is KILLED.
{code}
+assertTrue("no states equal to KILLED",
+(array.getJSONObject(0).getString("state").equals("KILLED")) ||
+(array.getJSONObject(1).getString("state").equals("KILLED")));
+assertTrue("no states equal to ACCEPTED",
+(array.getJSONObject(0).getString("state").equals("ACCEPTED")) ||
+(array.getJSONObject(1).getString("state").equals("ACCEPTED")));
{code}
> Enable multiple states to to be specified in Resource Manager apps REST call
>
>
> Key: YARN-696
> URL: https://issues.apache.org/jira/browse/YARN-696
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.0.4-alpha
>Reporter: Trevor Lorimer
>Assignee: Trevor Lorimer
> Attachments: YARN-696.diff, YARN-696.diff
>
>
> Within the YARN Resource Manager REST API the GET call which returns all
> Applications can be filtered by a single State query parameter (http:// http address:port>/ws/v1/cluster/apps).
> There are 8 possible states (New, Submitted, Accepted, Running, Finishing,
> Finished, Failed, Killed), if no state parameter is specified all states are
> returned, however if a sub-set of states is required then multiple REST calls
> are required (max. of 7).
> The proposal is to be able to specify multiple states in a single REST call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[
https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755063#comment-13755063
]
Hadoop QA commented on YARN-1065:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600840/YARN-1065.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1809//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1809//console
This message is automatically generated.
> NM should provide AuxillaryService data to the container
>
>
> Key: YARN-1065
> URL: https://issues.apache.org/jira/browse/YARN-1065
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-1065.1.patch, YARN-1065.2.patch
>
>
> Start container returns auxillary service data to the AM but does not provide
> the same information to the task itself. It could add that information to the
> container env with key=service_name and value=service_data. This allows the
> container to start using the service without having to depend on the AM to
> send the info to it indirectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1129) Job hungs when any node is blacklisted after RMrestart
yeshavora created YARN-1129: --- Summary: Job hungs when any node is blacklisted after RMrestart Key: YARN-1129 URL: https://issues.apache.org/jira/browse/YARN-1129 Project: Hadoop YARN Issue Type: Bug Reporter: yeshavora When RM restarted, if during restart one NM went bad (bad disk), NM got blacklisted by AM and RM keeps giving the containers on the same node even though AM doesn't want it there. Need to change AM to specifically blacklist node in the RM requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-896) Roll up for long lived YARN
[ https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754812#comment-13754812 ] Jason Lowe commented on YARN-896: - bq. Chris, feel free to file a JIRA for rolling of stdout and stderr and we can look into what it will take to support that properly. [~ste...@apache.org] recently filed YARN-1104 as a subtask of this JIRA which covers the NM rolling stdout/stderr. We can transmute that JIRA into whatever ends up rolling the logs if it's not the NM. > Roll up for long lived YARN > --- > > Key: YARN-896 > URL: https://issues.apache.org/jira/browse/YARN-896 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Robert Joseph Evans > > YARN is intended to be general purpose, but it is missing some features to be > able to truly support long lived applications and long lived containers. > This ticket is intended to > # discuss what is needed to support long lived processes > # track the resulting JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1129) Job hungs when any node is blacklisted after RMrestart
[ https://issues.apache.org/jira/browse/YARN-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1129: -- Assignee: Zhijie Shen > Job hungs when any node is blacklisted after RMrestart > -- > > Key: YARN-1129 > URL: https://issues.apache.org/jira/browse/YARN-1129 > Project: Hadoop YARN > Issue Type: Bug >Reporter: yeshavora >Assignee: Zhijie Shen > > When RM restarted, if during restart one NM went bad (bad disk), NM got > blacklisted by AM and RM keeps giving the containers on the same node even > though AM doesn't want it there. > Need to change AM to specifically blacklist node in the RM requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-957) Capacity Scheduler tries to reserve the memory more than what node manager reports.
[
https://issues.apache.org/jira/browse/YARN-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755115#comment-13755115
]
Hadoop QA commented on YARN-957:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600849/YARN-957-20130830.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1810//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1810//console
This message is automatically generated.
> Capacity Scheduler tries to reserve the memory more than what node manager
> reports.
> ---
>
> Key: YARN-957
> URL: https://issues.apache.org/jira/browse/YARN-957
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Attachments: YARN-957-20130730.1.patch, YARN-957-20130730.2.patch,
> YARN-957-20130730.3.patch, YARN-957-20130731.1.patch,
> YARN-957-20130830.1.patch
>
>
> I have 2 node managers.
> * one with 1024 MB memory.(nm1)
> * second with 2048 MB memory.(nm2)
> I am submitting simple map reduce application with 1 mapper and one reducer
> with 1024mb each. The steps to reproduce this are
> * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's
> heartbeat doesn't reach RM first).
> * now submit application. As soon as it receives first node's (nm1) heartbeat
> it will try to reserve memory for AM-container (2048MB). However it has only
> 1024MB of memory.
> * now start nm2 with 2048 MB memory.
> It hangs forever... Ideally this has two potential issues.
> * It should not try to reserve memory on a node manager which is never going
> to give requested memory. i.e. Current max capability of node manager is
> 1024MB but 2048MB is reserved on it. But it still does that.
> * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available
> memory. In this case if the original request was made without any locality
> then scheduler should unreserve memory on nm1 and allocate requested 2048MB
> container on nm2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1127) reservation exchange and excess reservation is not working for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755103#comment-13755103 ] Omkar Vinit Joshi commented on YARN-1127: - No as per Arun I am separating out issues which are causing this failure. * YARN-957 :- Fix if container is getting reserved on a node manager which exceeds its memory. * this jira :- Ideally the switch should have taken place from one to other node manager if another node manager has sufficient memory. However that did not happen. This must have occurred either because excess reservation did not work or reservation exchange did not occur. We need to find the root cause and fix this. > reservation exchange and excess reservation is not working for capacity > scheduler > - > > Key: YARN-1127 > URL: https://issues.apache.org/jira/browse/YARN-1127 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. > * We support a notion where if say we have 5 nodes with 4 AM and all node > managers have 8GB each and AM 2 GB each. Each AM is requesting 8GB each. Now > to avoid deadlock AM will make an extra reservation. By doing this we would > never hit the deadlock situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1128) FifoPolicy.computeShares throws NPE on empty list of Schedulables
[
https://issues.apache.org/jira/browse/YARN-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla reassigned YARN-1128:
--
Assignee: Karthik Kambatla
> FifoPolicy.computeShares throws NPE on empty list of Schedulables
> -
>
> Key: YARN-1128
> URL: https://issues.apache.org/jira/browse/YARN-1128
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.1.0-beta
>Reporter: Sandy Ryza
>Assignee: Karthik Kambatla
>
> FifoPolicy gives all of a queue's share to the earliest-scheduled application.
> {code}
> Schedulable earliest = null;
> for (Schedulable schedulable : schedulables) {
> if (earliest == null ||
> schedulable.getStartTime() < earliest.getStartTime()) {
> earliest = schedulable;
> }
> }
> earliest.setFairShare(Resources.clone(totalResources));
> {code}
> If the queue has no schedulables in it, earliest will be left null, leading
> to an NPE on the last line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: YARN-696.diff Okay i'll take your word for it, uploading latest diff. The unit tests use 2 different applications with states of ACCEPTED and KILLED. > Enable multiple states to to be specified in Resource Manager apps REST call > > > Key: YARN-696 > URL: https://issues.apache.org/jira/browse/YARN-696 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.0.4-alpha >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: YARN-696.diff, YARN-696.diff > > > Within the YARN Resource Manager REST API the GET call which returns all > Applications can be filtered by a single State query parameter (http:// http address:port>/ws/v1/cluster/apps). > There are 8 possible states (New, Submitted, Accepted, Running, Finishing, > Finished, Failed, Killed), if no state parameter is specified all states are > returned, however if a sub-set of states is required then multiple REST calls > are required (max. of 7). > The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1127) reservation exchange and excess reservation is not working for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755095#comment-13755095 ] Bikas Saha commented on YARN-1127: -- How is this different from YARN-957 > reservation exchange and excess reservation is not working for capacity > scheduler > - > > Key: YARN-1127 > URL: https://issues.apache.org/jira/browse/YARN-1127 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. > * We support a notion where if say we have 5 nodes with 4 AM and all node > managers have 8GB each and AM 2 GB each. Each AM is requesting 8GB each. Now > to avoid deadlock AM will make an extra reservation. By doing this we would > never hit the deadlock situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1127) reservation exchange and excess reservation is not working for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755090#comment-13755090 ] Bikas Saha commented on YARN-1127: -- Isnt this similar to a jira opened by you already? The issue being that the scheduler puts a reservation on a node whose total capacity is smaller than the reservation resource size. In this case, nm1 has capacity=1024 but the scheduler is putting a reservation of 2048 on it and that can never be satisfied. So it does not make sense to make that reservation at all. > reservation exchange and excess reservation is not working for capacity > scheduler > - > > Key: YARN-1127 > URL: https://issues.apache.org/jira/browse/YARN-1127 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. > * We support a notion where if say we have 5 nodes with 4 AM and all node > managers have 8GB each and AM 2 GB each. Each AM is requesting 8GB each. Now > to avoid deadlock AM will make an extra reservation. By doing this we would > never hit the deadlock situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-957) Capacity Scheduler tries to reserve the memory more than what node manager reports.
[ https://issues.apache.org/jira/browse/YARN-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-957: --- Attachment: YARN-957-20130830.1.patch > Capacity Scheduler tries to reserve the memory more than what node manager > reports. > --- > > Key: YARN-957 > URL: https://issues.apache.org/jira/browse/YARN-957 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: YARN-957-20130730.1.patch, YARN-957-20130730.2.patch, > YARN-957-20130730.3.patch, YARN-957-20130731.1.patch, > YARN-957-20130830.1.patch > > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * It should not try to reserve memory on a node manager which is never going > to give requested memory. i.e. Current max capability of node manager is > 1024MB but 2048MB is reserved on it. But it still does that. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-957) Capacity Scheduler tries to reserve the memory more than what node manager reports.
[ https://issues.apache.org/jira/browse/YARN-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755089#comment-13755089 ] Omkar Vinit Joshi commented on YARN-957: uploading a patch which only fixes excess memory reservation issue. > Capacity Scheduler tries to reserve the memory more than what node manager > reports. > --- > > Key: YARN-957 > URL: https://issues.apache.org/jira/browse/YARN-957 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: YARN-957-20130730.1.patch, YARN-957-20130730.2.patch, > YARN-957-20130730.3.patch, YARN-957-20130731.1.patch > > > I have 2 node managers. > * one with 1024 MB memory.(nm1) > * second with 2048 MB memory.(nm2) > I am submitting simple map reduce application with 1 mapper and one reducer > with 1024mb each. The steps to reproduce this are > * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's > heartbeat doesn't reach RM first). > * now submit application. As soon as it receives first node's (nm1) heartbeat > it will try to reserve memory for AM-container (2048MB). However it has only > 1024MB of memory. > * now start nm2 with 2048 MB memory. > It hangs forever... Ideally this has two potential issues. > * It should not try to reserve memory on a node manager which is never going > to give requested memory. i.e. Current max capability of node manager is > 1024MB but 2048MB is reserved on it. But it still does that. > * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available > memory. In this case if the original request was made without any locality > then scheduler should unreserve memory on nm1 and allocate requested 2048MB > container on nm2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755088#comment-13755088 ] Bikas Saha commented on YARN-1065: -- There seems to be some misunderstanding here. The NM already provides service metadata in startContainerResponse to the AM. The main issue is that when the NM actually launches the process of that container then that process has no way to find information about the auxillary services mentioned in the startContainerRequest for that container. So the desired solution is the following 1) StartContainerRequest sends request for auxillary services to NM (this already happens today) 2) NM sends back auxillary service metadata (say Map) in the StartContainerResponse (this happens today) 3) When NM launches the process for this container then it should put the service metadata (Map) in the container environment. Eg. add to environment key=FOO_NM_SERVICE_KEY with value=base64String(ByteBuffer). 4) There should be some helper API to enable the process in the container to retrieve those keys and value. Process uses api, say, Helper.getServiceMetadata(FOO) which looks up FOO_NM_SERVICE_KEY in the env and converts the base64String value back to ByteBuffer. NM could also user Helper.set(FOO, ByteBuffer, env) that sets key=FOO_NM_SERVICE_KEY and base64String(ByteBuffer) into the env. Using helper API will keep both sides consistent. > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1104) NMs to support rolling logs of stdout & stderr
[
https://issues.apache.org/jira/browse/YARN-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755087#comment-13755087
]
Steve Loughran commented on YARN-1104:
--
Currently most output setups are being done in the command line itself with a
>/out.txt 2>/err.txt
we should allow apps to ask for the NM set up the logging, which could then be
a direct capture from the process streams and rolling logging into the
directories, maybe even allow for other options. Routing it via log4j isn't
ideal because it does tend to add extra noise:
{code}
d=0}
2013-08-29 17:07:40,594 [AMRM Callback Handler Thread] DEBUG
yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:getProgress(1280)) -
Heartbeat, percentage =0.5
2013-08-29 17:07:40,900 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:40 DEBUG client.ClientScanner:
Finished region={ENCODED => 92bbefc5114a871444d6665dbddd7fda, NAME =>
'hbase:namesp
2013-08-29 17:07:40,900 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) -
ace,,1377821255366.92bbefc5114a871444d6665dbddd7fda.', STARTKEY => '', ENDKEY
=> ''}
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:40 INFO
zookeeper.RecoverableZooKeeper: Node
/yarnapps_hoya_stevel_TestClusterFlex2DownTo1/namespace/default alrea
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - dy exists and this is not a retry
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG
hbase.ZKNamespaceManager: Updating namespace cache from node default with data:
\x0A\x07default
2013-08-29 17:07:41,101 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG
hbase.ZKNamespaceManager: Updating namespace cache from node hbase with data:
\x0A\x05hbase
2013-08-29 17:07:41,429 [IPC Server handler 1 on 52037] INFO
yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:stopCluster(1306)) -
HoyaAppMasterApi.stopCluster()
2013-08-29 17:07:41,429 [IPC Server handler 1 on 52037] INFO
yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:signalAMComplete(709)) -
Triggering shutdown of the AM: stopCluster() invoked
2013-08-29 17:07:41,502 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 INFO
zookeeper.RecoverableZooKeeper: Node
/yarnapps_hoya_stevel_TestClusterFlex2DownTo1/namespace/hbase already
2013-08-29 17:07:41,503 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - exists and this is not a retry
2013-08-29 17:07:41,597 [AMRM Callback Handler Thread] DEBUG
yarn.appmaster.HoyaAppMaster (HoyaAppMaster.java:getProgress(1280)) -
Heartbeat, percentage =0.5
2013-08-29 17:07:41,904 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - 13/08/29 17:07:41 DEBUG client.ClientScanner:
Finished region={ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY =>
'', EN
2013-08-29 17:07:41,904 [Thread-12] WARN hoya.exec.RunLongLivedApp
(RunLongLivedApp.java:run(351)) - DKEY => ''}
{code}
Perhaps we could have a log4j formatter that does nothing at all -just prints
the output direct -then route it to a specific log roll just for the outputs
> NMs to support rolling logs of stdout & stderr
> --
>
> Key: YARN-1104
> URL: https://issues.apache.org/jira/browse/YARN-1104
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.1.0-beta
>Reporter: Steve Loughran
>
> Currently NMs stream the stdout and stderr streams of a container to a file.
> For longer lived processes those files need to be rotated so that the log
> doesn't overflow
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1128) FifoPolicy.computeShares throws NPE on empty list of Schedulables
Sandy Ryza created YARN-1128:
Summary: FifoPolicy.computeShares throws NPE on empty list of
Schedulables
Key: YARN-1128
URL: https://issues.apache.org/jira/browse/YARN-1128
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
FifoPolicy gives all of a queue's share to the earliest-scheduled application.
{code}
Schedulable earliest = null;
for (Schedulable schedulable : schedulables) {
if (earliest == null ||
schedulable.getStartTime() < earliest.getStartTime()) {
earliest = schedulable;
}
}
earliest.setFairShare(Resources.clone(totalResources));
{code}
If the queue has no schedulables in it, earliest will be left null, leading to
an NPE on the last line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1120) Make ApplicationConstants.Environment.USER definition OS neutral
[
https://issues.apache.org/jira/browse/YARN-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755055#comment-13755055
]
Hadoop QA commented on YARN-1120:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600495/YARN-1120.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1808//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1808//console
This message is automatically generated.
> Make ApplicationConstants.Environment.USER definition OS neutral
>
>
> Key: YARN-1120
> URL: https://issues.apache.org/jira/browse/YARN-1120
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Minor
> Attachments: YARN-1120.patch
>
>
> In YARN-557, we added some code to make {{
> ApplicationConstants.Environment.USER}} has OS-specific definition in order
> to fix the unit test TestUnmanagedAMLauncher. In YARN-571, the relevant test
> code was corrected. In YARN-602, we actually will explicitly set the
> environment variables for the child containers. With these changes, I think
> we can revert the YARN-557 change to make {{
> ApplicationConstants.Environment.USER}} OS neutral. The main benefit is that
> we can use the same method over the Enum constants. This should also fix the
> TestContainerLaunch#testContainerEnvVariables failure on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-707: Attachment: YARN-707-20130830.branch-0.23.txt Patch for branch-0.23. [~vinodkv] could you take a peek at it? All of the tests in mapreduce-client-app, yarn-common, and yarn-server-resourcemanager pass after this change, and I manually tested it on a secure cluster with the branch-0.23 patch from MAPREDUCE-5475 to verify ACLs can be implemented properly. > Add user info in the YARN ClientToken > - > > Key: YARN-707 > URL: https://issues.apache.org/jira/browse/YARN-707 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Jason Lowe >Priority: Blocker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, > YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt, > YARN-707-20130830.branch-0.23.txt > > > If user info is present in the client token then it can be used to do limited > authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1127) reservation exchange and excess reservation is not working for capacity scheduler
Omkar Vinit Joshi created YARN-1127: --- Summary: reservation exchange and excess reservation is not working for capacity scheduler Key: YARN-1127 URL: https://issues.apache.org/jira/browse/YARN-1127 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Priority: Blocker I have 2 node managers. * one with 1024 MB memory.(nm1) * second with 2048 MB memory.(nm2) I am submitting simple map reduce application with 1 mapper and one reducer with 1024mb each. The steps to reproduce this are * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's heartbeat doesn't reach RM first). * now submit application. As soon as it receives first node's (nm1) heartbeat it will try to reserve memory for AM-container (2048MB). However it has only 1024MB of memory. * now start nm2 with 2048 MB memory. It hangs forever... Ideally this has two potential issues. * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available memory. In this case if the original request was made without any locality then scheduler should unreserve memory on nm1 and allocate requested 2048MB container on nm2. * We support a notion where if say we have 5 nodes with 4 AM and all node managers have 8GB each and AM 2 GB each. Each AM is requesting 8GB each. Now to avoid deadlock AM will make an extra reservation. By doing this we would never hit the deadlock situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1065: Attachment: YARN-1065.2.patch > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755048#comment-13755048 ] Xuan Gong commented on YARN-1065: - [~bikassaha] Please ignore the YARN-1065.1.patch, please take a look at YARN-1065.2.patch. > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch, YARN-1065.2.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container
[ https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1065: Attachment: YARN-1065.1.patch > NM should provide AuxillaryService data to the container > > > Key: YARN-1065 > URL: https://issues.apache.org/jira/browse/YARN-1065 > Project: Hadoop YARN > Issue Type: Task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1065.1.patch > > > Start container returns auxillary service data to the AM but does not provide > the same information to the task itself. It could add that information to the > container env with key=service_name and value=service_data. This allows the > container to start using the service without having to depend on the AM to > send the info to it indirectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-896) Roll up for long lived YARN
[ https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754787#comment-13754787 ] Robert Joseph Evans commented on YARN-896: -- I agree that providing a good way handle stdout and stderr is important. I don't know if I want the NM to be doing this for us though, but that is an implementation detail that we can talk about on the follow up JIRA. Chris, feel free to file a JIRA for rolling of stdout and stderr and we can look into what it will take to support that properly. > Roll up for long lived YARN > --- > > Key: YARN-896 > URL: https://issues.apache.org/jira/browse/YARN-896 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Robert Joseph Evans > > YARN is intended to be general purpose, but it is missing some features to be > able to truly support long lived applications and long lived containers. > This ticket is intended to > # discuss what is needed to support long lived processes > # track the resulting JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754741#comment-13754741 ] Hudson commented on YARN-707: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1534 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1534/]) YARN-707. Added user information also in the YARN ClientToken so that AMs can implement authorization based on incoming users. Contributed by Jason Lowe. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518868) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java > Add user info in the YARN ClientToken > - > > Key: YARN-707 > URL: https://issues.apache.org/jira/browse/YARN-707 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Jason Lowe >Priority: Blocker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, > YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt > > > If user info is present in the client token then it can be used to do limited > authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1080) Improve help message for $ yarn logs
[
https://issues.apache.org/jira/browse/YARN-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754747#comment-13754747
]
Hudson commented on YARN-1080:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1534 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1534/])
YARN-1080. Improved help message for "yarn logs" command. Contributed by Xuan
Gong. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518731)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestLogDumper.java
> Improve help message for $ yarn logs
>
>
> Key: YARN-1080
> URL: https://issues.apache.org/jira/browse/YARN-1080
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Fix For: 2.1.1-beta
>
> Attachments: YARN-1080.1.patch, YARN-1080.2.patch
>
>
> There are 2 parts I am proposing in this jira. They can be fixed together in
> one patch.
> 1. Standardize help message for required parameter of $ yarn logs
> YARN CLI has a command "logs" ($ yarn logs). The command always requires a
> parameter of "-applicationId ". However, help message of the command
> does not make it clear. It lists -applicationId as optional parameter. If I
> don't set it, YARN CLI will complain this is missing. It is better to use
> standard required notation used in other Linux command for help message. Any
> user familiar to the command can understand that this parameter is needed
> more easily.
> {code:title=current help message}
> -bash-4.1$ yarn logs
> usage: general options are:
> -applicationIdApplicationId (required)
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> {code:title=proposed help message}
> -bash-4.1$ yarn logs
> usage: yarn logs -applicationId [OPTIONS]
> general options are:
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> 2. Add description for help command. As far as I know, a user cannot get logs
> for running job. Since I spent some time trying to get logs of running
> applications, it should be nice to say this in command description.
> {code:title=proposed help}
> Retrieve logs for completed/killed YARN application
> usage: general options are...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754665#comment-13754665 ] Hudson commented on YARN-707: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1507 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1507/]) YARN-707. Added user information also in the YARN ClientToken so that AMs can implement authorization based on incoming users. Contributed by Jason Lowe. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518868) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java > Add user info in the YARN ClientToken > - > > Key: YARN-707 > URL: https://issues.apache.org/jira/browse/YARN-707 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Jason Lowe >Priority: Blocker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, > YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt > > > If user info is present in the client token then it can be used to do limited > authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1080) Improve help message for $ yarn logs
[
https://issues.apache.org/jira/browse/YARN-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754671#comment-13754671
]
Hudson commented on YARN-1080:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1507 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1507/])
YARN-1080. Improved help message for "yarn logs" command. Contributed by Xuan
Gong. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518731)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestLogDumper.java
> Improve help message for $ yarn logs
>
>
> Key: YARN-1080
> URL: https://issues.apache.org/jira/browse/YARN-1080
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Fix For: 2.1.1-beta
>
> Attachments: YARN-1080.1.patch, YARN-1080.2.patch
>
>
> There are 2 parts I am proposing in this jira. They can be fixed together in
> one patch.
> 1. Standardize help message for required parameter of $ yarn logs
> YARN CLI has a command "logs" ($ yarn logs). The command always requires a
> parameter of "-applicationId ". However, help message of the command
> does not make it clear. It lists -applicationId as optional parameter. If I
> don't set it, YARN CLI will complain this is missing. It is better to use
> standard required notation used in other Linux command for help message. Any
> user familiar to the command can understand that this parameter is needed
> more easily.
> {code:title=current help message}
> -bash-4.1$ yarn logs
> usage: general options are:
> -applicationIdApplicationId (required)
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> {code:title=proposed help message}
> -bash-4.1$ yarn logs
> usage: yarn logs -applicationId [OPTIONS]
> general options are:
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> 2. Add description for help command. As far as I know, a user cannot get logs
> for running job. Since I spent some time trying to get logs of running
> applications, it should be nice to say this in command description.
> {code:title=proposed help}
> Retrieve logs for completed/killed YARN application
> usage: general options are...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1080) Improve help message for $ yarn logs
[
https://issues.apache.org/jira/browse/YARN-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754586#comment-13754586
]
Hudson commented on YARN-1080:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #317 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/317/])
YARN-1080. Improved help message for "yarn logs" command. Contributed by Xuan
Gong. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518731)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestLogDumper.java
> Improve help message for $ yarn logs
>
>
> Key: YARN-1080
> URL: https://issues.apache.org/jira/browse/YARN-1080
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Reporter: Tassapol Athiapinya
>Assignee: Xuan Gong
> Fix For: 2.1.1-beta
>
> Attachments: YARN-1080.1.patch, YARN-1080.2.patch
>
>
> There are 2 parts I am proposing in this jira. They can be fixed together in
> one patch.
> 1. Standardize help message for required parameter of $ yarn logs
> YARN CLI has a command "logs" ($ yarn logs). The command always requires a
> parameter of "-applicationId ". However, help message of the command
> does not make it clear. It lists -applicationId as optional parameter. If I
> don't set it, YARN CLI will complain this is missing. It is better to use
> standard required notation used in other Linux command for help message. Any
> user familiar to the command can understand that this parameter is needed
> more easily.
> {code:title=current help message}
> -bash-4.1$ yarn logs
> usage: general options are:
> -applicationIdApplicationId (required)
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> {code:title=proposed help message}
> -bash-4.1$ yarn logs
> usage: yarn logs -applicationId [OPTIONS]
> general options are:
> -appOwner AppOwner (assumed to be current user if not
> specified)
> -containerId ContainerId (must be specified if node address is
> specified)
> -nodeAddress NodeAddress in the format nodename:port (must be
> specified if container id is specified)
> {code}
> 2. Add description for help command. As far as I know, a user cannot get logs
> for running job. Since I spent some time trying to get logs of running
> applications, it should be nice to say this in command description.
> {code:title=proposed help}
> Retrieve logs for completed/killed YARN application
> usage: general options are...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754580#comment-13754580 ] Hudson commented on YARN-707: - SUCCESS: Integrated in Hadoop-Yarn-trunk #317 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/317/]) YARN-707. Added user information also in the YARN ClientToken so that AMs can implement authorization based on incoming users. Contributed by Jason Lowe. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1518868) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java > Add user info in the YARN ClientToken > - > > Key: YARN-707 > URL: https://issues.apache.org/jira/browse/YARN-707 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Jason Lowe >Priority: Blocker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, > YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt > > > If user info is present in the client token then it can be used to do limited > authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-879) Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
[
https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754505#comment-13754505
]
Hadoop QA commented on YARN-879:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600751/YARN-879-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1806//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1806//console
This message is automatically generated.
> Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
> --
>
> Key: YARN-879
> URL: https://issues.apache.org/jira/browse/YARN-879
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-879.patch, YARN-879-v2.patch, YARN-879-v3.patch
>
>
> getResources() will return a list of containers that allocated by RM.
> However, it is now return null directly. The worse thing is: if LOG.debug is
> enabled, then it will definitely cause NPE exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[
https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754494#comment-13754494
]
Hadoop QA commented on YARN-771:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600749/YARN-771-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1805//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1805//console
This message is automatically generated.
> AMRMClient support for resource blacklisting
> -
>
> Key: YARN-771
> URL: https://issues.apache.org/jira/browse/YARN-771
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Junping Du
> Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch,
> YARN-771-v3.patch, YARN-771-v4.patch
>
>
> After YARN-750 AMRMClient should support blacklisting via the new YARN API's
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-879) Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
[ https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-879: Attachment: YARN-879-v3.patch Sync up patch with latest trunk branch. Can someone take a review on this? Thx! > Fix tests w.r.t o.a.h.y.server.resourcemanager.Application > -- > > Key: YARN-879 > URL: https://issues.apache.org/jira/browse/YARN-879 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-879.patch, YARN-879-v2.patch, YARN-879-v3.patch > > > getResources() will return a list of containers that allocated by RM. > However, it is now return null directly. The worse thing is: if LOG.debug is > enabled, then it will definitely cause NPE exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-771: Attachment: YARN-771-v4.patch > AMRMClient support for resource blacklisting > - > > Key: YARN-771 > URL: https://issues.apache.org/jira/browse/YARN-771 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Junping Du > Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, > YARN-771-v3.patch, YARN-771-v4.patch > > > After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
