[jira] [Updated] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated YARN-276: --- Attachment: YARN-276.patch > Capacity Scheduler can hang when submit many jobs concurrently > -- > > Key: YARN-276 > URL: https://issues.apache.org/jira/browse/YARN-276 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.0.0, 2.0.1-alpha >Reporter: nemon lou >Assignee: nemon lou > Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity > scheduler can hang with most resources taken up by AM and don't have enough > resources for tasks.And then all applications hang there. > The cause is that "yarn.scheduler.capacity.maximum-am-resource-percent" not > check directly.Instead ,this property only used for maxActiveApplications. > And maxActiveApplications is computed by minimumAllocation (not by Am > actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638923#comment-13638923
]
Hadoop QA commented on YARN-276:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1258/YARN-276.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/804//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/804//console
This message is automatically generated.
> Capacity Scheduler can hang when submit many jobs concurrently
> --
>
> Key: YARN-276
> URL: https://issues.apache.org/jira/browse/YARN-276
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.0.0, 2.0.1-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch,
> YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch,
> YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch,
> YARN-276.patch, YARN-276.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity
> scheduler can hang with most resources taken up by AM and don't have enough
> resources for tasks.And then all applications hang there.
> The cause is that "yarn.scheduler.capacity.maximum-am-resource-percent" not
> check directly.Instead ,this property only used for maxActiveApplications.
> And maxActiveApplications is computed by minimumAllocation (not by Am
> actually used).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638944#comment-13638944 ] nemon lou commented on YARN-276: Parameter maxActiveApplicationsPerUsers is changed to maxAMResourcePerQueuePerUserPercent.Also checked by user's actual AM used resources. And the webService has this change,too. [~tgraves] > Capacity Scheduler can hang when submit many jobs concurrently > -- > > Key: YARN-276 > URL: https://issues.apache.org/jira/browse/YARN-276 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.0.0, 2.0.1-alpha >Reporter: nemon lou >Assignee: nemon lou > Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity > scheduler can hang with most resources taken up by AM and don't have enough > resources for tasks.And then all applications hang there. > The cause is that "yarn.scheduler.capacity.maximum-am-resource-percent" not > check directly.Instead ,this property only used for maxActiveApplications. > And maxActiveApplications is computed by minimumAllocation (not by Am > actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM
[ https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638947#comment-13638947 ] Hudson commented on YARN-549: - Integrated in Hadoop-Yarn-trunk #192 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/192/]) YARN-549. YarnClient.submitApplication should wait for application to be accepted by the RM (Zhijie Shen via bikas) (Revision 1470797) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470797 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > YarnClient.submitApplication should wait for application to be accepted by > the RM > - > > Key: YARN-549 > URL: https://issues.apache.org/jira/browse/YARN-549 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: Proposal of Asynchronous Application Submission_v1.pdf, > YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch > > > Currently, when submitting an application, storeApplication will be called > for recovery. However, it is a blocking API, and is likely to block > concurrent application submissions. Therefore, it is good to make application > submission asynchronous, and postpone storeApplication. YarnClient needs to > change to wait for the whole operation to complete so that clients can be > notified after the application is really submitted. YarnClient needs to wait > for application to reach SUBMITTED state or beyond. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache
[ https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638949#comment-13638949 ] Hudson commented on YARN-583: - Integrated in Hadoop-Yarn-trunk #192 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/192/]) YARN-583. Moved application level local resources to be localized under the filecache sub-directory under application directory. Contributed by Omkar Vinit Joshi. (Revision 1470812) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470812 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Application cache files should be localized under > local-dir/usercache/userid/appcache/appid/filecache > - > > Key: YARN-583 > URL: https://issues.apache.org/jira/browse/YARN-583 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, > yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch > > > Currently application cache files are getting localized under > local-dir/usercache/userid/appcache/appid/. however they should be localized > under filecache sub directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated YARN-276: --- Labels: incompatible (was: ) > Capacity Scheduler can hang when submit many jobs concurrently > -- > > Key: YARN-276 > URL: https://issues.apache.org/jira/browse/YARN-276 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.0.0, 2.0.1-alpha >Reporter: nemon lou >Assignee: nemon lou > Labels: incompatible > Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, > YARN-276.patch, YARN-276.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity > scheduler can hang with most resources taken up by AM and don't have enough > resources for tasks.And then all applications hang there. > The cause is that "yarn.scheduler.capacity.maximum-am-resource-percent" not > check directly.Instead ,this property only used for maxActiveApplications. > And maxActiveApplications is computed by minimumAllocation (not by Am > actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM
[ https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639005#comment-13639005 ] Hudson commented on YARN-549: - Integrated in Hadoop-Hdfs-trunk #1381 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1381/]) YARN-549. YarnClient.submitApplication should wait for application to be accepted by the RM (Zhijie Shen via bikas) (Revision 1470797) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470797 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > YarnClient.submitApplication should wait for application to be accepted by > the RM > - > > Key: YARN-549 > URL: https://issues.apache.org/jira/browse/YARN-549 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: Proposal of Asynchronous Application Submission_v1.pdf, > YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch > > > Currently, when submitting an application, storeApplication will be called > for recovery. However, it is a blocking API, and is likely to block > concurrent application submissions. Therefore, it is good to make application > submission asynchronous, and postpone storeApplication. YarnClient needs to > change to wait for the whole operation to complete so that clients can be > notified after the application is really submitted. YarnClient needs to wait > for application to reach SUBMITTED state or beyond. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache
[ https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639007#comment-13639007 ] Hudson commented on YARN-583: - Integrated in Hadoop-Hdfs-trunk #1381 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1381/]) YARN-583. Moved application level local resources to be localized under the filecache sub-directory under application directory. Contributed by Omkar Vinit Joshi. (Revision 1470812) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470812 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Application cache files should be localized under > local-dir/usercache/userid/appcache/appid/filecache > - > > Key: YARN-583 > URL: https://issues.apache.org/jira/browse/YARN-583 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, > yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch > > > Currently application cache files are getting localized under > local-dir/usercache/userid/appcache/appid/. however they should be localized > under filecache sub directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639056#comment-13639056 ] Thomas Graves commented on YARN-126: Thanks for working on this. A few comments. Unfortunately we aren't consistent across hadoop how we print usages. - In the usage print out, I'd prefer to separate out the generic options like -D, -conf, and -rm from the actual commands so that its more obvious to users what the commands are. - Capitalize usage in yarn rmadmin (Usage:) - the usage should say something like: yarn rmadmin [generic options] COMMAND We should also fix the GenericOptionsParser to not reference the jobtracker anymore, but I can file a separate jira for that. > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM
[ https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639084#comment-13639084 ] Hudson commented on YARN-549: - Integrated in Hadoop-Mapreduce-trunk #1408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1408/]) YARN-549. YarnClient.submitApplication should wait for application to be accepted by the RM (Zhijie Shen via bikas) (Revision 1470797) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470797 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > YarnClient.submitApplication should wait for application to be accepted by > the RM > - > > Key: YARN-549 > URL: https://issues.apache.org/jira/browse/YARN-549 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: Proposal of Asynchronous Application Submission_v1.pdf, > YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch > > > Currently, when submitting an application, storeApplication will be called > for recovery. However, it is a blocking API, and is likely to block > concurrent application submissions. Therefore, it is good to make application > submission asynchronous, and postpone storeApplication. YarnClient needs to > change to wait for the whole operation to complete so that clients can be > notified after the application is really submitted. YarnClient needs to wait > for application to reach SUBMITTED state or beyond. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache
[ https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639086#comment-13639086 ] Hudson commented on YARN-583: - Integrated in Hadoop-Mapreduce-trunk #1408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1408/]) YARN-583. Moved application level local resources to be localized under the filecache sub-directory under application directory. Contributed by Omkar Vinit Joshi. (Revision 1470812) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470812 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Application cache files should be localized under > local-dir/usercache/userid/appcache/appid/filecache > - > > Key: YARN-583 > URL: https://issues.apache.org/jira/browse/YARN-583 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, > yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch > > > Currently application cache files are getting localized under > local-dir/usercache/userid/appcache/appid/. however they should be localized > under filecache sub directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639090#comment-13639090 ] Timothy St. Clair commented on YARN-600: What happens when a node does not have it enabled? SetAffinity? > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-561: --- Attachment: YARN-561.10.patch > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, > YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, > YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-289) Fair scheduler allows reservations that won't fit on node
[ https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-289: Attachment: YARN-289-1.patch > Fair scheduler allows reservations that won't fit on node > - > > Key: YARN-289 > URL: https://issues.apache.org/jira/browse/YARN-289 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-289-1.patch, YARN-289.patch > > > An application requests a container with 1024 MB. It then requests a > container with 2048 MB. A node shows up with 1024 MB available. Even if the > application is the only one running, neither request will be scheduled on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-289) Fair scheduler allows reservations that won't fit on node
[ https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639282#comment-13639282 ] Sandy Ryza commented on YARN-289: - Uploaded patch to fix javadoc warnings > Fair scheduler allows reservations that won't fit on node > - > > Key: YARN-289 > URL: https://issues.apache.org/jira/browse/YARN-289 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-289-1.patch, YARN-289.patch > > > An application requests a container with 1024 MB. It then requests a > container with 2048 MB. A node shows up with 1024 MB available. Even if the > application is the only one running, neither request will be scheduled on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639309#comment-13639309
]
Hadoop QA commented on YARN-561:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580081/YARN-561.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/805//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/805//console
This message is automatically generated.
> Nodemanager should set some key information into the environment of every
> container that it launches.
> -
>
> Key: YARN-561
> URL: https://issues.apache.org/jira/browse/YARN-561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
> Labels: usability
> Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch,
> YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch,
> YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch
>
>
> Information such as containerId, nodemanager hostname, nodemanager port is
> not set in the environment when any container is launched.
> For an AM, the RM does all of this for it but for a container launched by an
> application, all of the above need to be set by the ApplicationMaster.
> At the minimum, container id would be a useful piece of information. If the
> container wishes to talk to its local NM, the nodemanager related information
> would also come in handy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-289) Fair scheduler allows reservations that won't fit on node
[
https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639320#comment-13639320
]
Hadoop QA commented on YARN-289:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580088/YARN-289-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/806//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/806//console
This message is automatically generated.
> Fair scheduler allows reservations that won't fit on node
> -
>
> Key: YARN-289
> URL: https://issues.apache.org/jira/browse/YARN-289
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-289-1.patch, YARN-289.patch
>
>
> An application requests a container with 1024 MB. It then requests a
> container with 2048 MB. A node shows up with 1024 MB available. Even if the
> application is the only one running, neither request will be scheduled on it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639348#comment-13639348
]
Hitesh Shah commented on YARN-561:
--
Comments:
ApplicationAttemptId appAttemptIdEnv - please use a more appropriate variable
name.
System.getenv(Environment.CONTAINER_ID.name()) - what happens if the container
id is not set in the env?
{code}
putEnvIfNotNull(environment, Environment.USER.name(), container.getUser());
{code}
Is the user allowed to override the user env var?
{code}
+ LOG.info("Adding " + container.getContainer().getId()
+ " to application " + app.toString());
{code}
- Doesn't the above happen so often that it will be a performance overhead?
Should this be changed to DEBUG? @Vinod, any comments on this?
> Nodemanager should set some key information into the environment of every
> container that it launches.
> -
>
> Key: YARN-561
> URL: https://issues.apache.org/jira/browse/YARN-561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
> Labels: usability
> Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch,
> YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch,
> YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch
>
>
> Information such as containerId, nodemanager hostname, nodemanager port is
> not set in the environment when any container is launched.
> For an AM, the RM does all of this for it but for a container launched by an
> application, all of the above need to be set by the ApplicationMaster.
> At the minimum, container id would be a useful piece of information. If the
> container wishes to talk to its local NM, the nodemanager related information
> would also come in handy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639476#comment-13639476 ] Vinod Kumar Vavilapalli commented on YARN-561: -- bq. ApplicationAttemptId appAttemptIdEnv - please use a more appropriate variable name. +1 bq. System.getenv(Environment.CONTAINER_ID.name()) - what happens if the container id is not set in the env? Unless you are looking to run new MR APP against old YARN, this shouldn't happen because NM always sets it now, no? bq. Is the user allowed to override the user env var? No. Mostly PWD too. But let's do those separately, this one dragged on for too long. bq. Doesn't the above happen so often that it will be a performance overhead? Should be fine for now, as there are other places like this. We should do a comprehensive log cleanup separately. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, > YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, > YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639481#comment-13639481 ] Vinod Kumar Vavilapalli commented on YARN-126: -- bq. We should also fix the GenericOptionsParser to not reference the jobtracker anymore, but I can file a separate jira for that. Shouldn't this be the correct fix instead of throwing out GenericOptionsParser altogether? > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639491#comment-13639491
]
Alejandro Abdelnur commented on YARN-392:
-
The idea of a locality delay gives more flexibility, I like it.
Though I think it may complicate things quite a bit on the scheduler to be able
to do the right book-keeping. Today, because the delay is at app level there is
not delay counting at allocation requests level.
If we move the delay at allocation request level, it means we'd have to keep a
counter at 'rack' level that gets decremented on every allocation attempt and
when hits zero we go rack if node was not fulfilled.
This means that the allocation request in the scheduler will have to have a new
delay-counter property.
The complexity comes that when an AM places a new allocation request, the AM
must provide the non-empty allocation requests in full.
For example:
Requesting 5 containers for node1/rack1:
{code}
location=* - containers=5
location=rack1 - containers=5
location=node1 - containers=5
{code}
Requesting 5 additional containers for node2/rack1 (with original allocation
still pending):
{code}
location=* - containers=10
location=rack1 - containers=10
location=node2 - containers=5
{code}
The current contract allows the scheduler just to put the */rack container
requests without having to do a lookup and aggregation.
If we are now keeping a delay counter associated at */rack level and we do a
put, we'll reset the delay-counter for the node1 request family. If we keep the
delay-counter of node1 request family and use it for the node2 request family
we'll be shorting the node2 request expected locality delay.
The delay-locality per container request has value but I think it may require
much more work.
Given that, don't you think getting the current approach suggested/implemented
by Arun/Sandy makes sense in the short term?
> Make it possible to schedule to specific nodes without dropping locality
>
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639513#comment-13639513 ] Rémy SAISSY commented on YARN-126: -- I removed the GenericOptionsParser because I saw that the yarn distributed shell ApplicationMaster.java relies on org.apache.commons.cli.GnuParser instead. So I though that since an options parser is not something specific to Hadoop, it might be a good thing to remove the GenericOptionParser dependency where it is possible to favor the commons.cli library instead. Isn't it the case? > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639514#comment-13639514 ] Vinod Kumar Vavilapalli commented on YARN-561: -- The latest patch also removed testing of Environment.MALLOC_ARENA_MAX, but I believe the current test isn't really the right way neither is it sufficient. We should really be validating YarnConfiguration.NM_ADMIN_USER_ENV. Can you please file a separate ticket for that? > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, > YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, > YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639568#comment-13639568 ] Hitesh Shah commented on YARN-561: -- @Vinod, there are also checks in the MRAppMaster which pretty much validate everything that the NM is supposed to be setting in the env. ( look for callers of MRAppMaster#validateInputParam ). This can be probably addressed separately in any case. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, > YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, > YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-561: --- Attachment: YARN-561.11.patch This patch and last patch remove the testing of Environment.MALLOC_ARENA_MAX from TestContainerLaunch. Will file a new jira ticket for it. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources
[ https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639587#comment-13639587 ] Alejandro Abdelnur commented on YARN-595: - patch looks good, couple of NITs: * FairScheduler: patch is adding a bunch of System.out.println(), I guess this is debris from your testing, if we need to output any of this, please use the log. * FSSchedulerNode: no need to initialize availableResource var on definition, it is always assigned in the constructor. > Refactor fair scheduler to use common Resources > --- > > Key: YARN-595 > URL: https://issues.apache.org/jira/browse/YARN-595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-595.patch, YARN-595.patch > > > resourcemanager.fair and resourcemanager.resources have two copies of > basically the same code for operations on Resource objects -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639590#comment-13639590
]
Thomas Graves commented on YARN-126:
{quote}Shouldn't this be the correct fix instead of throwing out
GenericOptionsParser altogether?{quote}
I'm a bit torn on that. yes it does have a couple of the options used but it
also has others that do not apply to the rmadmin command. It also has the
generic usage statement of: "bin/hadoop command [genericOptions]
[commandOptions]" which doesn't apply at all. I guess another option is to
change the GenericOptionsParser to be able to specify which options can be used
and have it print the correct usage statements and so forth.
Ideally I think we should redo all of the hadoop/hdfs/yarn/mapred commands to
have consistent usage and interfaces but that is a much bigger change.
> yarn rmadmin help message contains reference to hadoop cli and JT
> -
>
> Key: YARN-126
> URL: https://issues.apache.org/jira/browse/YARN-126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Rémy SAISSY
> Labels: usability
> Attachments: YARN-126.patch
>
>
> has option to specify a job tracker and the last line for general command
> line syntax had "bin/hadoop command [genericOptions] [commandOptions]"
> ran "yarn rmadmin" to get usage:
> RMAdmin
> Usage: java RMAdmin
>[-refreshQueues]
>[-refreshNodes]
>[-refreshUserToGroupsMappings]
>[-refreshSuperUserGroupsConfiguration]
>[-refreshAdminAcls]
>[-refreshServiceAcl]
>[-help [cmd]]
> Generic options supported are
> -conf specify an application configuration file
> -D use value for given property
> -fs specify a namenode
> -jt specify a job tracker
> -files specify comma separated files to be
> copied to the map reduce cluster
> -libjars specify comma separated jar files
> to include in the classpath.
> -archives specify comma separated
> archives to be unarchived on the compute machines.
> The general command line syntax is
> bin/hadoop command [genericOptions] [commandOptions]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers
[ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639595#comment-13639595 ] Karthik Kambatla commented on YARN-45: -- Barely skimmed through the patch, it looks good. Noticed a few javadoc typos we might like to fix. Will try to get in a more detailed review "soon". > Scheduler feedback to AM to release containers > -- > > Key: YARN-45 > URL: https://issues.apache.org/jira/browse/YARN-45 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Chris Douglas >Assignee: Carlo Curino > Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch > > > The ResourceManager strikes a balance between cluster utilization and strict > enforcement of resource invariants in the cluster. Individual allocations of > containers must be reclaimed- or reserved- to restore the global invariants > when cluster load shifts. In some cases, the ApplicationMaster can respond to > fluctuations in resource availability without losing the work already > completed by that task (MAPREDUCE-4584). Supplying it with this information > would be helpful for overall cluster utilization [1]. To this end, we want to > establish a protocol for the RM to ask the AM to release containers. > [1] http://research.yahoo.com/files/yl-2012-003.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: proposal_v1.pdf Some initial thoughts about creating AMNMClient. Your comments, please. > Add AM-NM client library > > > Key: YARN-422 > URL: https://issues.apache.org/jira/browse/YARN-422 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Zhijie Shen > Attachments: proposal_v1.pdf > > > Create a simple wrapper over the AM-NM container protocol to provide hide the > details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rémy SAISSY updated YARN-126: - Attachment: YARN-126.patch Here is the good patch (for the fixed usage). > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.patch, YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639613#comment-13639613 ] Rémy SAISSY commented on YARN-126: -- | Ideally I think we should redo all of the hadoop/hdfs/yarn/mapred commands to have consistent usage and interfaces but that is a much bigger change. Yes and the command line options of these tools might be confusing sometimes. In my daily life I am an IT consultant and when I install/train clients on Hadoop, they very often encounter two issues when it comes to using the command line: - which options are optional for a specific command (-fs in yarn rmadmin?) - where does some arguments must be specified (-Dmapreduce.map.java.opts when submiting a job on command line using hadoop jar). Maybe the redo could be done in an umbrella JIRA with several issues, one per project? > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.patch, YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639624#comment-13639624
]
Hadoop QA commented on YARN-561:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580122/YARN-561.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/807//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/807//console
This message is automatically generated.
> Nodemanager should set some key information into the environment of every
> container that it launches.
> -
>
> Key: YARN-561
> URL: https://issues.apache.org/jira/browse/YARN-561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
> Labels: usability
> Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch,
> YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch,
> YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch
>
>
> Information such as containerId, nodemanager hostname, nodemanager port is
> not set in the environment when any container is launched.
> For an AM, the RM does all of this for it but for a container launched by an
> application, all of the above need to be set by the ApplicationMaster.
> At the minimum, container id would be a useful piece of information. If the
> container wishes to talk to its local NM, the nodemanager related information
> would also come in handy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639627#comment-13639627
]
Hadoop QA commented on YARN-126:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580129/YARN-126.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/808//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/808//console
This message is automatically generated.
> yarn rmadmin help message contains reference to hadoop cli and JT
> -
>
> Key: YARN-126
> URL: https://issues.apache.org/jira/browse/YARN-126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Rémy SAISSY
> Labels: usability
> Attachments: YARN-126.patch, YARN-126.patch
>
>
> has option to specify a job tracker and the last line for general command
> line syntax had "bin/hadoop command [genericOptions] [commandOptions]"
> ran "yarn rmadmin" to get usage:
> RMAdmin
> Usage: java RMAdmin
>[-refreshQueues]
>[-refreshNodes]
>[-refreshUserToGroupsMappings]
>[-refreshSuperUserGroupsConfiguration]
>[-refreshAdminAcls]
>[-refreshServiceAcl]
>[-help [cmd]]
> Generic options supported are
> -conf specify an application configuration file
> -D use value for given property
> -fs specify a namenode
> -jt specify a job tracker
> -files specify comma separated files to be
> copied to the map reduce cluster
> -libjars specify comma separated jar files
> to include in the classpath.
> -archives specify comma separated
> archives to be unarchived on the compute machines.
> The general command line syntax is
> bin/hadoop command [genericOptions] [commandOptions]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart
[ https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639638#comment-13639638 ] Jian He commented on YARN-581: -- bq. I remember we discussed this on some other ticket too? Why not stop RM1 before starting RM2 in the test-case? Because its incorrect to stop rm1 first. If rm1.stop(), the store dispatcher is stopped, its not able to process remove event any more, app1 still remain. I also comment this on YARN-594 > Test and verify that app delegation tokens are restored after RM restart > > > Key: YARN-581 > URL: https://issues.apache.org/jira/browse/YARN-581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-581.1.patch > > > The code already saves the delegation tokens in AppSubmissionContext. Upon > restart the AppSubmissionContext is used to submit the application again and > so restores the delegation tokens. This jira tracks testing and verifying > this functionality in a secure setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639639#comment-13639639 ] Vinod Kumar Vavilapalli commented on YARN-422: -- Comments on your proposal - AMNMClient is good. - We should have only one instance per AM which talks to *all* NMs. Just like ContainerLauncherImpl in MR App. Clearly, the interface you proposed is already accommodating for that. - All the APIs will be blocking? It isn't clear. Today's MR App's ContainerLauncher is non-blocking and it has call-backs/signals as to what happened with a container launch/stop - succeeded/failed. I like making it non-blocking as AMs more commonly should want to kick off the launch and go away. A blocking API could be implemented on top of the non-blocking API if need be. - I think we should change AMLauncher also to use this, but we can scope it into a separate ticket depending on this patch's size What do others think? > Add AM-NM client library > > > Key: YARN-422 > URL: https://issues.apache.org/jira/browse/YARN-422 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Zhijie Shen > Attachments: proposal_v1.pdf > > > Create a simple wrapper over the AM-NM container protocol to provide hide the > details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639640#comment-13639640 ] Vinod Kumar Vavilapalli commented on YARN-561: -- bq. there are also checks in the MRAppMaster which pretty much validate everything that the NM is supposed to be setting in the env. ( look for callers of MRAppMaster#validateInputParam ). This can be probably addressed separately in any case. Yeah, let's do those separately. The latest patch looks good, +1, checking it in. Xuan, can you please file all the follow up tickets? Thanks. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639643#comment-13639643 ] Xuan Gong commented on YARN-561: Yes, I will do that. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-562) NM should reject containers allocated by previous RM
[ https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-562: - Attachment: YARN-562.8.patch New patch does: 1. Add test to verify new container requests are blocked when NM start from scratch until it registers with RM. 2. Add test to verify RMIdentifier is set in RegisterNodeManagerResponse 3. Add test to verify RMIdentifier is set on allocated containers. 4. Address above comments > NM should reject containers allocated by previous RM > > > Key: YARN-562 > URL: https://issues.apache.org/jira/browse/YARN-562 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, > YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, > YARN-562.8.patch > > > Its possible that after RM shutdown, before AM goes down,AM still call > startContainer on NM with containers allocated by previous RM. When RM comes > back, NM doesn't know whether this container launch request comes from > previous RM or the current RM. we should reject containers allocated by > previous RM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-601) Refactoring the code which computes the user file cache and user application file cache paths
Omkar Vinit Joshi created YARN-601: -- Summary: Refactoring the code which computes the user file cache and user application file cache paths Key: YARN-601 URL: https://issues.apache.org/jira/browse/YARN-601 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Priority: Minor At present at multiple places user local cache file path and user application file path are getting calculated. It is better to expose them as a single static utility method and reuse them everywhere else. Locations : * ContainerLaunch * DefaultContainerExecutor: this already has some methods like this * ResourceLocalizationService: getUserCacheFilePath getAppFileCachePath * ContainerLocalizer * ShuffleHandler.Shuffle * TestContainerLocalizer, TestContainerManager, TestDefaultContainerExecutor and TestResourceLocalizationService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM
[
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639659#comment-13639659
]
Hadoop QA commented on YARN-562:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580136/YARN-562.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/809//console
This message is automatically generated.
> NM should reject containers allocated by previous RM
>
>
> Key: YARN-562
> URL: https://issues.apache.org/jira/browse/YARN-562
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch,
> YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch,
> YARN-562.8.patch
>
>
> Its possible that after RM shutdown, before AM goes down,AM still call
> startContainer on NM with containers allocated by previous RM. When RM comes
> back, NM doesn't know whether this container launch request comes from
> previous RM or the current RM. we should reject containers allocated by
> previous RM
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639660#comment-13639660 ] Bikas Saha commented on YARN-422: - I was hoping that the final AMNMClient library would be a one stop library for AM's to communicate with all the NM's in the cluster. The document outlines a wrapper around the NMProtocol for talking to 1 NM. This wrapper is useful by itself because it hides the PBImpl etc implementation details of the protocol itself, similar to how YarnClient started off as. AMRMClient does a little bit more than simply wrapping the Java protocol. So as a first step the above design for wrapping the boiler plate code of the NM Protocol into a utility library wrapper sounds good. I think the method calls are all blocking since the RPC itself is blocking and this is a simple wrapper around the RPC. Maybe we can call be NMClient. Once this is done we can look at creating an AMNMClient library that will provide a single object that an AM can use to manage its containers. This may have its own internal threading etc which will allow the AM to make non-blocking calls. Lets figure that out in a subsequent jira and use this one to just write the wrapper NMCLient that you have proposed. > Add AM-NM client library > > > Key: YARN-422 > URL: https://issues.apache.org/jira/browse/YARN-422 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Zhijie Shen > Attachments: proposal_v1.pdf > > > Create a simple wrapper over the AM-NM container protocol to provide hide the > details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-581) Test and verify that app delegation tokens are restored after RM restart
[ https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-581: - Attachment: YARN-581.2.patch New patch fixed above comments > Test and verify that app delegation tokens are restored after RM restart > > > Key: YARN-581 > URL: https://issues.apache.org/jira/browse/YARN-581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-581.1.patch, YARN-581.2.patch > > > The code already saves the delegation tokens in AppSubmissionContext. Upon > restart the AppSubmissionContext is used to submit the application again and > so restores the delegation tokens. This jira tracks testing and verifying > this functionality in a secure setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639673#comment-13639673 ] Vinod Kumar Vavilapalli commented on YARN-561: -- Xuan, we need a separate patch for branch-2, the trunk patch isn't applying. Can you please generate one, run all the tests on branch-2 applying that patch? We can't do that on Jenkins, as it isn't built for running tests on branches. Tx. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639676#comment-13639676 ] Hudson commented on YARN-561: - Integrated in Hadoop-trunk-Commit #3651 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3651/]) YARN-561. Modified NodeManager to set key information into the environment of every container that it launches. Contributed by Xuan Gong. MAPREDUCE-5175. Updated MR App to not set envs that will be set by NMs anyways after YARN-561. Contributed by Xuan Gong. (Revision 1471156) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1471156 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/ContainerInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanage
[jira] [Created] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches
Xuan Gong created YARN-602: -- Summary: NodeManager should mandatorily set some Environment variables into every containers that it launches Key: YARN-602 URL: https://issues.apache.org/jira/browse/YARN-602 Project: Hadoop YARN Issue Type: Bug Environment: NodeManager should mandatorily set some Environment variables into every containers that it launches, such as Environment.user, Environment.pwd. If both users and NodeManager set those variables, the value set by NM should be used Reporter: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-603) Create testcase to validate Environment.MALLOC_ARENA_MAX
Xuan Gong created YARN-603: -- Summary: Create testcase to validate Environment.MALLOC_ARENA_MAX Key: YARN-603 URL: https://issues.apache.org/jira/browse/YARN-603 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong The current test to validate Environment.MALLOC_ARENA_MAX isn't sufficient. We need validate YarnConfiguration.NM_ADMIN_USER_ENV, too. And YARN-561 removed testing of Environment.MALLOC_ARENA_MAX, we need to create new test case to test it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart
[
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639688#comment-13639688
]
Hadoop QA commented on YARN-581:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580139/YARN-581.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/810//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/810//console
This message is automatically generated.
> Test and verify that app delegation tokens are restored after RM restart
>
>
> Key: YARN-581
> URL: https://issues.apache.org/jira/browse/YARN-581
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-581.1.patch, YARN-581.2.patch
>
>
> The code already saves the delegation tokens in AppSubmissionContext. Upon
> restart the AppSubmissionContext is used to submit the application again and
> so restores the delegation tokens. This jira tracks testing and verifying
> this functionality in a secure setup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart
[ https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639748#comment-13639748 ] Vinod Kumar Vavilapalli commented on YARN-581: -- Latest patch looks good, checking it in. > Test and verify that app delegation tokens are restored after RM restart > > > Key: YARN-581 > URL: https://issues.apache.org/jira/browse/YARN-581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-581.1.patch, YARN-581.2.patch > > > The code already saves the delegation tokens in AppSubmissionContext. Upon > restart the AppSubmissionContext is used to submit the application again and > so restores the delegation tokens. This jira tracks testing and verifying > this functionality in a secure setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart
[ https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639761#comment-13639761 ] Hudson commented on YARN-581: - Integrated in Hadoop-trunk-Commit #3652 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3652/]) YARN-581. Added a test to verify that app delegation tokens are restored after RM restart. Contributed by Jian He. (Revision 1471187) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1471187 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > Test and verify that app delegation tokens are restored after RM restart > > > Key: YARN-581 > URL: https://issues.apache.org/jira/browse/YARN-581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.0.5-beta > > Attachments: YARN-581.1.patch, YARN-581.2.patch > > > The code already saves the delegation tokens in AppSubmissionContext. Upon > restart the AppSubmissionContext is used to submit the application again and > so restores the delegation tokens. This jira tracks testing and verifying > this functionality in a secure setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639767#comment-13639767 ] Vinod Kumar Vavilapalli commented on YARN-422: -- I don't see any value in a wrapper around communication to a single NM. It does hide-away the records, but what is useful and important is a library which takes in a container given by RM and launches in whatever node it is allocated on. In that sense, this "ContainerLauncher" library is what should be the primary end-point for AM writers. If one wants to launch containers on a single node, the same library should be enough. And yeah, the more I think about, non-blocking APIs seem like must-have. > Add AM-NM client library > > > Key: YARN-422 > URL: https://issues.apache.org/jira/browse/YARN-422 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Zhijie Shen > Attachments: proposal_v1.pdf > > > Create a simple wrapper over the AM-NM container protocol to provide hide the > details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-562) NM should reject containers allocated by previous RM
[ https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-562: - Attachment: YARN-562.9.patch fixed build failure.. > NM should reject containers allocated by previous RM > > > Key: YARN-562 > URL: https://issues.apache.org/jira/browse/YARN-562 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, > YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, > YARN-562.8.patch, YARN-562.9.patch > > > Its possible that after RM shutdown, before AM goes down,AM still call > startContainer on NM with containers allocated by previous RM. When RM comes > back, NM doesn't know whether this container launch request comes from > previous RM or the current RM. we should reject containers allocated by > previous RM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-561: --- Attachment: YARN-561.12.branch_2.patch Create patch for branch_2 only > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, > YARN-561.12.branch_2.patch, YARN-561.1.patch, YARN-561.2.patch, > YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, > YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639794#comment-13639794
]
Hadoop QA commented on YARN-561:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580178/YARN-561.12.branch_2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/812//console
This message is automatically generated.
> Nodemanager should set some key information into the environment of every
> container that it launches.
> -
>
> Key: YARN-561
> URL: https://issues.apache.org/jira/browse/YARN-561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
> Labels: usability
> Attachments: YARN-561.10.patch, YARN-561.11.patch,
> YARN-561.12.branch_2.patch, YARN-561.1.patch, YARN-561.2.patch,
> YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch,
> YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch
>
>
> Information such as containerId, nodemanager hostname, nodemanager port is
> not set in the environment when any container is launched.
> For an AM, the RM does all of this for it but for a container launched by an
> application, all of the above need to be set by the ApplicationMaster.
> At the minimum, container id would be a useful piece of information. If the
> container wishes to talk to its local NM, the nodemanager related information
> would also come in handy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639834#comment-13639834 ] Bikas Saha commented on YARN-422: - Sounds good to me to skip the first wrapper step if there are no consumers for it. This would still be an NMClient library since potentially the RM and AM can both use it. > Add AM-NM client library > > > Key: YARN-422 > URL: https://issues.apache.org/jira/browse/YARN-422 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Zhijie Shen > Attachments: proposal_v1.pdf > > > Create a simple wrapper over the AM-NM container protocol to provide hide the > details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-561: --- Attachment: YARN-561.13.branch-2.patch 1.Create new patch for branch-2 2.Pass all Yarn test cases locally > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Attachments: YARN-561.10.patch, YARN-561.11.patch, > YARN-561.12.branch_2.patch, YARN-561.13.branch-2.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM
[
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639844#comment-13639844
]
Hadoop QA commented on YARN-562:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580175/YARN-562.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/811//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/811//console
This message is automatically generated.
> NM should reject containers allocated by previous RM
>
>
> Key: YARN-562
> URL: https://issues.apache.org/jira/browse/YARN-562
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch,
> YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch,
> YARN-562.8.patch, YARN-562.9.patch
>
>
> Its possible that after RM shutdown, before AM goes down,AM still call
> startContainer on NM with containers allocated by previous RM. When RM comes
> back, NM doesn't know whether this container launch request comes from
> previous RM or the current RM. we should reject containers allocated by
> previous RM
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639848#comment-13639848 ] Zhijie Shen commented on YARN-599: -- In YARN-599.1.patch, there're the following changes: 1. ClientRMService#submitApplication calls RMAppManager#submitApplication directly. APP_SUBMIT event is removed at all. RMAppManager#submitApplication throws YarnRemoteException. 2. Move getCurrentUser and validateResourceRequest from ClientRMService#submitApplication to RMAppManager#submitApplication. Move getQueue and getApplicationName from RMAppManager#submitApplication to ClientRMService#submitApplication. Adjust the test cases in TestClientRMService and TestAppManger accordingly. 3. Refactor try-catch block in RMAppManager#submitApplication to avoid sending APP_REJECTED event to existing app in rmContext given duplicate applicateId. 4. Refactor TestAppManger to extract common part of the tests and push them to setup(). > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-599.1.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.
[ https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-561. -- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Incompatible change,Reviewed branch-2 patch looks good too. +1. Committed this to trunk and branch-2. Thanks Xuan. Marking this as incompatible, as it is renaming the LOCAL_DIRS env constant. > Nodemanager should set some key information into the environment of every > container that it launches. > - > > Key: YARN-561 > URL: https://issues.apache.org/jira/browse/YARN-561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Labels: usability > Fix For: 2.0.5-beta > > Attachments: YARN-561.10.patch, YARN-561.11.patch, > YARN-561.12.branch_2.patch, YARN-561.13.branch-2.patch, YARN-561.1.patch, > YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, > YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch > > > Information such as containerId, nodemanager hostname, nodemanager port is > not set in the environment when any container is launched. > For an AM, the RM does all of this for it but for a container launched by an > application, all of the above need to be set by the ApplicationMaster. > At the minimum, container id would be a useful piece of information. If the > container wishes to talk to its local NM, the nodemanager related information > would also come in handy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-603) Create a testcase to validate Environment.MALLOC_ARENA_MAX
[ https://issues.apache.org/jira/browse/YARN-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-603: --- Summary: Create a testcase to validate Environment.MALLOC_ARENA_MAX (was: Create testcase to validate Environment.MALLOC_ARENA_MAX) > Create a testcase to validate Environment.MALLOC_ARENA_MAX > -- > > Key: YARN-603 > URL: https://issues.apache.org/jira/browse/YARN-603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > > The current test to validate Environment.MALLOC_ARENA_MAX isn't sufficient. > We need validate YarnConfiguration.NM_ADMIN_USER_ENV, too. > And YARN-561 removed testing of Environment.MALLOC_ARENA_MAX, we need to > create new test case to test it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-604) The checks in the MRAppMaster should validate everything that the NM is supposed to be setting in the env
Xuan Gong created YARN-604: -- Summary: The checks in the MRAppMaster should validate everything that the NM is supposed to be setting in the env Key: YARN-604 URL: https://issues.apache.org/jira/browse/YARN-604 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-604) The checkers in the MRAppMaster should validate everything that the NM is supposed to be setting in the env
[ https://issues.apache.org/jira/browse/YARN-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-604: --- Summary: The checkers in the MRAppMaster should validate everything that the NM is supposed to be setting in the env (was: The checks in the MRAppMaster should validate everything that the NM is supposed to be setting in the env) > The checkers in the MRAppMaster should validate everything that the NM is > supposed to be setting in the env > --- > > Key: YARN-604 > URL: https://issues.apache.org/jira/browse/YARN-604 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-595) Refactor fair scheduler to use common Resources
[ https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-595: Attachment: YARN-595-1.patch > Refactor fair scheduler to use common Resources > --- > > Key: YARN-595 > URL: https://issues.apache.org/jira/browse/YARN-595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch > > > resourcemanager.fair and resourcemanager.resources have two copies of > basically the same code for operations on Resource objects -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources
[ https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639901#comment-13639901 ] Sandy Ryza commented on YARN-595: - Updated patch addresses Alejandro's comments > Refactor fair scheduler to use common Resources > --- > > Key: YARN-595 > URL: https://issues.apache.org/jira/browse/YARN-595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch > > > resourcemanager.fair and resourcemanager.resources have two copies of > basically the same code for operations on Resource objects -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-604) The checkers in the MRAppMaster should validate everything that the NM is supposed to be setting in the env
[ https://issues.apache.org/jira/browse/YARN-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639911#comment-13639911 ] Vinod Kumar Vavilapalli commented on YARN-604: -- No, wait. I thought we don't need to validate them anymore as NM itself is setting it. Isn't that the case? > The checkers in the MRAppMaster should validate everything that the NM is > supposed to be setting in the env > --- > > Key: YARN-604 > URL: https://issues.apache.org/jira/browse/YARN-604 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM
[ https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639914#comment-13639914 ] Vinod Kumar Vavilapalli commented on YARN-562: -- Looking at the patch for final review/commit. > NM should reject containers allocated by previous RM > > > Key: YARN-562 > URL: https://issues.apache.org/jira/browse/YARN-562 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, > YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, > YARN-562.8.patch, YARN-562.9.patch > > > Its possible that after RM shutdown, before AM goes down,AM still call > startContainer on NM with containers allocated by previous RM. When RM comes > back, NM doesn't know whether this container launch request comes from > previous RM or the current RM. we should reject containers allocated by > previous RM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches
[ https://issues.apache.org/jira/browse/YARN-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-602: - Description: NodeManager should mandatorily set some Environment variables into every containers that it launches, such as Environment.user, Environment.pwd. If both users and NodeManager set those variables, the value set by NM should be used > NodeManager should mandatorily set some Environment variables into every > containers that it launches > > > Key: YARN-602 > URL: https://issues.apache.org/jira/browse/YARN-602 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > > NodeManager should mandatorily set some Environment variables into every > containers that it launches, such as Environment.user, Environment.pwd. If > both users and NodeManager set those variables, the value set by NM should be > used -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches
[ https://issues.apache.org/jira/browse/YARN-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-602: - Environment: (was: NodeManager should mandatorily set some Environment variables into every containers that it launches, such as Environment.user, Environment.pwd. If both users and NodeManager set those variables, the value set by NM should be used ) > NodeManager should mandatorily set some Environment variables into every > containers that it launches > > > Key: YARN-602 > URL: https://issues.apache.org/jira/browse/YARN-602 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources
[
https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639950#comment-13639950
]
Hadoop QA commented on YARN-595:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580198/YARN-595-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/813//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/813//console
This message is automatically generated.
> Refactor fair scheduler to use common Resources
> ---
>
> Key: YARN-595
> URL: https://issues.apache.org/jira/browse/YARN-595
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch
>
>
> resourcemanager.fair and resourcemanager.resources have two copies of
> basically the same code for operations on Resource objects
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-577) ApplicationReport does not provide progress value of application
[ https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned YARN-577: Assignee: Hitesh Shah (was: Vinod Kumar Vavilapalli) > ApplicationReport does not provide progress value of application > > > Key: YARN-577 > URL: https://issues.apache.org/jira/browse/YARN-577 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Hitesh Shah > > An application sends its progress % to the RM via AllocateRequest. This > should be able to be retrieved by a client via the ApplicationReport. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-556) RM Restart phase 2 - Design for work preserving restart
[ https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-556: Labels: gsoc2013 (was: ) > RM Restart phase 2 - Design for work preserving restart > --- > > Key: YARN-556 > URL: https://issues.apache.org/jira/browse/YARN-556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Bikas Saha > Labels: gsoc2013 > > The basic idea is already documented on YARN-128. This will describe further > details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-556) RM Restart phase 2 - Design for work preserving restart
[
https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640012#comment-13640012
]
Bikas Saha commented on YARN-556:
-
Adding brief description of proposal from YARN-128 design document for Google
Summer of Code.
{noformat}
Work preserving restart - RM will have to make the RMAppAttempt state machine
enter
the Running state before starting the internal services. The RM can obtain
information
about all running containers from the NM’s when the NM’s heartbeat with it.
This
information can be used to repopulate the allocation state of scheduler. When
the
running AM’s heartbeat with RM then the RM can ask them to resend their
container
requests so that the RM can repopulate all the pending requests. Repopulating
the
running container and pending container information completes all the data
needed by
the RM to start normal operations.
{noformat}
> RM Restart phase 2 - Design for work preserving restart
> ---
>
> Key: YARN-556
> URL: https://issues.apache.org/jira/browse/YARN-556
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Labels: gsoc2013
>
> The basic idea is already documented on YARN-128. This will describe further
> details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-605) Failing unit test in TestNMWebServices when using git for source control
Hitesh Shah created YARN-605: Summary: Failing unit test in TestNMWebServices when using git for source control Key: YARN-605 URL: https://issues.apache.org/jira/browse/YARN-605 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Failed tests: testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-605) Failing unit test in TestNMWebServices when using git for source control
[
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640048#comment-13640048
]
Hitesh Shah commented on YARN-605:
--
This seems to fix it:
{code}
diff --git
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-
index d82771b..b1cff2c 100644
---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
+++
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
@@ -76,7 +76,7 @@ public static String getXmlAttrString(Element element, String
name) {
public static void checkStringMatch(String print, String expected, String
got) {
assertTrue(
print + " doesn't match, got: " + got + " expected: " + expected,
-got.matches(expected));
+got.equals(expected));
}
public static void checkStringContains(String print, String expected, String
got) {
{code}
[~tgraves] Any comments on whether matches() is needed? I am guessing something
in the strings ( maybe a '-' ) is causing problems when using a regex match.
> Failing unit test in TestNMWebServices when using git for source control
> -
>
> Key: YARN-605
> URL: https://issues.apache.org/jira/browse/YARN-605
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Hitesh Shah
>
> Failed tests:
> testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2
[jira] [Assigned] (YARN-605) Failing unit test in TestNMWebServices when using git for source control
[ https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned YARN-605: Assignee: Hitesh Shah > Failing unit test in TestNMWebServices when using git for source control > - > > Key: YARN-605 > URL: https://issues.apache.org/jira/browse/YARN-605 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-605.1.patch > > > Failed tests: > testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-605) Failing unit test in TestNMWebServices when using git for source control
[ https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-605: - Attachment: YARN-605.1.patch > Failing unit test in TestNMWebServices when using git for source control > - > > Key: YARN-605 > URL: https://issues.apache.org/jira/browse/YARN-605 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-605.1.patch > > > Failed tests: > testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 > > testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): > hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from > fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, > mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 > expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, > origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum > f89f5c9b9c9d44cf3be5c2686f2d789 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-577) ApplicationReport does not provide progress value of application
[ https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-577: - Attachment: YARN-577.1.patch > ApplicationReport does not provide progress value of application > > > Key: YARN-577 > URL: https://issues.apache.org/jira/browse/YARN-577 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-577.1.patch > > > An application sends its progress % to the RM via AllocateRequest. This > should be able to be retrieved by a client via the ApplicationReport. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-556) RM Restart phase 2 - Design for work preserving restart
[ https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640108#comment-13640108 ] Junping Du commented on YARN-556: - Hi Bikas, beside resending resource request through AM-RM hearbeat, do we think some other way like: putting resource requests to ZK store? > RM Restart phase 2 - Design for work preserving restart > --- > > Key: YARN-556 > URL: https://issues.apache.org/jira/browse/YARN-556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Bikas Saha > Labels: gsoc2013 > > The basic idea is already documented on YARN-128. This will describe further > details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-577) ApplicationReport does not provide progress value of application
[
https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640118#comment-13640118
]
Hadoop QA commented on YARN-577:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580238/YARN-577.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/815//console
This message is automatically generated.
> ApplicationReport does not provide progress value of application
>
>
> Key: YARN-577
> URL: https://issues.apache.org/jira/browse/YARN-577
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
> Attachments: YARN-577.1.patch
>
>
> An application sends its progress % to the RM via AllocateRequest. This
> should be able to be retrieved by a client via the ApplicationReport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-605) Failing unit test in TestNMWebServices when using git for source control
[
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640123#comment-13640123
]
Hadoop QA commented on YARN-605:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580234/YARN-605.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/814//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/814//console
This message is automatically generated.
> Failing unit test in TestNMWebServices when using git for source control
> -
>
> Key: YARN-605
> URL: https://issues.apache.org/jira/browse/YARN-605
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
> Attachments: YARN-605.1.patch
>
>
> Failed tests:
> testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD,
> origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum
> f89f5c9b9c9d44cf3be5c2686f2d789
>
> testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
> hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from
> fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD,
> mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789
> expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2c
[jira] [Commented] (YARN-513) Verify all clients will wait for RM to restart
[ https://issues.apache.org/jira/browse/YARN-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640150#comment-13640150 ] nemon lou commented on YARN-513: What about admin client? refreshQueues,refreshNodes,etc. These will be needed in HA. > Verify all clients will wait for RM to restart > -- > > Key: YARN-513 > URL: https://issues.apache.org/jira/browse/YARN-513 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Xuan Gong > > When the RM is restarting, the NM, AM and Clients should wait for some time > for the RM to come back up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
