[jira] [Commented] (YARN-80) Support delay scheduling for node locality in MR2's capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551902#comment-13551902 ] Hudson commented on YARN-80: Integrated in Hadoop-Hdfs-0.23-Build #492 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/492/]) YARN-80. Add support for delaying rack-local containers in CapacityScheduler. (acmurthy) (Revision 1432290) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1432290 Files : * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Support delay scheduling for node locality in MR2's capacity scheduler -- Key: YARN-80 URL: https://issues.apache.org/jira/browse/YARN-80 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Todd Lipcon Assignee: Arun C Murthy Fix For: 2.0.2-alpha Attachments: YARN-80.patch, YARN-80.patch The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-80) Support delay scheduling for node locality in MR2's capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-80: -- Fix Version/s: 0.23.6 Support delay scheduling for node locality in MR2's capacity scheduler -- Key: YARN-80 URL: https://issues.apache.org/jira/browse/YARN-80 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Todd Lipcon Assignee: Arun C Murthy Fix For: 2.0.2-alpha, 0.23.6 Attachments: YARN-80.patch, YARN-80.patch The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-319) Submit a job to a queue that not allowed in fairScheduler, client will hold forever.
[ https://issues.apache.org/jira/browse/YARN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552142#comment-13552142 ] shenhong commented on YARN-319: --- Of course, Our version already includes this patch. Submit a job to a queue that not allowed in fairScheduler, client will hold forever. Key: YARN-319 URL: https://issues.apache.org/jira/browse/YARN-319 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.2-alpha Reporter: shenhong Fix For: 2.0.3-alpha Attachments: YARN-319.patch RM use fairScheduler, when client submit a job to a queue, but the queue do not allow the user to submit job it, in this case, client will hold forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-319) Submit a job to a queue that not allowed in fairScheduler, client will hold forever.
[ https://issues.apache.org/jira/browse/YARN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552143#comment-13552143 ] shenhong commented on YARN-319: --- Here is the log of yarn client: 13/01/13 13:18:26 ERROR security.UserGroupInformation: PriviledgedActionException as:yuling.sh cause:java.io.IOException: Failed to run job : User yuling.sh cannot submit applications to queue root.cug-dev-tbdp java.io.IOException: Failed to run job : User yuling.sh cannot submit applications to queue root.cug-dev-tbdp at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1266) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236) at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:112) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Submit a job to a queue that not allowed in fairScheduler, client will hold forever. Key: YARN-319 URL: https://issues.apache.org/jira/browse/YARN-319 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.2-alpha Reporter: shenhong Fix For: 2.0.3-alpha Attachments: YARN-319.patch RM use fairScheduler, when client submit a job to a queue, but the queue do not allow the user to submit job it, in this case, client will hold forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-319) Submit a job to a queue that not allowed in fairScheduler, client will hold forever.
[ https://issues.apache.org/jira/browse/YARN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552144#comment-13552144 ] shenhong commented on YARN-319: --- Here is the log of ResourceManager: 2013-01-13 13:18:26,922 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: User yuling .sh cannot submit applications to queue root.cug-dev-tbdp 2013-01-13 13:18:26,924 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattemp t_1357617565562_0696_01 State change from SUBMITTED to FAILED 2013-01-13 13:18:26,924 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_135761756556 2_0696 State change from SUBMITTED to FAILED 2013-01-13 13:18:26,924 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yuling.sh OPER ATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILE D PERMISSIONS=User yuling.sh cannot submit applications to queue root.cug-dev-tbdpAPPID=application_13 57617565562_0696 2013-01-13 13:18:26,924 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=ap plication_1357617565562_0696,name=Sleep job,user=yuling.sh,queue=cug-dev-tbdp,state=FAILED,trackingUrl=hdpdevrm:5003 0/proxy/application_1357617565562_0696/,appMasterHost=N/A,startTime=1358054306921,finishTime=1358054306924 Submit a job to a queue that not allowed in fairScheduler, client will hold forever. Key: YARN-319 URL: https://issues.apache.org/jira/browse/YARN-319 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.2-alpha Reporter: shenhong Fix For: 2.0.3-alpha Attachments: YARN-319.patch RM use fairScheduler, when client submit a job to a queue, but the queue do not allow the user to submit job it, in this case, client will hold forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
