[ https://issues.apache.org/jira/browse/SPARK-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972864#comment-13972864 ]
Mridul Muralidharan commented on SPARK-1524: -------------------------------------------- The expectation is to fallback to a previous schedule type in case the higher level is not valid : though this is tricky in general case. Will need to take a look at it - though given that I am tied up with other things, if someone else wants to take a crack, please feel free to do so ! Btw, use of IP's and multiple hostnames for a host is not supported in spark - so that is something that will need to be resolved at the deployment end. > TaskSetManager'd better not schedule tasks which has no preferred executorId > using PROCESS_LOCAL in the first search process > ---------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-1524 > URL: https://issues.apache.org/jira/browse/SPARK-1524 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: YanTang Zhai > Priority: Minor > > ShuffleMapTask is constructed with TaskLocation which has only host not > (host, executorID) pair in DAGScheduler. > When TaskSetManager schedules ShuffleMapTask which has no preferred > executorId using specific execId host and PROCESS_LOCAL locality level, no > tasks match the given locality constraint in the first search process. > We also find that the host used by Scheduler is hostname while the host used > by TaskLocation is IP in our cluster. The tow hosts do not match, that makes > pendingTasksForHost HashMap empty and the finding task process against our > expectation. -- This message was sent by Atlassian JIRA (v6.2#6252)