[jira] [Commented] (SPARK-2350) Master throws NPE

2014-07-02 Thread Andrew Or (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050891#comment-14050891
 ] 

Andrew Or commented on SPARK-2350:
--

In general, if Master dies because of an exception, it automatically restarts 
and the exception message is hidden in the logs. It took a while for 
[~ilikerps] and I to find the exception as we are scrolling through the logs. 

 Master throws NPE
 -

 Key: SPARK-2350
 URL: https://issues.apache.org/jira/browse/SPARK-2350
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 1.1.0
Reporter: Andrew Or
 Fix For: 1.1.0


 ... if we launch a driver and there are more waiting drivers to be launched. 
 This is because we remove from a list while iterating through this.
 Here is the culprit from Master.scala (L487 as of the creation of this JIRA, 
 commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c).
 {code}
 for (driver - waitingDrivers) {
   if (worker.memoryFree = driver.desc.mem  worker.coresFree = 
 driver.desc.cores) {
 launchDriver(worker, driver)
 waitingDrivers -= driver
   }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (SPARK-2350) Master throws NPE

2014-07-02 Thread Andrew Or (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050894#comment-14050894
 ] 

Andrew Or commented on SPARK-2350:
--

This is the root cause of SPARK-2154

 Master throws NPE
 -

 Key: SPARK-2350
 URL: https://issues.apache.org/jira/browse/SPARK-2350
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 1.1.0
Reporter: Andrew Or
 Fix For: 1.1.0


 ... if we launch a driver and there are more waiting drivers to be launched. 
 This is because we remove from a list while iterating through this.
 Here is the culprit from Master.scala (L487 as of the creation of this JIRA, 
 commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c).
 {code}
 for (driver - waitingDrivers) {
   if (worker.memoryFree = driver.desc.mem  worker.coresFree = 
 driver.desc.cores) {
 launchDriver(worker, driver)
 waitingDrivers -= driver
   }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)