[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2350: - Affects Version/s: (was: 1.1.0) 1.0.0 > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.0.0 >Reporter: Andrew Or >Assignee: Aaron Davidson > Fix For: 0.9.2, 1.0.1, 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > Here is the culprit from Master.scala (L487 as of the creation of this JIRA, > commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c). > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2350: --- Fix Version/s: 0.9.2 > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Andrew Or >Assignee: Aaron Davidson > Fix For: 0.9.2, 1.0.1, 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > Here is the culprit from Master.scala (L487 as of the creation of this JIRA, > commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c). > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2350: --- Assignee: Aaron Davidson > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Andrew Or >Assignee: Aaron Davidson > Fix For: 1.0.1, 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > Here is the culprit from Master.scala (L487 as of the creation of this JIRA, > commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c). > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2350: - Description: ... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. Here is the culprit from Master.scala (L487 as of the creation of this JIRA, commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c). {code} for (driver <- waitingDrivers) { if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) { launchDriver(worker, driver) waitingDrivers -= driver } } {code} was: ... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. {code} for (driver <- waitingDrivers) { if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) { launchDriver(worker, driver) waitingDrivers -= driver } } {code} > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Andrew Or > Fix For: 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > Here is the culprit from Master.scala (L487 as of the creation of this JIRA, > commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c). > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2350: - Description: ... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. {code} for (driver <- waitingDrivers) { if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) { launchDriver(worker, driver) waitingDrivers -= driver } } {code} was:... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Andrew Or > Fix For: 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-2350) Master throws NPE
[ https://issues.apache.org/jira/browse/SPARK-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2350: - Description: ... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. {code} for (driver <- waitingDrivers) { if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) { launchDriver(worker, driver) waitingDrivers -= driver } } {code} was: ... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this. {code} for (driver <- waitingDrivers) { if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) { launchDriver(worker, driver) waitingDrivers -= driver } } {code} > Master throws NPE > - > > Key: SPARK-2350 > URL: https://issues.apache.org/jira/browse/SPARK-2350 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Andrew Or > Fix For: 1.1.0 > > > ... if we launch a driver and there are more waiting drivers to be launched. > This is because we remove from a list while iterating through this. > {code} > for (driver <- waitingDrivers) { > if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= > driver.desc.cores) { > launchDriver(worker, driver) > waitingDrivers -= driver > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)