[jira] [Updated] (SPARK-3411) Improve load-balancing of concurrently-submitted drivers across workers
[ https://issues.apache.org/jira/browse/SPARK-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangTaoTheTonic updated SPARK-3411: --- Summary: Improve load-balancing of concurrently-submitted drivers across workers (was: Optimize the schedule procedure in Master) > Improve load-balancing of concurrently-submitted drivers across workers > --- > > Key: SPARK-3411 > URL: https://issues.apache.org/jira/browse/SPARK-3411 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: WangTaoTheTonic >Priority: Minor > > If the waiting driver array is too big, the drivers in it will be dispatched > to the first worker we get(if it has enough resources), with or without the > Randomization. > We should do randomization every time we dispatch a driver, in order to > better balance drivers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3411) Improve load-balancing of concurrently-submitted drivers across workers
[ https://issues.apache.org/jira/browse/SPARK-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangTaoTheTonic updated SPARK-3411: --- Description: If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization. We should do randomization every time we dispatch a driver, in order to better balance drivers. Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid it. was: If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization. We should do randomization every time we dispatch a driver, in order to better balance drivers. > Improve load-balancing of concurrently-submitted drivers across workers > --- > > Key: SPARK-3411 > URL: https://issues.apache.org/jira/browse/SPARK-3411 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: WangTaoTheTonic >Priority: Minor > > If the waiting driver array is too big, the drivers in it will be dispatched > to the first worker we get(if it has enough resources), with or without the > Randomization. > We should do randomization every time we dispatch a driver, in order to > better balance drivers. > Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3411) Improve load-balancing of concurrently-submitted drivers across workers
[ https://issues.apache.org/jira/browse/SPARK-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3411: - Affects Version/s: 1.1.0 > Improve load-balancing of concurrently-submitted drivers across workers > --- > > Key: SPARK-3411 > URL: https://issues.apache.org/jira/browse/SPARK-3411 > Project: Spark > Issue Type: Improvement > Components: Deploy >Affects Versions: 1.1.0 >Reporter: WangTaoTheTonic >Assignee: WangTaoTheTonic >Priority: Minor > Fix For: 1.2.0 > > > If the waiting driver array is too big, the drivers in it will be dispatched > to the first worker we get(if it has enough resources), with or without the > Randomization. > We should do randomization every time we dispatch a driver, in order to > better balance drivers. > Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
