[ https://issues.apache.org/jira/browse/SPARK-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
WangTaoTheTonic updated SPARK-3411: ----------------------------------- Description: If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization. We should do randomization every time we dispatch a driver, in order to better balance drivers. Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid it. was: If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization. We should do randomization every time we dispatch a driver, in order to better balance drivers. > Improve load-balancing of concurrently-submitted drivers across workers > ----------------------------------------------------------------------- > > Key: SPARK-3411 > URL: https://issues.apache.org/jira/browse/SPARK-3411 > Project: Spark > Issue Type: Improvement > Components: Deploy > Reporter: WangTaoTheTonic > Priority: Minor > > If the waiting driver array is too big, the drivers in it will be dispatched > to the first worker we get(if it has enough resources), with or without the > Randomization. > We should do randomization every time we dispatch a driver, in order to > better balance drivers. > Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org