Zhihui created SPARK-1946: ----------------------------- Summary: Submit stage after executors have been registered Key: SPARK-1946 URL: https://issues.apache.org/jira/browse/SPARK-1946 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.0.0 Reporter: Zhihui
Because creating TaskSetManager and registering executors are asynchronous, in most situation, early stages' tasks run without preferred locality. A simple solution is sleeping few seconds in application, so that executors have enough time to register. A better way is to make DAGScheduler submit stage after a few of executors have been registered by configuration properties. # submit stage only after successfully registered executors arrived the ratio, default value 0 spark.executor.registeredRatio = 0.8 # whatever registeredRatio is arrived, submit stage after the maxRegisteredWaitingTime(millisecond), default value 10000 spark.executor.maxRegisteredWaitingTime = 5000 -- This message was sent by Atlassian JIRA (v6.2#6252)