[ https://issues.apache.org/jira/browse/SPARK-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968390#comment-13968390 ]
Mridul Muralidharan commented on SPARK-1453: -------------------------------------------- (d) becomes relevant in case of headless/cron'ed jobs. If the job is user initiated, then I agree, the user would typically kill and restart the job. > Improve the way Spark on Yarn waits for executors before starting > ----------------------------------------------------------------- > > Key: SPARK-1453 > URL: https://issues.apache.org/jira/browse/SPARK-1453 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 1.0.0 > Reporter: Thomas Graves > Assignee: Thomas Graves > > Currently Spark on Yarn just delays a few seconds between when the spark > context is initialized and when it allows the job to start. If you are on a > busy hadoop cluster is might take longer to get the number of executors. > In the very least we could make this timeout a configurable value. Its > currently hardcoded to 3 seconds. > Better yet would be to allow user to give a minimum number of executors it > wants to wait for, but that looks much more complex. -- This message was sent by Atlassian JIRA (v6.2#6252)