Github user subrotosanyal commented on the issue: https://github.com/apache/spark/pull/13658 hi @jerryshao `16/06/13 10:50:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0 16/06/13 10:50:35 DEBUG yarn.ApplicationMaster: Done running users class 16/06/13 10:50:42 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 500000 ms. Please check earlier log output for errors. Failing the application. ` From the above log we can see that the the job has succeeded further, which is clear by the DEBUG log. In the method ApplicationMaster#waitForSparkContextInitialized there is a wait for 10 seconds and within these 10 seconds the job finishes. But, due to some reason the wait is not notified (not sure why the notification doesn't reach) and once the 10 seconds are over the loop ends with condition that _finished_ flag is not any more false. At this point the code check if the reference to SparkContext is null or not which comes out to be null at this point and thus the ERROR log. Once execution of this method is over ApplicationMaster#runDriver makes the similar check and marks the job as failed. The change in pull request is trying to figure out if the job is finished or not before marking the job as failed in such scenario.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org