Github user subrotosanyal commented on the issue:
https://github.com/apache/spark/pull/13658
hi @jerryshao
`16/06/13 10:50:35 INFO yarn.ApplicationMaster: Final app status:
SUCCEEDED, exitCode: 0
16/06/13 10:50:35 DEBUG yarn.ApplicationMaster: Done running users class
16/06/13 10:50:42 ERROR yarn.ApplicationMaster: SparkContext did not
initialize after waiting for 500000 ms. Please check earlier log output for
errors. Failing the application.
`
From the above log we can see that the the job has succeeded further, which
is clear by the DEBUG log. In the method
ApplicationMaster#waitForSparkContextInitialized there is a wait for 10 seconds
and within these 10 seconds the job finishes. But, due to some reason the wait
is not notified (not sure why the notification doesn't reach) and once the 10
seconds are over the loop ends with condition that _finished_ flag is not any
more false. At this point the code check if the reference to SparkContext is
null or not which comes out to be null at this point and thus the ERROR log.
Once execution of this method is over ApplicationMaster#runDriver makes the
similar check and marks the job as failed.
The change in pull request is trying to figure out if the job is finished
or not before marking the job as failed in such scenario.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
