[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105133#comment-16105133 ]
Thomas Graves commented on SPARK-21541: --------------------------------------- it was merged https://github.com/apache/spark/commit/69ab0e4bddccb461f960fcb48a390a1517e504dd but I guess the pr link didn't pick it up. I missed that the title wasn't quite right [Spark-21541] so perhaps jira didn't pick it up properly. > Spark Logs show incorrect job status for a job that does not create > SparkContext > -------------------------------------------------------------------------------- > > Key: SPARK-21541 > URL: https://issues.apache.org/jira/browse/SPARK-21541 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 2.2.0 > Reporter: Parth Gandhi > Assignee: Parth Gandhi > Priority: Minor > Fix For: 2.3.0 > > > If you run a spark job without creating the SparkSession or SparkContext, the > spark job logs says it succeeded but yarn says it fails and retries 3 times. > Also, since, Application Master unregisters with Resource Manager and exits > successfully, it deletes the spark staging directory, so when yarn makes > subsequent retries, it fails to find the staging directory and thus, the > retries fail. > *Steps:* > 1. For example, run a pyspark job without creating SparkSession or > SparkContext. > *Example:* > import sys > from random import random > from operator import add > from pyspark import SparkContext > if __name__ == "__main__": > print("hello world") > 2. Spark will mark it as FAILED. Got to the UI and check the container logs. > 3. You will see the following information in the logs: > spark: > 7/07/14 13:22:10 INFO ApplicationMaster: Final app status: SUCCEEDED, > exitCode: 0 > 17/07/14 13:22:10 INFO ApplicationMaster: Unregistering ApplicationMaster > with SUCCEEDED > But yarn logs will show: > 2017-07-14 01:14:33,203 [AsyncDispatcher event handler] INFO > attempt.RMAppAttemptImpl: appattempt_1493735952617_12443844_000001 State > change from FINAL_SAVING to FAILED -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org