[
https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Parth Gandhi updated SPARK-21541:
---------------------------------
Description:
If you run a spark job without creating the SparkSession or SparkContext, the
spark job logs says it succeeded but yarn says it fails and retries 3 times.
Also, since, Application Master unregisters with Resource Manager and exits
successfully, it deletes the spark staging directory, so when yarn makes
subsequent retries, it fails to find the staging directory and thus, the
retries fail.
*Steps:*
1. For example, run a pyspark job without creating SparkSession or
SparkContext.
*Example:*
import sys
from random import random
from operator import add
from pyspark import SparkContext
if __name__ == "__main__":
print("hello world")
2. Spark will mark it as FAILED. Got to the UI and check the container logs.
3. You will see the following information in the logs:
spark:
7/07/14 13:22:10 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
17/07/14 13:22:10 INFO ApplicationMaster: Unregistering ApplicationMaster with
SUCCEEDED
But yarn logs will show:
2017-07-14 01:14:33,203 [AsyncDispatcher event handler] INFO
attempt.RMAppAttemptImpl: appattempt_1493735952617_12443844_000001 State change
from FINAL_SAVING to FAILED
was:
If you run a spark job without creating the SparkSession or SparkContext, the
spark job logs says it succeeded but yarn says it fails and retries 3 times.
*Steps:*
1. For example, run a pyspark job without creating SparkSession or
SparkContext.
*Example:*
import sys
from random import random
from operator import add
from pyspark import SparkContext
if __name__ == "__main__":
print("hello world")
2. Spark will mark it as FAILED. Got to the UI and check the container logs.
3. You will see the following information in the logs:
spark:
7/07/14 13:22:10 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
17/07/14 13:22:10 INFO ApplicationMaster: Unregistering ApplicationMaster with
SUCCEEDED
But yarn logs will show:
2017-07-14 01:14:33,203 [AsyncDispatcher event handler] INFO
attempt.RMAppAttemptImpl: appattempt_1493735952617_12443844_000001 State change
from FINAL_SAVING to FAILED
> Spark Logs show incorrect job status for a job that does not create
> SparkContext
> --------------------------------------------------------------------------------
>
> Key: SPARK-21541
> URL: https://issues.apache.org/jira/browse/SPARK-21541
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Parth Gandhi
> Priority: Minor
>
> If you run a spark job without creating the SparkSession or SparkContext, the
> spark job logs says it succeeded but yarn says it fails and retries 3 times.
> Also, since, Application Master unregisters with Resource Manager and exits
> successfully, it deletes the spark staging directory, so when yarn makes
> subsequent retries, it fails to find the staging directory and thus, the
> retries fail.
> *Steps:*
> 1. For example, run a pyspark job without creating SparkSession or
> SparkContext.
> *Example:*
> import sys
> from random import random
> from operator import add
> from pyspark import SparkContext
> if __name__ == "__main__":
> print("hello world")
> 2. Spark will mark it as FAILED. Got to the UI and check the container logs.
> 3. You will see the following information in the logs:
> spark:
> 7/07/14 13:22:10 INFO ApplicationMaster: Final app status: SUCCEEDED,
> exitCode: 0
> 17/07/14 13:22:10 INFO ApplicationMaster: Unregistering ApplicationMaster
> with SUCCEEDED
> But yarn logs will show:
> 2017-07-14 01:14:33,203 [AsyncDispatcher event handler] INFO
> attempt.RMAppAttemptImpl: appattempt_1493735952617_12443844_000001 State
> change from FINAL_SAVING to FAILED
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
