[GitHub] spark pull request #18741: [Spark-21541][YARN]: Spark Logs show incorrect jo...

[pgandhi999]

GitHub user pgandhi999 opened a pull request:

    https://github.com/apache/spark/pull/18741

    [Spark-21541][YARN]: Spark Logs show incorrect job status for a job that 
does not create SparkContext

    If you run a spark job without creating the SparkSession or SparkContext, 
the spark job logs says it succeeded but yarn says it fails and retries 3 
times. Also, since, Application Master unregisters with Resource Manager and 
exits successfully, it deletes the spark staging directory, so when yarn makes 
subsequent retries, it fails to find the staging directory and thus, the 
retries fail.
    
    Added a flag to check whether user has initialized SparkContext. If it is 
true, we let Application Master unregister with Resource Manager else, we do 
not let AM unregister with RM.
    
    ## How was this patch tested?
    Manually tested the fix.
    Before:
    <img width="1253" alt="screen shot-before" 
src="https://user-images.githubusercontent.com/22228190/28647214-69bf81e2-722b-11e7-9ed0-d416d2bf23be.png";>
    
    After:
    <img width="1319" alt="screen shot-after" 
src="https://user-images.githubusercontent.com/22228190/28647220-70f9eea2-722b-11e7-85c6-e56276b15614.png";>
    
    
    Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pgandhi999/spark SPARK-21541

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18741
    
----
commit 172fc20898896058b7288360eb5292ed9df9d79c
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-21T21:00:22Z

    [SPARK-21503]: Fixed the issue
    
    Added the case ExecutorLostFailure which was previously not there, thus, 
the default case would be executed in which case, task would be marked as 
completed.

commit 81422e0f634c0f06eb2ea29fba4281176a1ab528
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-25T14:54:41Z

    [SPARK-21503][UI]: Adding changes as per comments

commit 55c6c37d09b41ae6914edb5d067e7f2c252ac92a
Author: pgandhi999 <parthkgand...@gmail.com>
Date:   2017-07-26T21:26:27Z

    Merge pull request #1 from apache/master
    
    Apache Spark Pull Request - July 26, 2017

commit f454c8933e07967548095e068063bd313ae4845c
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-26T21:41:16Z

    [SPARK-21541]: Spark Logs show incorrect job status for a job that does not 
create SparkContext
    
    Added a flag to check whether user has initialized Spark Context. If it is 
true, then we let Application Master unregister with Resource Manager else we 
do not.

commit 6b7d5c6e2565c7c4dd97f31fe404c59e73c7474c
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-26T21:58:27Z

    Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job that 
does not create SparkContext"
    
    This reverts commit f454c8933e07967548095e068063bd313ae4845c.
    
    "Merged another issue to this one by mistake"

commit 5c52c72949efdf14a40ff60e8edbe98e011bc084
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-26T22:23:30Z

    [SPARK-21541]: Making Changes
    
    Added a flag to check whether user has initialized SparkContext. If it is 
true, we let Application Master unregister with Resource Manager else, we do 
not let AM unregister with RM.

commit cfd9f71497c5411f697555c9d604923d998cf406
Author: pgandhi <pgan...@yahoo-inc.com>
Date:   2017-07-26T22:40:34Z

    Reverting Commits from Master Branch

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org