[
https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627593#comment-16627593
]
Li Yuanjian commented on SPARK-25527:
-------------------------------------
{quote}
There are no Tasks waiting for completion, and the job just hangs.
{quote}
You can check the driver thread dump at this time, generally speaking driver is
doing some heavy work like commit in this scenario. We need more clues to
confirm this is a bug.
> Job stuck waiting for last stage to start
> -----------------------------------------
>
> Key: SPARK-25527
> URL: https://issues.apache.org/jira/browse/SPARK-25527
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Ran Haim
> Priority: Major
>
> Sometimes it can somehow happen that a job is stuck waiting for the last
> stage to start.
> There are no Tasks waiting for completion, and the job just hangs.
> There are available Executors for the job to run.
> I do not know how to reproduce this, all I know is that it happens randomly
> after couple days of hard load.
> Another thing that might help is that it seems to happen when some tasks fail
> because one or more executors killed (due to memory issues or something).
> Those tasks eventually do get finished by other executors because of retries,
> but the next stage hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
