[
https://issues.apache.org/jira/browse/SPARK-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148622#comment-16148622
]
Fei Chen commented on SPARK-14327:
----------------------------------
I also came across this problem. In TaskScheduler, a thread which holds the
lock is blocked by some unknown factors. Other threads must wait until the lock
is released. As a result, the scheduler delay takes up tens seconds or even
several minutes. How can i solve this problem?
> Scheduler holds locks which cause huge scheulder delays and executor timeouts
> -----------------------------------------------------------------------------
>
> Key: SPARK-14327
> URL: https://issues.apache.org/jira/browse/SPARK-14327
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Chris Bannister
> Attachments: driver.jstack
>
>
> I have a job which after a while in one of its stages grinds to a halt, from
> processing around 300k tasks in 15 minutes to less than 1000 in the next
> hour. The driver ends up using 100% CPU on a single core (out of 4) and the
> executors start failing to receive heartbeat responses, tasks are not
> scheduled and results trickle in.
> For this stage the max scheduler delay is 15 minutes, and the 75% percentile
> is 4ms.
> It appears that TaskScheulderImpl does most of its work whilst holding the
> global synchronised lock for the class, this synchronised lock is shared
> between at least,
> TaskSetManager.canFetchMoreResults
> TaskSchedulerImpl.handleSuccessfulTask
> TaskSchedulerImpl.executorHeartbeatReceived
> TaskSchedulerImpl.statusUpdate
> TaskSchedulerImpl.checkSpeculatableTasks
> This looks to severely limit the latency and throughput of the scheduler, and
> casuses my job to straight up fail due to taking too long.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
