[
https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119275#comment-16119275
]
Imran Rashid edited comment on SPARK-18838 at 8/9/17 1:24 AM:
--------------------------------------------------------------
+1 to Marcelo's request for logs from cases where listener bus dropping events
leads to spark hanging. I wouldn't expect that either (given his caveats above
-- no Dynamic Allocation and UI-weirdness).
[~milesc] -- from the log snippet you posted, it looks like ~40k messages get
dropped in a minute. I expect most messages are related to tasks starting &
completing, so I assume the number of tasks you have is in the same ballpark?
for those considering experimenting with making it blocking ([~dirkraft]) --
one relatively simple alternative to just making the queue blocking would be to
instead stop the TaskSchedulerImpl from scheduling any more tasks when the
listener bus is over half full. I wouldn't want to keep scheduling tasks if
the listener bus is full, because then you'll just create a huge backlog in
other parts of the scheduler as they all wait to get things on the listener
bus. This would help avoid some of the increased memory usage that Marcelo was
mentioning above. Obviously, it'll still decrease scheduler throughput, but
maybe that's OK. Also this is just an idea off the top of my head, so YMMV,
but as long as you're experimenting with other approaches ...
was (Author: irashid):
+1 to Marcelo's request for logs from cases where listener bus dropping events
leads to spark hanging. I wouldn't expect that either (given his caveats above
-- no Dynamic Allocation and UI-weirdness).
[~milesc] -- from the log snippet you posted, it looks like ~40k messages get
dropped in a minute. I expect most messages are related to tasks starting &
completing, so I assume the number of tasks you have is in the same ballpark?
for those considering experimenting with making it blocking ([~dirkraft]) --
one relatively simple alternative to just making the queue blocking would be to
instead stop the TaskSchedulerImpl from scheduling any more tasks when the
listener bus is over half full. I wouldn't want to keep scheduling tasks if
the listener bus is full, because then you'll just create a huge backlog in
other parts of the scheduler as you they all wait to get things on the listener
bus. This would help avoid some of the increased memory usage that Marcelo was
mentioning above. Obviously, it'll still decrease scheduler throughput, but
maybe that's OK. Also this is just an idea off the top of my head, so YMMV,
but as long as you're experimenting with other approaches ...
> High latency of event processing for large jobs
> -----------------------------------------------
>
> Key: SPARK-18838
> URL: https://issues.apache.org/jira/browse/SPARK-18838
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Sital Kedia
> Attachments: perfResults.pdf, SparkListernerComputeTime.xlsx
>
>
> Currently we are observing the issue of very high event processing delay in
> driver's `ListenerBus` for large jobs with many tasks. Many critical
> component of the scheduler like `ExecutorAllocationManager`,
> `HeartbeatReceiver` depend on the `ListenerBus` events and this delay might
> hurt the job performance significantly or even fail the job. For example, a
> significant delay in receiving the `SparkListenerTaskStart` might cause
> `ExecutorAllocationManager` manager to mistakenly remove an executor which is
> not idle.
> The problem is that the event processor in `ListenerBus` is a single thread
> which loops through all the Listeners for each event and processes each event
> synchronously
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala#L94.
> This single threaded processor often becomes the bottleneck for large jobs.
> Also, if one of the Listener is very slow, all the listeners will pay the
> price of delay incurred by the slow listener. In addition to that a slow
> listener can cause events to be dropped from the event queue which might be
> fatal to the job.
> To solve the above problems, we propose to get rid of the event queue and the
> single threaded event processor. Instead each listener will have its own
> dedicate single threaded executor service . When ever an event is posted, it
> will be submitted to executor service of all the listeners. The Single
> threaded executor service will guarantee in order processing of the events
> per listener. The queue used for the executor service will be bounded to
> guarantee we do not grow the memory indefinitely. The downside of this
> approach is separate event queue per listener will increase the driver memory
> footprint.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
