Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16291
Hmm... I took a quick look and I'm not sure I understand exactly what's
going on. It seems you're wrapping each listener with a `ListenerEvenProcessor`
(note the typo), and each processor has its own thread pool for processing
events.
If that's the case, that sounds wrong. Each listener should process events
serially otherwise you risk getting into funny situations like a task end event
being processed before the task start event for the same task.
I think this would benefit from a proper explanation of the changes being
proposed, instead of a bunch of code. How will the listener bus work with the
changes? Where will we have thread pools, where will we have message queues?
Will each listener get its own dedicated thread, or will there be a limit? What
kind of controls will there be to avoid memory pressure? Would it be worth it
to allow certain listeners to have dedicated threads while others share the
same one like the current approach? That kind of thing. Can you write something
up and post it to the bug?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
