Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/18083
> I am not sure that monitoring (with real metrics) the number of dropped
events really worth it. You just want to know if messages have been dropped
(and having the number in the log is fine).
Even if the absolute number of dropped events doesn't matter that much I
would still like to have this metric: it's simple to implement and being able
to use my existing metrics-based monitoring infrastructure correlate dropped
events with other signals can be helpful.
> For the execution time of message processing it is very interesting, but
not having the by listener or by event type breakdowns (just the global timing)
will not allow to do a fine grained analysis and so not to do improvements.
For now my timing is capturing the total time to process each message,
counting the time to dequeue plus the aggregate time across all of the
listeners. Given the current single-threaded processing strategy this is still
a useful signal, even if not quite as useful as per-listener metrics. I agree
that per-listener metrics would be more useful, though, so let me see if
there's a clean refactoring to get the metrics at the per-listener level.
> So putting the counters in ListenerBus is more appropriate for me. This
will allows to not only monitor the LiveListenerBus, but the other one too
(like: StreamingQueryListenerBus, StreamingListenerBus, ...)
I considered this and I'll look into it, but it's less of a priority for me
given that I'm mostly concerned about perf. bottlenecks in LiveListenerBus
event delivery. The other listener busses don't queue/drop events and the two
that you mentioned are actually wrapping `LiveListenerBus` and are both
listener bus implementations as well as listeners themselves. Thus my cop-out
suggestion is going to be to deal with those in a followup PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
