Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/16697
@CodingCat Thinking more, I propose the following:
There are two changes here :
a) Changing the log message in onDropEvent()
If the queue is getting swamped due to a specific event (type), sampled
message emitted helps here. If the event queue is getting swamping due to slow
downstream processing or general application slowness - then it wont help.
Given the potential confusion, I think reverting onDropEvent() to earlier
code is better.
b) Logging all dropped message.
This is primarily a spark developer feature, and as such the change is
extremely expensive without too much upside. logTrace is a better alternative -
it has zero overhead when not enabled, keeps the user logs clean when there is
a spike and allows for developers to debug and get more info when required.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
