Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22609
Task events still might need to be processed - even for live applications
with the UI disabled. See the API that has been mentioned before
(`SparkStatusTracker`) for why. Processing task events is different from
keeping information about individual tasks, though.
I agree it's a regression - but, at the same time, if you set the
"retainedTasks" limit to a low value it should at least help a bit.
> Any implications of making it disk backed ?
I tried that a lot while writing this code, but the disk store is just too
slow for when you have bursts of lots of events. Even with a lot of
improvements I made to this code, I still got dropped events with the disk
store, or high memory usage if I tried to do fancy things like using a separate
writer thread to unblock the listener bus...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
