Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
@vanzin , @gengliangwang To clarify earlier behavior - if UI is disabled,
any task related metrics were reported as zero: granularity of update was at
the stage level.
UI is typically disabled when application does not want additional
overheads associated with it (users leverage history server to see the
application state) : particularly in multi-tenant settings (like thrift server)
or very large applications (processing PB's of data).
If UI is enabled (default) or if application is not `live` (history
server), task events would be processed.
Unfortunately, application + config, which used to work before 2.3 are now
throwing OOM (typically, the long running tests are run for 7+ days to find any
kerberos related issues - here a LR stress test failed within 28 hours) - and
large part of the heap was occupied by TaskDataWrapper; making this a
regression.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
