Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/23088
Hi @srowen . Yes. Currently for disk store case, we need to have a more
optimized code.
> While it makes some sense I have two concerns: different answers based on
disk vs memory store which shouldn't really affect things. But would a user
ever have both and see both side by side and be confused?
This configurable store is only for history server. So user can configure
either one at a time for history server. But, the live UI (which open from Yarn
UI), also goes through the same code flow. Where it has 'ElementTrackingStore',
which is also inMemory. So, if a user configure disk store for History server
and open both live and inProgress History UI, the summary metrics will be
different.
> changing the way the indexing works, so that you can index by specific
metrics for successful and failed tasks differently, would be tricky, and also
would require changing the disk store version (to invalidate old stores).
I think @vanzin suggestion seems work, but need time to give it a try and
to test it. May be we can add as "TODO" for diskStore case or open a seperate
JIRA for that.
> Second is, that seems like it should still entail pushing down all the
quantile logic into the KVStore, to be clean, right? and that's a bigger change.
Thanks @srowen for the suggestion. Probably @vanzin can answer this well.
I have modified the code, for InMemory case. Disk store still uses the old
code.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
