Thanks for JIRA ticket reference! Frankly, I was aware of this work, but didn't know that there was an API for storage implementation.
Will try exploring that as well, thanks! On Wed, 19 Jul 2017 at 4:18 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > See SPARK-18085. That has much of the same goals re: SHS resource > usage, and also provides a (currently non-public) API where you could > just create a MongoDB implementation if you want. > > On Tue, Jul 18, 2017 at 12:56 AM, Ivan Sadikov <ivan.sadi...@gmail.com> > wrote: > > Hello everyone! > > > > I have been working on Spark history server that uses MongoDB as a > datastore > > for processed events to iterate on idea that Spree project uses for Spark > > UI. Project was originally designed to improve on standalone history > server > > with reduced memory footprint. > > > > Project lives here: https://github.com/lightcopy/history-server > > > > These are just very early days of the project, sort of pre-alpha (some > > features are missing, and metrics in some failed jobs cases are > > questionable). Code is being tested on several 8gb and 2gb logs and aims > to > > lower resource usage since we run history server together with several > other > > systems. > > > > Would greatly appreciate any feedback on repository (issues/pull > > requests/suggestions/etc.). Thanks a lot! > > > > > > Cheers, > > > > Ivan > > > > > > -- > Marcelo >
