On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava <anshi.shrivast...@exadatum.com> wrote: > I've found a KVStore wrapper which stores all the metrics in a LevelDb > store. This KVStore wrapper is available as a spark-dependency but we cannot > access the metrics directly from spark since they are all private.
I'm not sure what it is you're trying to do exactly, but there's a public REST API that exposes all the data Spark keeps about applications. There's also a programmatic status tracker (SparkContext.statusTracker) that's easier to use from within the running Spark app, but has a lot less info. > Can we use this store to store our own metrics? No. > Also can we retrieve these metrics based on timestamp? Only if the REST API has that feature, don't remember off the top of my head. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
