Re: spark metrics question

Thanks Takeshi, that's exactly what I was looking for. On Fri, Feb 5, 2016 at 12:32 PM, Takeshi Yamamuro wrote: > How about using `spark.jars` to send jars into a cluster? > > On Sat, Feb 6, 2016 at 12:00 AM, Matt K wrote: > >> Yes. And what I'm

Re: spark metrics question

Yes. And what I'm trying to figure out if there's a way to package the jar in such a way that I don't have to install it on every Executor node. On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas wrote: > Hi Matt, > > does the custom class you want to package reports metrics

Re: spark metrics question

How about using `spark.jars` to send jars into a cluster? On Sat, Feb 6, 2016 at 12:00 AM, Matt K wrote: > Yes. And what I'm trying to figure out if there's a way to package the jar > in such a way that I don't have to install it on every Executor node. > > > On Wed, Feb

spark metrics question

Hi guys, I'm looking to create a custom sync based on Spark's Metrics System: https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala If I want to collect metrics from the Driver, Master, and Executor nodes,

Re: spark metrics question

Hi Matt, there is some related work I recently did in IBM Research for visualizing the metrics produced. You can read about it here http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/ We recently opensourced it if you are interested to

Re: spark metrics question

Hi Matt, does the custom class you want to package reports metrics of each Executor? Thanks On 3 February 2016 at 15:56, Matt K wrote: > Thanks for sharing Yiannis, looks very promising! > > Do you know if I can package a custom class with my application, or does > it

Re: spark metrics question

Thanks for sharing Yiannis, looks very promising! Do you know if I can package a custom class with my application, or does it have to be pre-deployed on all Executor nodes? On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas wrote: > Hi Matt, > > there is some related work I