Yes. And what I'm trying to figure out if there's a way to package the jar in such a way that I don't have to install it on every Executor node.
On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas <johngou...@gmail.com> wrote: > Hi Matt, > > does the custom class you want to package reports metrics of each Executor? > > Thanks > > On 3 February 2016 at 15:56, Matt K <matvey1...@gmail.com> wrote: > >> Thanks for sharing Yiannis, looks very promising! >> >> Do you know if I can package a custom class with my application, or does >> it have to be pre-deployed on all Executor nodes? >> >> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <johngou...@gmail.com> >> wrote: >> >>> Hi Matt, >>> >>> there is some related work I recently did in IBM Research for >>> visualizing the metrics produced. >>> You can read about it here >>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/ >>> We recently opensourced it if you are interested to have a deeper look >>> to it: https://github.com/ibm-research-ireland/sparkoscope >>> >>> Thanks, >>> Yiannis >>> >>> On 3 February 2016 at 13:32, Matt K <matvey1...@gmail.com> wrote: >>> >>>> Hi guys, >>>> >>>> I'm looking to create a custom sync based on Spark's Metrics System: >>>> >>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala >>>> >>>> If I want to collect metrics from the Driver, Master, and Executor >>>> nodes, should the jar with the custom class be installed on Driver, Master, >>>> and Executor nodes? >>>> >>>> Also, on Executor nodes, does the MetricsSystem run inside the >>>> Executor's JVM? >>>> >>>> Thanks, >>>> -Matt >>>> >>> >>> >> >> >> -- >> www.calcmachine.com - easy online calculator. >> > > -- www.calcmachine.com - easy online calculator.
