Re: Custom metrics sink
Just set your custom sink in the org.apache.spark.metrics.sink namespace and configure metrics.properties. Use ConsoleSink as an example. Obviously since it’s private the API may change, but in the meantime that should work… https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala#L28 From: Christopher Piggott Date: Friday, March 16, 2018 at 4:09 PM To: "user@spark.apache.org" Subject: Custom metrics sink Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active. That way you can 'hear' what the cluster is doing and how it's distributing work. I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark. What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime. It's just a toy, for fun. How do you add a custom Sink when these classes don't seem to be exposed? --C
Re: Custom metrics sink
There is a proposal to expose them. See SPARK-14151 From: Christopher Piggott Sent: Friday, March 16, 2018 1:09:38 PM To: user@spark.apache.org Subject: Custom metrics sink Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active. That way you can 'hear' what the cluster is doing and how it's distributing work. I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark. What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime. It's just a toy, for fun. How do you add a custom Sink when these classes don't seem to be exposed? --C
Custom metrics sink
Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active. That way you can 'hear' what the cluster is doing and how it's distributing work. I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark. What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime. It's just a toy, for fun. How do you add a custom Sink when these classes don't seem to be exposed? --C
Re: Custom Metrics Sink
Hi, I was wondering if there've been any responses to this? Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Custom-Metrics-Sink-tp10068p23425.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Custom Metrics Sink
What is the preferred way of adding a custom metrics sink to Spark? I noticed that the Sink Trait has been private since April, so I cannot simply extend Sink in an outside package, but I would like to avoid having to create a custom build of Spark. Is this possible? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Custom-Metrics-Sink-tp10068.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
