Just set your custom sink in the org.apache.spark.metrics.sink namespace and configure metrics.properties. Use ConsoleSink as an example. Obviously since it’s private the API may change, but in the meantime that should work…
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala#L28 From: Christopher Piggott <cpigg...@gmail.com> Date: Friday, March 16, 2018 at 4:09 PM To: "user@spark.apache.org" <user@spark.apache.org> Subject: Custom metrics sink Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active. That way you can 'hear' what the cluster is doing and how it's distributing work. I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark. What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime. It's just a toy, for fun. How do you add a custom Sink when these classes don't seem to be exposed? --C
