There is a proposal to expose them. See SPARK-14151 ________________________________ From: Christopher Piggott <cpigg...@gmail.com> Sent: Friday, March 16, 2018 1:09:38 PM To: user@spark.apache.org Subject: Custom metrics sink
Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active. That way you can 'hear' what the cluster is doing and how it's distributing work. I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark. What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime. It's just a toy, for fun. How do you add a custom Sink when these classes don't seem to be exposed? --C
