Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18083#discussion_r118629022
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -226,3 +246,61 @@ private[spark] object LiveListenerBus {
val name = "SparkListenerBus"
}
+private[spark]
+class LiveListenerBusMetrics(queue: LinkedBlockingQueue[_]) extends Source
with Logging {
+ override val sourceName: String = "LiveListenerBus"
+ override val metricRegistry: MetricRegistry = new MetricRegistry
+
+ /**
+ * The total number of events posted to the LiveListenerBus. This is a
count of the total number
+ * of events which have been produced by the application and sent to the
listener bus, NOT a
+ * count of the number of events which have been processed and delivered
to listeners (or dropped
+ * without being delivered).
+ */
+ val numEventsPosted: Counter =
metricRegistry.counter(MetricRegistry.name("numEventsPosted"))
+
+ /**
+ * The total number of events that were dropped without being delivered
to listeners.
+ */
+ val numDroppedEvents: Counter =
metricRegistry.counter(MetricRegistry.name("numEventsDropped"))
+
+ /**
+ * The amount of time taken to post a single event to all listeners.
+ */
+ val eventProcessingTime: Timer =
metricRegistry.timer(MetricRegistry.name("eventProcessingTime"))
+
+ /**
+ * The number of messages waiting in the queue.
+ */
+ val queueSize: Gauge[Int] = {
+ metricRegistry.register(MetricRegistry.name("queueSize"), new
Gauge[Int]{
+ override def getValue: Int = queue.size()
+ })
+ }
+
+ // Guarded by synchronization.
+ private val perListenerClassTimers = mutable.Map[String, Timer]()
+
+ /**
+ * Returns a timer tracking the processing time of the given listener
class.
+ * events processed by that listener. This method is thread-safe.
+ */
+ def getTimerForListener(listener: SparkListenerInterface): Option[Timer]
= {
+ synchronized {
+ val className = listener.getClass.getName
--- End diff --
I'll update the PR description to discuss this per-listener metric.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
