EMR Spark Custom Metrics

[Mark Kelly]

Hi,

So i would like some custom metrics.

The environment we use is AWS EMR 4.5.0 with spark 1.6.1 and Ganglia.

the code snippit below shows how we register custom metrics (this worked in EMR 
4.2.0 with spark 1.5.2)

package org.apache.spark.metrics.source

import com.codahale.metrics._
import org.apache.spark.{Logging, SparkEnv}

class SparkInstrumentation(val prefix: String) extends Serializable with 
Logging {

  class InstrumentationSource(val prefix: String) extends Source {

    log.info <http://log.info/>(s"Starting spark instrumentation with prefix 
$prefix")
    override val sourceName = prefix
    override val metricRegistry = new MetricRegistry()

    def registerCounter(name: String): Counter = {
      metricRegistry.counter(MetricRegistry.name(name))
    }

    def registerTimer(name: String): Timer = {
      metricRegistry.timer(MetricRegistry.name(name))
    }
  }

  val source = new InstrumentationSource(prefix)

  def register() {
    SparkEnv.get.metricsSystem.registerSource(source)
  }

}

This unfortunately no longer works.

How is it possible to create custom metrics?

Thanks