It may be helpful to note that I'm running in Yarn cluster mode. My goal is to avoid having to manually distribute the JAR to all of the various nodes as this makes versioning deployments difficult.
On Thu, Jun 25, 2020 at 5:32 PM Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: > Hello. > > I am running Spark 2.4.4. I have implemented a custom metrics producer. It > works well when I run locally, or specify the metrics producer only for the > driver. When I ask for executor metrics I run into ClassNotFoundExceptions > > *Is it possible to pass a metrics JAR via --jars? If so what am I > missing?* > > Deploy driver stats via: > --jars hdfs:///custommetricsprovider.jar > --conf > spark.metrics.conf.driver.sink.metrics.class=org.apache.spark.mycustommetricssink > > However, when I pass the JAR with the metrics provider to executors via: > --jars hdfs:///custommetricsprovider.jar > --conf > spark.metrics.conf.executor.sink.metrics.class=org.apache.spark.mycustommetricssink > > I get ClassNotFoundException: > > 20/06/25 21:19:35 ERROR MetricsSystem: Sink class > org.apache.spark.custommetricssink cannot be instantiated > Exception in thread "main" java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1748) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.custommetricssink > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at org.apache.spark.util.Utils$.classForName(Utils.scala:238) > at > org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198) > at > org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > at > org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:194) > at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:102) > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365) > at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:201) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:221) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > ... 4 more > > Is it possible to pass a metrics JAR via --jars? If so what am I missing? > > Thank you, > > Bryan >
