Yesha Vora created SPARK-15847: ---------------------------------- Summary: DecisionTreeRunner example stucks with "NoClassDefFoundError: org/apache/avro/generic/GenericRecord" Key: SPARK-15847 URL: https://issues.apache.org/jira/browse/SPARK-15847 Project: Spark Issue Type: Bug Components: Spark Core Reporter: Yesha Vora
In Spark-2, DecisionTreeRunner example hangs with "NoClassDefFoundError: org/apache/avro/generic/GenericRecord". The same application passes in yarn-cluster mode. I'm hitting this issue with yarn-client mode only. {code} spark-submit --class org.apache.spark.examples.mllib.DecisionTreeRunner --master yarn-client --jars hadoop-lzo-*.jar /xxx/lib/spark-examples_*jar /tmp/sparkMLLInput/sample_libsvm_data.txt{code} {code} 16/05/27 02:37:50 INFO SparkContext: Starting job: countByValue at DecisionTreeRunner.scala:185 Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: org/apache/avro/generic/GenericRecord at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:112) at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:274) at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:259) at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:175) at org.apache.spark.serializer.KryoSerializer.supportsRelocationOfSerializedObjects$lzycompute(KryoSerializer.scala:182) at org.apache.spark.serializer.KryoSerializer.supportsRelocationOfSerializedObjects(KryoSerializer.scala:178) at org.apache.spark.shuffle.sort.SortShuffleManager$.canUseSerializedShuffle(SortShuffleManager.scala:187) at org.apache.spark.shuffle.sort.SortShuffleManager.registerShuffle(SortShuffleManager.scala:99) at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:90) at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:91) at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:234) at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:232) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.dependencies(RDD.scala:232) at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:391) at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:403) at org.apache.spark.scheduler.DAGScheduler.getParentStagesAndId(DAGScheduler.scala:304) at org.apache.spark.scheduler.DAGScheduler.newResultStage(DAGScheduler.scala:339) at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:849) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1626) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.lang.ClassNotFoundException: org.apache.avro.generic.GenericRecord at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) .. 23 more{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org