Thanks for finding this. It appears to be because the jar passed to Spark with classes to be serialized was not updated when some code was refactored. We have a fix under test that will be in the next RC. If you could test the next RC (maybe ready tomorrow) we’d be very grateful.
On Mar 3, 2017, at 12:58 PM, Michael Müller <michael.muel...@condat.de> wrote: > So you are downloading the binary and running the Mahout spark-itemsimilarity > driver from that binary? yes > You say “using the same Spark cluster” How is this setup, an env var like > MASTER=? > Can you supply you you point to the cluster and your CLI for the job? These are my environment settings for Spark and Mahout: export MAHOUT_HOME=/home/aml/mahout/apache-mahout-distribution-0.13.0 #export MAHOUT_LOCAL=true export SPARK_HOME=/home/aml/spark/spark-1.6.3-bin-hadoop2.6 export MASTER=spark://ubuntu:7077 export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre I'm starting the job like this: /home/aml/mahout/apache-mahout-distribution-0.13.0/bin/mahout spark-itemsimilarity --master spark://ubuntu:7077 --input ~/data/rating_200k.csv --output ~/data/rating_200k_output --itemIDColumn 1 --rowIDColumn 0 --sparkExecutorMem 6g And when i change MAHOUT_HOME to point to my Mahout 0.12.2 installation (-> /home/aml/mahout/apache-mahout-distribution-0.12.2) and then start the job like that, it succeeds: /home/aml/mahout/apache-mahout-distribution-0.12.2/bin/mahout spark-itemsimilarity --master spark://ubuntu:7077 --input ~/data/rating_200k.csv --output ~/data/rating_200k_output --itemIDColumn 1 --rowIDColumn 0 --sparkExecutorMem 6g -----Ursprüngliche Nachricht----- Von: Pat Ferrel [mailto:p...@occamsmachete.com] Gesendet: Freitag, 3. März 2017 20:49 An: Michael Müller Cc: user@mahout.apache.org Betreff: Re: 0.13.0-RC not fully compatible with Spark 1.6.3? Thanks, I’ll see if I can reproduce. So you are downloading the binary and running the Mahout spark-itemsimilarity driver from that binary? You say “using the same Spark cluster” How is this setup, an env var like MASTER=? Can you supply you you point to the cluster and your CLI for the job? On Mar 3, 2017, at 1:26 AM, Michael Müller <michael.muel...@condat.de> wrote: Hi all, is Mahout 0.13.0 supposed to work with Spark 1.6.3? I would think so as the master-pom.xml explicitly references Spark 1.6.3. But when I run a spark-itemsimilarity command (on the 0.13.0-RC) against my Spark 1.6.3-standalone cluster, the command fails with: 17/03/03 10:08:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, reco-master): java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) ... Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at scala.Option.map(Option.scala:145) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123) When I run the exactly same command on the 0.12.2 release distribution against the same Spark cluster, the command completes sucessfully. My Environment is: * Ubuntu 14.04 * Oracle-JDK 1.8.0_121 * Spark standalone cluster using this distribution: http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz * Mahout 0.13.0-RC: https://repository.apache.org/content/repositories/orgapachemahout-1034/org/apache/mahout/apache-mahout-distribution/0.13.0/apache-mahout-distribution-0.13.0.tar.gz TIA -- Michael Müller Condat AG, Berlin
