Hmm, it’s because MAHOUT_HOME is not discovered even though it is set in the environment.
Nothing to do with commons, it’s just that some exceptions leave the classpath messed up and so unable to perform the “finally" On Mar 16, 2015, at 9:01 AM, Pat Ferrel <[email protected]> wrote: Trying to write an app that does the following: sparkConf.set("spark.kryo.referenceTracking", "false") .set("spark.kryoserializer.buffer.mb", "200")// this is default for Mahout optimizer, change it with -D option mc = mahoutSparkContext(masterUrl = master, appName = appName, sparkConf = sparkConf) But I get the following runtime error Connected to the target VM, address: '127.0.0.1:63145', transport: 'socket' Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/common/IOUtils at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:98) at com.fb.spark.SparkJobContext$class.setupSparkContext(SparkJobContext.scala:37) at com.fb.drivers.CooccurrenceDriver$.setupSparkContext(CooccurrenceDriver.scala:35) at com.fb.drivers.CooccurrenceDriver$delayedInit$body.apply(CooccurrenceDriver.scala:61) IOUTils are used in the mahoutSparkContext call and I’ve tried using them in my own code, which works fine. commons.oi is in the Mahout main pom and for good measure I put it in my own app’s build.sbt in an identical manner so I’m a bit baffled. It’s as if something in the mahoutSparkContext call itself is somehow removing the jar with common.io in it. Any ideas?
