Trying to write an app that does the following:
sparkConf.set("spark.kryo.referenceTracking", "false")
.set("spark.kryoserializer.buffer.mb", "200")// this is default for
Mahout optimizer, change it with -D option
mc = mahoutSparkContext(masterUrl = master, appName = appName, sparkConf =
sparkConf)
But I get the following runtime error
Connected to the target VM, address: '127.0.0.1:63145', transport: 'socket'
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/mahout/common/IOUtils
at
org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:98)
at
com.fb.spark.SparkJobContext$class.setupSparkContext(SparkJobContext.scala:37)
at
com.fb.drivers.CooccurrenceDriver$.setupSparkContext(CooccurrenceDriver.scala:35)
at
com.fb.drivers.CooccurrenceDriver$delayedInit$body.apply(CooccurrenceDriver.scala:61)
IOUTils are used in the mahoutSparkContext call and I’ve tried using them in my
own code, which works fine.
commons.oi is in the Mahout main pom and for good measure I put it in my own
app’s build.sbt in an identical manner so I’m a bit baffled. It’s as if
something in the mahoutSparkContext call itself is somehow removing the jar
with common.io in it.
Any ideas?
