Can you show more code in CreateMasterData ? How do you run your code ?
Thanks On Thu, Apr 2, 2015 at 11:06 AM, ganterm <[email protected]> wrote: > Hello, > > I started to use the dataframe API in Spark 1.3 with Scala. > I am trying to implement a UDF and am following the sample here: > > https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction > meaning > val predict = udf((score: Double) => if (score > 0.5) true else false) > df.select( predict(df("score")) ) > All compiles just fine but when I run it, I get a ClassNotFoundException > (see more details below) > I am sure that I load the data correctly and that I have a field called > "score" with the correct data type. > Do I need to do anything else like registering the function? > > Thanks! > Markus > > Exception in thread "main" org.apache.spark.SparkException: Job aborted due > to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: > Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC): > java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1 > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65) > ... > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
