My apologizes. I was running this locally and the JAR I was building using Intellij had some issues. This was not related to UDFs. All works fine now.
On Thu, Apr 2, 2015 at 2:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Can you show more code in CreateMasterData ? > > How do you run your code ? > > Thanks > > On Thu, Apr 2, 2015 at 11:06 AM, ganterm <gant...@gmail.com> wrote: >> >> Hello, >> >> I started to use the dataframe API in Spark 1.3 with Scala. >> I am trying to implement a UDF and am following the sample here: >> >> https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction >> meaning >> val predict = udf((score: Double) => if (score > 0.5) true else false) >> df.select( predict(df("score")) ) >> All compiles just fine but when I run it, I get a ClassNotFoundException >> (see more details below) >> I am sure that I load the data correctly and that I have a field called >> "score" with the correct data type. >> Do I need to do anything else like registering the function? >> >> Thanks! >> Markus >> >> Exception in thread "main" org.apache.spark.SparkException: Job aborted >> due >> to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: >> Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC): >> java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1 >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> >> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65) >> ... >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
