Spark 1.3 UDF ClassNotFoundException
Hello, I started to use the dataframe API in Spark 1.3 with Scala. I am trying to implement a UDF and am following the sample here: https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction meaning val predict = udf((score: Double) => if (score > 0.5) true else false) df.select( predict(df("score")) ) All compiles just fine but when I run it, I get a ClassNotFoundException (see more details below) I am sure that I load the data correctly and that I have a field called "score" with the correct data type. Do I need to do anything else like registering the function? Thanks! Markus Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC): java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1 at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65) ... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.3 UDF ClassNotFoundException
Can you show more code in CreateMasterData ? How do you run your code ? Thanks On Thu, Apr 2, 2015 at 11:06 AM, ganterm wrote: > Hello, > > I started to use the dataframe API in Spark 1.3 with Scala. > I am trying to implement a UDF and am following the sample here: > > https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction > meaning > val predict = udf((score: Double) => if (score > 0.5) true else false) > df.select( predict(df("score")) ) > All compiles just fine but when I run it, I get a ClassNotFoundException > (see more details below) > I am sure that I load the data correctly and that I have a field called > "score" with the correct data type. > Do I need to do anything else like registering the function? > > Thanks! > Markus > > Exception in thread "main" org.apache.spark.SparkException: Job aborted due > to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: > Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC): > java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1 > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65) > ... > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Spark 1.3 UDF ClassNotFoundException
My apologizes. I was running this locally and the JAR I was building using Intellij had some issues. This was not related to UDFs. All works fine now. On Thu, Apr 2, 2015 at 2:58 PM, Ted Yu wrote: > Can you show more code in CreateMasterData ? > > How do you run your code ? > > Thanks > > On Thu, Apr 2, 2015 at 11:06 AM, ganterm wrote: >> >> Hello, >> >> I started to use the dataframe API in Spark 1.3 with Scala. >> I am trying to implement a UDF and am following the sample here: >> >> https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction >> meaning >> val predict = udf((score: Double) => if (score > 0.5) true else false) >> df.select( predict(df("score")) ) >> All compiles just fine but when I run it, I get a ClassNotFoundException >> (see more details below) >> I am sure that I load the data correctly and that I have a field called >> "score" with the correct data type. >> Do I need to do anything else like registering the function? >> >> Thanks! >> Markus >> >> Exception in thread "main" org.apache.spark.SparkException: Job aborted >> due >> to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: >> Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC): >> java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1 >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> >> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65) >> ... >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org