Spark 1.3 UDF ClassNotFoundException
Hello,
I started to use the dataframe API in Spark 1.3 with Scala.
I am trying to implement a UDF and am following the sample here:
https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction
meaning
val predict = udf((score: Double) => if (score > 0.5) true else false)
df.select( predict(df("score")) )
All compiles just fine but when I run it, I get a ClassNotFoundException
(see more details below)
I am sure that I load the data correctly and that I have a field called
"score" with the correct data type.
Do I need to do anything else like registering the function?
Thanks!
Markus
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC):
java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
...
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.3 UDF ClassNotFoundException
Can you show more code in CreateMasterData ?
How do you run your code ?
Thanks
On Thu, Apr 2, 2015 at 11:06 AM, ganterm wrote:
> Hello,
>
> I started to use the dataframe API in Spark 1.3 with Scala.
> I am trying to implement a UDF and am following the sample here:
>
> https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction
> meaning
> val predict = udf((score: Double) => if (score > 0.5) true else false)
> df.select( predict(df("score")) )
> All compiles just fine but when I run it, I get a ClassNotFoundException
> (see more details below)
> I am sure that I load the data correctly and that I have a field called
> "score" with the correct data type.
> Do I need to do anything else like registering the function?
>
> Thanks!
> Markus
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure:
> Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC):
> java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
> ...
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
Re: Spark 1.3 UDF ClassNotFoundException
My apologizes. I was running this locally and the JAR I was building
using Intellij had some issues.
This was not related to UDFs. All works fine now.
On Thu, Apr 2, 2015 at 2:58 PM, Ted Yu wrote:
> Can you show more code in CreateMasterData ?
>
> How do you run your code ?
>
> Thanks
>
> On Thu, Apr 2, 2015 at 11:06 AM, ganterm wrote:
>>
>> Hello,
>>
>> I started to use the dataframe API in Spark 1.3 with Scala.
>> I am trying to implement a UDF and am following the sample here:
>>
>> https://spark.apache.org/docs/1.3.0/api/scala/index.html#org.apache.spark.sql.UserDefinedFunction
>> meaning
>> val predict = udf((score: Double) => if (score > 0.5) true else false)
>> df.select( predict(df("score")) )
>> All compiles just fine but when I run it, I get a ClassNotFoundException
>> (see more details below)
>> I am sure that I load the data correctly and that I have a field called
>> "score" with the correct data type.
>> Do I need to do anything else like registering the function?
>>
>> Thanks!
>> Markus
>>
>> Exception in thread "main" org.apache.spark.SparkException: Job aborted
>> due
>> to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure:
>> Lost task 0.3 in stage 6.0 (TID 11, BillSmithPC):
>> java.lang.ClassNotFoundException: test.CreateMasterData$$anonfun$1
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>>
>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
>> ...
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-UDF-ClassNotFoundException-tp22361.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
