Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

nts are >>> serializable? >>> >>> >>> >>> Just thinking out loud >>> >>> >>> >>> Simon Scott >>> >>> >>> >>> Research Developer @ viavisolutions.com >>> >>> >>> &

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

ott > > > > Research Developer @ viavisolutions.com > > > > *From:* Hao Ren [mailto:inv...@gmail.com] > *Sent:* 08 August 2016 09:03 > *To:* Muthu Jayakumar <bablo...@gmail.com> > *Cc:* user <user@spark.apache.org>; dev <d...@spark.apache.org> > *Subj

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Yes, it is. You can define a udf like that. Basically, it's a udf Int => Int which is a closure contains a non serializable object. The latter should cause Task not serializable exception. Hao On Mon, Aug 8, 2016 at 5:08 AM, Muthu Jayakumar wrote: > Hello Hao Ren, > >

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Hello Hao Ren, Doesn't the code... val add = udf { (a: Int) => a + notSer.value } Mean UDF function that Int => Int ? Thanks, Muthu On Sun, Aug 7, 2016 at 2:31 PM, Hao Ren wrote: > I am playing with spark 2.0 > What I tried to test is: > > Create a UDF in which

[SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

I am playing with spark 2.0 What I tried to test is: Create a UDF in which there is a non serializable object. What I expected is when this UDF is called during materializing the dataFrame where the UDF is used in "select", an task non serializable exception should be thrown. It depends also