[SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

I am playing with spark 2.0 What I tried to test is: Create a UDF in which there is a non serializable object. What I expected is when this UDF is called during materializing the dataFrame where the UDF is used in "select", an task non serializable exception should be thrown. It depends also which

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Yes, it is. You can define a udf like that. Basically, it's a udf Int => Int which is a closure contains a non serializable object. The latter should cause Task not serializable exception. Hao On Mon, Aug 8, 2016 at 5:08 AM, Muthu Jayakumar wrote: > Hello Hao Ren, > > Doesn't the code... > > va

RE: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

[mailto:inv...@gmail.com] Sent: 08 August 2016 09:03 To: Muthu Jayakumar Cc: user ; dev Subject: Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected Yes, it is. You can define a udf like that. Basically, it's a udf Int => Int which is a closure contai

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

@ viavisolutions.com > > > > *From:* Hao Ren [mailto:inv...@gmail.com] > *Sent:* 08 August 2016 09:03 > *To:* Muthu Jayakumar > *Cc:* user ; dev > *Subject:* Re: [SPARK-2.0][SQL] UDF containing non-serializable object > does not work as expected > > > > Yes, it

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Ints are >> serializable? >> >> >> >> Just thinking out loud >> >> >> >> Simon Scott >> >> >> >> Research Developer @ viavisolutions.com >> >> >> >> *From:* Hao Ren [mailto:inv...@gmail.com] >> *Sent:

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

zable? >>> >>> >>> >>> Just thinking out loud >>> >>> >>> >>> Simon Scott >>> >>> >>> >>> Research Developer @ viavisolutions.com >>> >>> >>> >>> *From:*