Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

[Daniel Haviv]

I tried but I'm getting the same error (task not serializable)

> On 25 בספט׳ 2015, at 20:10, Ted Yu  wrote:
> 
> Is the Schema.parse() call expensive ?
> 
> Can you call it in the closure ?
> 
>> On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv 
>>  wrote:
>> Hi,
>> I'm getting a NotSerializableException even though I'm creating all the my 
>> objects from within the closure:
>>  import org.apache.avro.generic.GenericDatumReader
>>  import   java.io.File
>> import org.apache.avro._
>> 
>>  val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>>   val READER = new GenericDatumReader[GenericRecord](schema)
>> 
>> val bd = sc.broadcast(orig_schema.toString)
>> 
>> 
>>  val rdd=sc.binaryFiles("/daniel").map(zibi => {  
>>  val schema_obj =  new Schema.Parser
>>  val schema2 = schema_obj.parse(bd.value)
>> })
>> 
>> I think that the problem is that Schema itself is an abstract class with 
>> static methods (such as Parser).
>> 
>> Am I correct? 
>> How can I overcome it ?
>> 
>> Thank you.
>> Daniel
> 

Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

[Ted Yu]

Is the Schema.parse() call expensive ?

Can you call it in the closure ?

On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:

> Hi,
> I'm getting a NotSerializableException even though I'm creating all the my
> objects from within the closure:
>  import org.apache.avro.generic.GenericDatumReader
>  import   java.io.File
> import org.apache.avro._
>
>  val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>   val READER = new GenericDatumReader[GenericRecord](schema)
>
> val bd = sc.broadcast(orig_schema.toString)
>
>
>  val rdd=sc.binaryFiles("/daniel").map(zibi => {
>  val schema_obj =  new Schema.Parser
>  val schema2 = schema_obj.parse(bd.value)
> })
>
> I think that the problem is that Schema itself is an abstract class with
> static methods (such as Parser).
>
> Am I correct?
> How can I overcome it ?
>
> Thank you.
> Daniel
>