Is the Schema.parse() call expensive ?
Can you call it in the closure ?
On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> Hi,
> I'm getting a NotSerializableException even though I'm creating all the my
> objects from within the closure:
> import org.apache.avro.generic.GenericDatumReader
> import java.io.File
> import org.apache.avro._
>
> val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
> val READER = new GenericDatumReader[GenericRecord](schema)
>
> val bd = sc.broadcast(orig_schema.toString)
>
>
> val rdd=sc.binaryFiles("/daniel").map(zibi => {
> val schema_obj = new Schema.Parser
> val schema2 = schema_obj.parse(bd.value)
> })
>
> I think that the problem is that Schema itself is an abstract class with
> static methods (such as Parser).
>
> Am I correct?
> How can I overcome it ?
>
> Thank you.
> Daniel
>
