Re: Wrong runtime type when using newAPIHadoopFile in Java

2017-03-06 Thread Steve Loughran
On 6 Mar 2017, at 12:30, Nira Amit > wrote: And it's very difficult if it's doing unexpected things. All serialisations do unexpected things. Nobody understands them. Sorry

Re: Wrong runtime type when using newAPIHadoopFile in Java

2017-03-06 Thread Nira Amit
And by the way - I don't want the Avro details to be hidden away from me. The whole purpose of the work I'm doing is to benchmark different serialization tools and strategies. If I want to use Kryo serialization for example, then I need to understand how the API works. And it's very difficult if

Re: Wrong runtime type when using newAPIHadoopFile in Java

2017-03-06 Thread Nira Amit
s for weeks and am getting the impression that very few Java developers >> figured this API out. I posted a question >> <http://stackoverflow.com/questions/41836851/wrong- >> runtime-type-in-rdd-when-reading-from-avro-with-custom-serializer> >> about it in StackOve

Re: Wrong runtime type when using newAPIHadoopFile in Java

2017-03-06 Thread Sean Owen
questions/41834120/override-avroio-default-coder-in-dataflow/ > > > about loading custom types in Google Dataflow got answered within a couple > of days). > > I think this behavior should be considered a bug. > > > > > -- > View this message in context: >

Wrong runtime type when using newAPIHadoopFile in Java

2017-03-06 Thread Nira
bout loading custom types in Google Dataflow got answered within a couple of days). I think this behavior should be considered a bug. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Wrong-runtime-type-when-using-newAPIHadoopFile-in-Jav