Re: Dataset API and avro type

if you are using the kryo encoder, you can only use it to to map to/from kryo encoded binary data. This is because spark does not understand kryo's encoding, its just using it as an opaque blob of bytes. On Mon, May 23, 2016 at 1:28 AM, Han JU wrote: > Just one more

Re: Dataset API and avro type

Just one more question: does Dataset suppose to be able to cast data to an avro type? For a very simple format (a string and a long), I can cast it to a tuple or case class, but not an avro type (also contains only a string and a long). The error is like this for this very simple type: ===

Re: Dataset API and avro type

What is the error? I would definitely expect it to work with kryo at least. On Fri, May 20, 2016 at 2:37 AM, Han JU wrote: > Hello, > > I'm looking at the Dataset API in 1.6.1 and also in upcoming 2.0. However > it does not seems to work with Avro data types: > > >

Dataset API and avro type

Hello, I'm looking at the Dataset API in 1.6.1 and also in upcoming 2.0. However it does not seems to work with Avro data types: object Datasets extends App { val conf = new SparkConf() conf.setAppName("Dataset") conf.setMaster("local[2]") conf.setIfMissing("spark.serializer",