Re: can we plz open up encoder on dataset

oh the map in DataFrame is actually using a RowEncoder. i left it out because it wasn't important: so this doesn't compile: def f[T]: Dataset[T] => Dataset[T] = dataset => { val df = dataset.toDF df.map(row => row)(RowEncoder(df.schema)).as[T] } now this does compile. but i don't like it,

Re: can we plz open up encoder on dataset

Hi Koert, map will take the value that has an implicit Encoder to any value that may or may not have an encoder in scope. That's why I'm asking about the map function to see what it does. Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0

Re: can we plz open up encoder on dataset

the map operation works on DataFrame so it doesn't need an encoder. It could have been any operation on DataFrame. the issue is at the end going back to Dataset[T] using as[T]. this requires an encoder for T which i know i already have since i started with a Dataset[T]. i could add an implicit

Re: can we plz open up encoder on dataset

Hi, Can you show the code from map to reproduce the issue? You can create encoders using Encoders object (I'm using it all over the place for schema generation). Jacek On 25 Jan 2017 10:19 p.m., "Koert Kuipers" wrote: > i often run into problems like this: > > i need to

can we plz open up encoder on dataset

i often run into problems like this: i need to write a Dataset[T] => Dataset[T], and inside i need to switch to DataFrame for a particular operation. but if i do: dataset.toDF.map(...).as[T] i get error: Unable to find encoder for type stored in a Dataset. i know it has an encoder, because i