You should use :
import org.apache.spark.sql.catalyst.encoders.RowEncoder
val df = spark.read.parquet(fileName)
implicit val encoder: ExpressionEncoder[Row] = RowEncoder(df.schema)
val df1 = df.flatMap { x => List(x) }
> On Jul 23, 2016, at 22:01, Julien Nauroy <julien.nau...@u-psud.fr> wrote:
>
> Thanks for your quick reply.
>
> I've tried with this encoder:
> implicit def RowEncoder: org.apache.spark.sql.Encoder[Row] =
> org.apache.spark.sql.Encoders.kryo[Row]
> Using a suggestion from
> http://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-a-dataset-in-spark-1-6
>
> <http://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-a-dataset-in-spark-1-6>
>
> How did you setup your encoder?
>
>
> De: "Sun Rui" <sunrise_...@163.com>
> À: "Julien Nauroy" <julien.nau...@u-psud.fr>
> Cc: user@spark.apache.org
> Envoyé: Samedi 23 Juillet 2016 15:55:21
> Objet: Re: Using flatMap on Dataframes with Spark 2.0
>
> I did a try. the schema after flatMap is the same, which is expected.
>
> What’s your Row encoder?
> On Jul 23, 2016, at 20:36, Julien Nauroy <julien.nau...@u-psud.fr
> <mailto:julien.nau...@u-psud.fr>> wrote:
>
> Hi,
>
> I'm trying to call flatMap on a Dataframe with Spark 2.0 (rc5).
> The code is the following:
> var data = spark.read.parquet(fileName).flatMap(x => List(x))
>
> Of course it's an overly simplified example, but the result is the same.
> The dataframe schema goes from this:
> root
> |-- field1: double (nullable = true)
> |-- field2: integer (nullable = true)
> (etc)
>
> to this:
> root
> |-- value: binary (nullable = true)
>
> Plus I have to provide an encoder for Row.
> I expect to get the same schema after calling flatMap.
> Any idea what I could be doing wrong?
>
>
> Best regards,
> Julien
>
>
>
>
