You are converting DataFrame to Dataset[Entry]. DataFrame is Dataset[Row].
mapPertitions works fine with simple Dataset. Just not with DataFrame. On Tue, Aug 2, 2016 at 4:50 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Using spark-shell of master branch: > > scala> case class Entry(id: Integer, name: String) > defined class Entry > > scala> val df = Seq((1,"one"), (2, "two")).toDF("id", "name").as[Entry] > 16/08/02 16:47:01 DEBUG package$ExpressionCanonicalizer: > === Result of Batch CleanExpressions === > !assertnotnull(input[0, scala.Tuple2, true], top level non-flat input > object)._1 AS _1#10 assertnotnull(input[0, scala.Tuple2, true], top level > non-flat input object)._1 > !+- assertnotnull(input[0, scala.Tuple2, true], top level non-flat input > object)._1 +- assertnotnull(input[0, scala.Tuple2, true], top level > non-flat input object) > ! +- assertnotnull(input[0, scala.Tuple2, true], top level non-flat > input object) +- input[0, scala.Tuple2, true] > ! +- input[0, scala.Tuple2, true] > ... > > scala> df.mapPartitions(_.take(1)) > > On Tue, Aug 2, 2016 at 1:55 PM, Dragisa Krsmanovic <dragi...@ticketfly.com > > wrote: > >> I am trying to use mapPartitions on DataFrame. >> >> Example: >> >> import spark.implicits._ >> val df: DataFrame = Seq((1,"one"), (2, "two")).toDF("id", "name") >> df.mapPartitions(_.take(1)) >> >> I am getting: >> >> Unable to find encoder for type stored in a Dataset. Primitive types >> (Int, String, etc) and Product types (case classes) are supported by >> importing spark.implicits._ Support for serializing other types will be >> added in future releases. >> >> Since DataFrame is Dataset[Row], I was expecting encoder for Row to be >> there. >> >> What's wrong with my code ? >> >> >> -- >> >> Dragiša Krsmanović | Platform Engineer | Ticketfly >> >> dragi...@ticketfly.com >> >> @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog | >> facebook.com/ticketfly >> > > -- Dragiša Krsmanović | Platform Engineer | Ticketfly dragi...@ticketfly.com @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog | facebook.com/ticketfly