I was expecting to get the same results with both:
dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
and
dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
"my_table")
In the first example, it behaves as expected. It creates a new tabl
,
>
> Xiao
>
>
> 2016-07-06 10:05 GMT-07:00 Dragisa Krsmanovic :
>
>> I was expecting to get the same results with both:
>>
>> dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
>>
>> and
>>
>> dataFrame.write.mod
I am trying to use mapPartitions on DataFrame.
Example:
import spark.implicits._
val df: DataFrame = Seq((1,"one"), (2, "two")).toDF("id", "name")
df.mapPartitions(_.take(1))
I am getting:
Unable to find encoder for type stored in a Dataset. Primitive types (Int,
String, etc) and Product types
rue], top level
> non-flat input object)
> ! +- assertnotnull(input[0, scala.Tuple2, true], top level non-flat
> input object)+- input[0, scala.Tuple2, true]
> ! +- input[0, scala.Tuple2, true]
> ...
>
> scala> df.mapPartitions(_.take(1))
>
> On Tue, Aug
Perfect ! That's what I was looking for.
Thanks Sun !
On Tue, Aug 2, 2016 at 6:58 PM, Sun Rui wrote:
> import org.apache.spark.sql.catalyst.encoders.RowEncoder
> implicit val encoder = RowEncoder(df.schema)
> df.mapPartitions(_.take(1))
>
> On Aug 3, 2016, at 04:55, Dragisa
