Consider the following test, which will fail on the final show:
* case class *UnitTestCaseClassWithDateInsideMap(map: Map[Date, Int])
test(*"Test a Date as key in a Map"*) {
*val *map = *UnitTestCaseClassWithDateInsideMap*(*Map*(Date.*valueOf*(
*"2018-06-28"*) -> 1))
*val *options = *Map*(*"timestampFormat" *-> *"yyyy/MM/dd
HH:mm:ss.SSS"*, *"dateFormat"
*-> *"yyyy/MM/dd"*)
*val *schema = Encoders.*product*
[UnitTestCaseClassWithDateInsideMap].schema
*val *mapDF = *Seq*(map).toDF()
*val *jsonDF = mapDF.select(*to_json*(*struct*(mapDF.columns.head,
mapDF.columns.tail:_*), options))
jsonDF.show()
*val *jsonString = jsonDF.map(_.getString(0)).collect().head
*val *stringDF = *Seq*(jsonString).toDF(*"json"*)
*val *parsedDF = stringDF.select(*from_json*(*$"json"*, schema, options))
parsedDF.show()
}
The result of the line "jsonDF.show()" is as follows:
+---------------------------------------------------+
|structstojson(named_struct(NamePlaceholder(), map))|
+---------------------------------------------------+
| {"map":{"17710":1}}|
+---------------------------------------------------+
As can be seen the date is not formatted correctly. The error with
"parsedDF.show()" is:
java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String
cannot be cast to java.lang.Integer
I have tried adding the options to to_json / from_json but it hasn't
helped. Am I using the wrong options?
Is there another way to do this?
Best regards,
Patrick
This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav
Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands
<https://maps.google.com/?q=Gustav+Mahlerlaan+10+(1082+PP)+Amsterdam,+the+Netherlands&entry=gmail&source=g>,
and is registered in the Commercial Register of Amsterdam under number
34334259.
