Hi all,
I tested this with a Date outside a map and it works fine so I think the
issue is simply for Dates inside Maps. I will create a Jira for this unless
there are objections.
Best regards,
Patrick
On Thu, 28 Jun 2018, 11:53 Patrick McGloin, <mcgloin.patr...@gmail.com>
wrote:
> Consider the following test, which will fail on the final show:
>
> * case class *UnitTestCaseClassWithDateInsideMap(map: Map[Date, Int])
>
> test(*"Test a Date as key in a Map"*) {
> *val *map = *UnitTestCaseClassWithDateInsideMap*(*Map*(Date.*valueOf*(
> *"2018-06-28"*) -> 1))
> *val *options = *Map*(*"timestampFormat" *-> *"yyyy/MM/dd HH:mm:ss.SSS"*,
> *"dateFormat" *-> *"yyyy/MM/dd"*)
> *val *schema = Encoders.*product*
> [UnitTestCaseClassWithDateInsideMap].schema
>
> *val *mapDF = *Seq*(map).toDF()
> *val *jsonDF = mapDF.select(*to_json*(*struct*(mapDF.columns.head,
> mapDF.columns.tail:_*), options))
> jsonDF.show()
>
> *val *jsonString = jsonDF.map(_.getString(0)).collect().head
>
> *val *stringDF = *Seq*(jsonString).toDF(*"json"*)
> *val *parsedDF = stringDF.select(*from_json*(*$"json"*, schema, options))
> parsedDF.show()
> }
>
>
> The result of the line "jsonDF.show()" is as follows:
>
> +---------------------------------------------------+
> |structstojson(named_struct(NamePlaceholder(), map))|
> +---------------------------------------------------+
> | {"map":{"17710":1}}|
> +---------------------------------------------------+
>
> As can be seen the date is not formatted correctly. The error with
> "parsedDF.show()" is:
>
> java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String
> cannot be cast to java.lang.Integer
>
> I have tried adding the options to to_json / from_json but it hasn't
> helped. Am I using the wrong options?
>
> Is there another way to do this?
>
> Best regards,
> Patrick
> This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav
> Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands
> <https://maps.google.com/?q=Gustav+Mahlerlaan+10+(1082+PP)+Amsterdam,+the+Netherlands&entry=gmail&source=g>,
> and is registered in the Commercial Register of Amsterdam under number
> 34334259.
>
