Karim Ramadan created SPARK-50365:
-------------------------------------
Summary: Row.jsonValue is not able to serialize LocalDateTime
Key: SPARK-50365
URL: https://issues.apache.org/jira/browse/SPARK-50365
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.3
Reporter: Karim Ramadan
Trying to serialize to JSON a Row containing a column of type
_TimestampNTZType_ ({_}LocalDateTime{_}) with methods:
{code:java}
def json: String = compact(jsonValue)
def prettyJson: String = pretty(render(jsonValue))
{code}
Will fail with:
{code:java}
[FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the
class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE"
in the JSON format. SQLSTATE: 2203G
org.apache.spark.SparkIllegalArgumentException: [FAILED_ROW_TO_JSON] Failed to
convert the row value '2018-05-14T12:13' of the class class
java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON
format. SQLSTATE: 2203G
at org.apache.spark.sql.Row.toJson$1(Row.scala:663)
at org.apache.spark.sql.Row.toJson$1(Row.scala:651)
at org.apache.spark.sql.Row.jsonValue(Row.scala:665)
at org.apache.spark.sql.Row.jsonValue$(Row.scala:598)
at
org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:28)
at
org.apache.spark.sql.RowJsonSuite.$anonfun$testJson$1(RowJsonSuite.scala:41)
{code}
This happens because the toJson method inside of the class is missing a
serializer for the LocalDateTime type
{code:java}
def toJson(value: Any, dataType: DataType): JValue = (value, dataType)
match {
case (null, _) => JNull
case (b: Boolean, _) => JBool(b)
case (b: Byte, _) => JLong(b)
case (s: Short, _) => JLong(s)
case (i: Int, _) => JLong(i)
case (l: Long, _) => JLong(l)
...
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
