Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15780#discussion_r90172243
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
---
@@ -590,7 +591,11 @@ object ScalaReflection extends ScalaReflection {
"cannot be used as field name\n" +
walkedTypePath.mkString("\n"))
}
- val fieldValue = Invoke(inputObject, fieldName,
dataTypeFor(fieldType))
+ // primitive take only non-null or struct takes non-null object
guarded by isNull
--- End diff --
Hmm, I found something wrong with the combination of `Option` and something
the serializer of which is made by `t if definedByConstructorParams(t)` case
like `Option[Tuple2[Int, String]]` (even at master).
If we add a following test to `ExpressionEncoderSuite` (at master):
```scala
encodeDecodeTest(Option.empty[Tuple2[Int, Sring]], "empty option of tuple2
of int and string")
```
it fails as:
```
Encoded/Decoded data does not match input data
in: None
out: Some((0,null))
types: scala.Tuple2
Encoded Data: [null,null]
Schema: _1#381,_2#382
root
-- _1: integer (nullable = false)
-- _2: string (nullable = true)
```
(BTW, there are no `Option` related tests in `ExpressionEncoderSuite`..)
The failure of `roundtrip in to_json and from_json` would be related to
this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
