Github user bdrillard commented on a diff in the pull request:
https://github.com/apache/spark/pull/20010#discussion_r157787536
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -99,6 +99,17 @@ object TypeCoercion {
case (_: TimestampType, _: DateType) | (_: DateType, _: TimestampType)
=>
Some(TimestampType)
+ case (t1 @ ArrayType(pointType1, nullable1), t2 @
ArrayType(pointType2, nullable2))
+ if t1.sameType(t2) =>
+ val dataType = findTightestCommonType(pointType1, pointType2).get
--- End diff --
@gczsjdy `sameType` will ignore the nullability of the two types, only
checking if the `DataType` is the same. We would expect then that the first
case of `findTightestCommonType` would pass the types on through, but during
its equality check, it also checks the type nullability, which for ArrayType
and MapType, may differ between t1 and t2:
```
case (t1, t2) if t1 == t1 => Some(t1)
```
That means we can establish that two StructFields of ArrayType/MapType are
the same DataType, but if they have different nullability, then the above case
match won't match them, nor will any other case in the match set. So in order
to find the tightest common type where nullabilities of the point-types may
differ, we'll need to recurse. See a case like:
```
widenTest(
StructType(StructField("a", ArrayType(StringType, containsNull=true))
:: Nil),
StructType(StructField("a", ArrayType(StringType, containsNull=false))
:: Nil),
Some(StructType(StructField("a", ArrayType(StringType,
containsNull=true)) :: Nil)))
```
At the moment, since we have no case to match ArrayType/MapType where the
nullabilities may differ, this case would fail. I can add this test explicitly
to the suite.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
