[
https://issues.apache.org/jira/browse/SPARK-22826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309794#comment-16309794
]
Aleksander Eskilson commented on SPARK-22826:
---------------------------------------------
This bug would affect 2.3.0, and so would also affect any encoder having fields
as described above, which are pretty common in Avro data. Which means it would
also preclude work in a
[fork|https://github.com/databricks/spark-avro/pull/217] of Spark-Avro that
enables creating datasets from Avro objects. Any chance the PR for this issue
can still make it in time for the final 2.3.0 release so that those Spark-Avro
forks can run over the latest version as we continue working over SPARK-22739?
cc: [~marmbrus]
> [SQL] findWiderTypeForTwo Fails over StructField of Array
> ---------------------------------------------------------
>
> Key: SPARK-22826
> URL: https://issues.apache.org/jira/browse/SPARK-22826
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Aleksander Eskilson
>
> The {{findWiderTypeForTwo}} codepath in Catalyst {{TypeCoercion}} fails when
> applied to to {{StructType}} having the following fields:
> {noformat}
> StructType(StructField("a", ArrayType(StringType, containsNull=true))
> :: Nil),
> StructType(StructField("a", ArrayType(StringType, containsNull=false))
> :: Nil)
> {noformat}
> When in {{findTightestCommonType}}, the function attempts to recursively find
> the tightest common type of two arrays. These two arrays are not equal types
> (since one would admit null elements and the other would not), but
> {{findTightestCommonType}} has no match case for {{ArrayType}} (or
> {{MapType}}), so the
> [get|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L108]
> operation on the dataType of the {{StructField}} throws a
> {{NoSuchElementException}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
