[
https://issues.apache.org/jira/browse/SPARK-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust resolved SPARK-12439.
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 10391
[https://github.com/apache/spark/pull/10391]
> Fix toCatalystArray and MapObjects
> ----------------------------------
>
> Key: SPARK-12439
> URL: https://issues.apache.org/jira/browse/SPARK-12439
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Assignee: Apache Spark
> Fix For: 2.0.0
>
>
> In toCatalystArray, we should look at the data type returned by dataTypeFor
> instead of silentSchemaFor, to determine if the element is native type. An
> obvious problem is when the element is Option[Int] class,
> catalsilentSchemaFor will return Int, then we will wrongly recognize the
> element is native type.
> There is another problem when using Option as array element. When we encode
> data like Seq(Some(1), Some(2), None) with encoder, we will use MapObjects to
> construct an array for it later. But in MapObjects, we don't check if the
> return value of lambdaFunction is null or not. That causes a bug that the
> decoded data for Seq(Some(1), Some(2), None) would be Seq(1, 2, -1), instead
> of Seq(1, 2, null).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
