[ https://issues.apache.org/jira/browse/SPARK-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-12439. -------------------------------------- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10391 [https://github.com/apache/spark/pull/10391] > Fix toCatalystArray and MapObjects > ---------------------------------- > > Key: SPARK-12439 > URL: https://issues.apache.org/jira/browse/SPARK-12439 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Liang-Chi Hsieh > Assignee: Apache Spark > Fix For: 2.0.0 > > > In toCatalystArray, we should look at the data type returned by dataTypeFor > instead of silentSchemaFor, to determine if the element is native type. An > obvious problem is when the element is Option[Int] class, > catalsilentSchemaFor will return Int, then we will wrongly recognize the > element is native type. > There is another problem when using Option as array element. When we encode > data like Seq(Some(1), Some(2), None) with encoder, we will use MapObjects to > construct an array for it later. But in MapObjects, we don't check if the > return value of lambdaFunction is null or not. That causes a bug that the > decoded data for Seq(Some(1), Some(2), None) would be Seq(1, 2, -1), instead > of Seq(1, 2, null). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org