[
https://issues.apache.org/jira/browse/SPARK-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133324#comment-15133324
]
Huaxin Gao edited comment on SPARK-13155 at 2/4/16 11:43 PM:
-------------------------------------------------------------
@Wenchen Fan
When I tried this yesterday, I also got the cast Exception as Jia. It seems to
me that the problem is because in ScalaReflection.schemaFor, java Int has
nullable true but scala.Int has nullable false. If I make these two consistent,
then the case Exception will go away. Do you know why the nullable values for
java Int and Scala Int are different?
case t if t <:< typeOf[java.lang.Integer] => Schema(IntegerType, nullable
= true)
.....
case t if t <:< definitions.IntTpe => Schema(IntegerType, nullable =
false)
was (Author: huaxing):
<a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cloud_fan";
class="user-hover" rel="cloud_fan">Wenchen Fan</a>
When I tried this yesterday, I also got the cast Exception as Jia. It seems to
me that the problem is because in ScalaReflection.schemaFor, java Int has
nullable true but scala.Int has nullable false. If I make these two consistent,
then the case Exception will go away. Do you know why the nullable values for
java Int and Scala Int are different?
case t if t <:< typeOf[java.lang.Integer] => Schema(IntegerType, nullable
= true)
.....
case t if t <:< definitions.IntTpe => Schema(IntegerType, nullable =
false)
> add runtime null check when convert catalyst array to external array
> --------------------------------------------------------------------
>
> Key: SPARK-13155
> URL: https://issues.apache.org/jira/browse/SPARK-13155
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Wenchen Fan
>
> {code}
> scala> Seq(("a", Seq(null, new Integer(1)))).toDS().as[(String,
> Array[Int])].collect()
> res5: Array[(String, Array[Int])] = Array((a,Array(0, 1)))
> {code}
> This is wrong, we should throw exception for this case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
