[
https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609467#comment-16609467
]
Xiangrui Meng commented on SPARK-25378:
---------------------------------------
I sent a PR to spark-tensorflow-connector at
https://github.com/tensorflow/ecosystem/pull/100 to use the suggested method
from [~hvanhovell].
I won't mark this ticket as resolved. The potential issue is that there are
other data sources relying on this behavior. If that is the case, users won't
be able to migrate to 2.4 before the data source owner published a new version.
If there doesn't exist a simple way to check, maybe we should send a notice to
dev@ list.
> ArrayData.toArray assume UTF8String
> -----------------------------------
>
> Key: SPARK-25378
> URL: https://issues.apache.org/jira/browse/SPARK-25378
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiangrui Meng
> Priority: Critical
>
> The following code works in 2.3.1 but failed in 2.4.0-SNAPSHOT:
> {code}
> import org.apache.spark.sql.catalyst.util._
> import org.apache.spark.sql.types.StringType
> ArrayData.toArrayData(Array("a", "b")).toArray[String](StringType)
> res0: Array[String] = Array(a, b)
> {code}
> In 2.4.0-SNAPSHOT, the error is
> {code}java.lang.ClassCastException: java.lang.String cannot be cast to
> org.apache.spark.unsafe.types.UTF8String
> at
> org.apache.spark.sql.catalyst.util.GenericArrayData.getUTF8String(GenericArrayData.scala:75)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at org.apache.spark.sql.catalyst.util.ArrayData.toArray(ArrayData.scala:178)
> ... 51 elided
> {code}
> cc: [~cloud_fan] [~yogeshg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
