[
https://issues.apache.org/jira/browse/SPARK-48863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-48863.
---------------------------------
> ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be
> cast to class org.apache.spark.sql.catalyst.util.ArrayData when parsing JSON
> with "spark.sql.json.enablePartialResults" enabled
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48863
> URL: https://issues.apache.org/jira/browse/SPARK-48863
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.1, 4.0.0
> Reporter: Ivan Sadikov
> Assignee: Ivan Sadikov
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.2, 4.0.0
>
>
> When enabling "spark.sql.json.enablePartialResults" and running the following
> query:
> {code:java}
> select from_json('{"a":"b","c":"d"}', 'array<struct<a:string, c:int>>'){code}
> the code fails with:
> {code:java}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0
> (TID 4) (ip-10-110-51-101.us-west-2.compute.internal executor driver):
> java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String
> cannot be cast to class org.apache.spark.sql.catalyst.util.ArrayData
> (org.apache.spark.unsafe.types.UTF8String and
> org.apache.spark.sql.catalyst.util.ArrayData are in unnamed module of loader
> 'app')
> at
> org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:172)
> at
> org.apache.spark.sql.catalyst.expressions.JsonToStructs.$anonfun$converter$2(jsonExpressions.scala:831)
> at
> org.apache.spark.sql.catalyst.expressions.JsonToStructs.nullSafeEval(jsonExpressions.scala:893)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:687)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
