[ https://issues.apache.org/jira/browse/SPARK-33134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-33134: ---------------------------- Issue Type: Bug (was: Improvement) > Incorrect nested complex JSON fields raise an exception > ------------------------------------------------------- > > Key: SPARK-33134 > URL: https://issues.apache.org/jira/browse/SPARK-33134 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.2, 3.1.0 > Reporter: Maxim Gekk > Assignee: Maxim Gekk > Priority: Major > Fix For: 3.0.2, 3.1.0 > > > The code below: > {code:scala} > val pokerhand_raw = Seq("""[{"cards": [19], "playerId": > 123456}]""").toDF("events") > val event = new StructType() > .add("playerId", LongType) > .add("cards", ArrayType( > new StructType() > .add("id", LongType) > .add("rank", StringType))) > val pokerhand_events = pokerhand_raw > .select(explode(from_json($"events", ArrayType(event))).as("event")) > pokerhand_events.show > {code} > throw the exception in the PERMISSIVE mode (default): > {code:java} > Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to > org.apache.spark.sql.catalyst.util.ArrayData > at > org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:48) > at > org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:48) > at > org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:195) > at > org.apache.spark.sql.catalyst.expressions.JsonToStructs.$anonfun$converter$2(jsonExpressions.scala:560) > at > org.apache.spark.sql.catalyst.expressions.JsonToStructs.nullSafeEval(jsonExpressions.scala:597) > at > org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:461) > at > org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:313) > at > org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108) > {code} > The same works in Spark 2.4: > {code:scala} > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.4.6 > /_/ > Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265) > ... > scala> pokerhand_events.show() > +-----+ > |event| > +-----+ > +-----+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org