[ https://issues.apache.org/jira/browse/SPARK-24391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594479#comment-16594479 ]
Hyukjin Kwon commented on SPARK-24391: -------------------------------------- For to_json, separate JIRA was filed in SPARK-25252 > from_json should support arrays of primitives, and more generally all JSON > --------------------------------------------------------------------------- > > Key: SPARK-24391 > URL: https://issues.apache.org/jira/browse/SPARK-24391 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Sam Kitajima-Kimbrel > Assignee: Maxim Gekk > Priority: Major > Fix For: 2.4.0 > > > https://issues.apache.org/jira/browse/SPARK-19849 and > https://issues.apache.org/jira/browse/SPARK-21513 brought support for more > column types to functions.to_json/from_json, but I also have cases where I'd > like to simply (de)serialize an array of primitives to/from JSON when > outputting to certain destinations, which does not work: > {code:java} > scala> import org.apache.spark.sql.functions._ > import org.apache.spark.sql.functions._ > scala> import spark.implicits._ > import spark.implicits._ > scala> val df = Seq("[1, 2, 3]").toDF("a") > df: org.apache.spark.sql.DataFrame = [a: string] > scala> val schema = new ArrayType(IntegerType, false) > schema: org.apache.spark.sql.types.ArrayType = ArrayType(IntegerType,false) > scala> df.select(from_json($"a", schema)) > org.apache.spark.sql.AnalysisException: cannot resolve 'jsontostructs(`a`)' > due to data type mismatch: Input schema array<int> must be a struct or an > array of structs.;; > 'Project [jsontostructs(ArrayType(IntegerType,false), a#3, > Some(America/Los_Angeles)) AS jsontostructs(a)#10] > scala> val arrayDf = Seq(Array(1, 2, 3)).toDF("arr") > arrayDf: org.apache.spark.sql.DataFrame = [arr: array<int>] > scala> arrayDf.select(to_json($"arr")) > org.apache.spark.sql.AnalysisException: cannot resolve 'structstojson(`arr`)' > due to data type mismatch: Input type array<int> must be a struct, array of > structs or a map or array of map.;; > 'Project [structstojson(arr#19, Some(America/Los_Angeles)) AS > structstojson(arr)#26] > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org