[ https://issues.apache.org/jira/browse/SPARK-19849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Cheung resolved SPARK-19849. ---------------------------------- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0 Target Version/s: 2.2.0 > Support ArrayType in to_json function/expression > ------------------------------------------------ > > Key: SPARK-19849 > URL: https://issues.apache.org/jira/browse/SPARK-19849 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.2.0 > Reporter: Hyukjin Kwon > Assignee: Hyukjin Kwon > Fix For: 2.2.0 > > > After SPARK-19595, we could > {code} > import org.apache.spark.sql.functions._ > import org.apache.spark.sql.types._ > val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil)) > Seq(("""[{"a": 1}, {"a": > 2}]""")).toDF("array").select(from_json(col("array"), schema)).show() > {code} > Maybe, it'd be better if we provide a way to read it back via {{to_json}} as > below: > {code} > import org.apache.spark.sql.functions._ > import org.apache.spark.sql.types._ > val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil)) > val df = Seq("""[{"a":1}, {"a":2}]""").toDF("json").select(from_json($"json", > schema).as("array")) > df.show() > // Read back. > df.select(to_json($"array").as("json")).show() > {code} > {code} > +----------+ > | array| > +----------+ > |[[1], [2]]| > +----------+ > +-----------------+ > | json| > +-----------------+ > |[{"a":1},{"a":2}]| > +-----------------+ > {code} > Currently, it throws an exception as below: > {code} > org.apache.spark.sql.AnalysisException: cannot resolve > 'structtojson(`array`)' due to data type mismatch: structtojson requires that > the expression is a struct expression.;; > 'Project [structtojson(array#30, Some(Asia/Seoul)) AS structtojson(array)#45] > +- Project > [jsontostruct(ArrayType(StructType(StructField(a,IntegerType,true)),true), > array#27, Some(Asia/Seoul)) AS array#30] > +- Project [value#25 AS array#27] > +- LocalRelation [value#25] > at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:80) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:72) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org