[ https://issues.apache.org/jira/browse/SPARK-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333316#comment-15333316 ]
Takeshi Yamamuro commented on SPARK-14048: ------------------------------------------ Also, I couldn't reproduce on master by using the query above. > Aggregation operations on structs fail when the structs have fields with > special characters > ------------------------------------------------------------------------------------------- > > Key: SPARK-14048 > URL: https://issues.apache.org/jira/browse/SPARK-14048 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.0 > Environment: Databricks w/ 1.6.0 > Reporter: Simeon Simeonov > Labels: sql > Attachments: bug_structs_with_backticks.html > > > Consider a schema where a struct has field names with special characters, > e.g., > {code} > |-- st: struct (nullable = true) > | |-- x.y: long (nullable = true) > {code} > Schema such as these are frequently generated by the JSON schema generator, > which seems to never want to map JSON data to {{MapType}} always preferring > to use {{StructType}}. > In SparkSQL, referring to these fields requires backticks, e.g., > {{st.`x.y`}}. There is no problem manipulating these structs unless one is > using an aggregation function. It seems that, under the covers, the code is > not escaping fields with special characters correctly. > For example, > {code} > select first(st) as st from tbl group by something > {code} > generates > {code} > org.apache.spark.sql.catalyst.util.DataTypeException: Unsupported dataType: > struct<x.y:bigint>. If you have a struct and a field name of it has any > special characters, please use backticks (`) to quote that field name, e.g. > `x+y`. Please note that backtick itself is not supported in a field name. > at > org.apache.spark.sql.catalyst.util.DataTypeParser$class.toDataType(DataTypeParser.scala:100) > at > org.apache.spark.sql.catalyst.util.DataTypeParser$$anon$1.toDataType(DataTypeParser.scala:112) > at > org.apache.spark.sql.catalyst.util.DataTypeParser$.parse(DataTypeParser.scala:116) > at > org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:884) > at > com.databricks.backend.daemon.driver.OutputAggregator$$anonfun$toJsonSchema$1.apply(OutputAggregator.scala:395) > at > com.databricks.backend.daemon.driver.OutputAggregator$$anonfun$toJsonSchema$1.apply(OutputAggregator.scala:394) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.List.foreach(List.scala:318) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at > com.databricks.backend.daemon.driver.OutputAggregator$.toJsonSchema(OutputAggregator.scala:394) > at > com.databricks.backend.daemon.driver.OutputAggregator$.maybeApplyOutputAggregation(OutputAggregator.scala:122) > at > com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation0(OutputAggregator.scala:82) > at > com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation(OutputAggregator.scala:42) > at > com.databricks.backend.daemon.driver.DriverLocal.executeSql(DriverLocal.scala:306) > at > com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:161) > at > com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$3.apply(DriverWrapper.scala:467) > at > com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$3.apply(DriverWrapper.scala:467) > at scala.util.Try$.apply(Try.scala:161) > at > com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:464) > at > com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:365) > at > com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:196) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org