[
https://issues.apache.org/jira/browse/SPARK-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008325#comment-14008325
]
llai commented on SPARK-1922:
-----------------------------
My table uses avro serde. Checked Hive [LanguageManual
DDL|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable]
and [Avro spec|https://avro.apache.org/docs/1.7.6/spec.html#Names], underscore
is valid.
> hql query throws "RuntimeException: Unsupported dataType" if table has a
> struct field which has a column with underscore in its name
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-1922
> URL: https://issues.apache.org/jira/browse/SPARK-1922
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.0.0
> Reporter: llai
>
> If table A has a struct field A_strct <a_col: string>, when doing an hql
> query like "select", "RuntimeException: Unsupported dataType" is thrown.
> Running a query wiht "sbt/sbt hive/console":
> {code}
> scala> hql("SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1
> limit 10").collect().foreach(x => println(x(1)))
> {code}
> Console output:
> {code}
> 14/05/25 19:50:27 INFO parse.ParseDriver: Parsing command: SELECT utc_time,
> pkg FROM pkg_table where year=2014 and month=1 limit 10
> 14/05/25 19:50:27 INFO parse.ParseDriver: Parse Completed
> 14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for
> batch MultiInstanceRelations
> 14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for
> batch CaseInsensitiveAttributeReferences
> 14/05/25 19:50:28 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://xxxxx
> 14/05/25 19:50:28 INFO hive.metastore: Waiting 1 seconds before next
> connection attempt.
> 14/05/25 19:50:29 INFO hive.metastore: Connected to metastore.
> java.lang.RuntimeException: Unsupported dataType:
> struct<adv_id:string,u_lat:double,u_lon:double>
> at scala.sys.package$.error(package.scala:27)
> at
> org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:219)
> at
> org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:273)
> at
> org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
> at
> org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
