Example below. The partition columns show up as regular columns.
I'll note that SHOW PARTITIONS works correctly in Spark SQL, so it's aware
of the partitions but it does not show them in DESCRIBE.
----------------------------------------
In Hive: "DESCRIBE pub.inventory_daily"
[['effective_date', 'string', ''],
['listing_skey', 'int', ''],
['car_durable_key', 'int', ''],
['car_id', 'int', ''],
['# Partition Information', 'NULL', 'NULL'],
['# col_name ', 'data_type ', 'comment '],
['', 'NULL', 'NULL'],
['year', 'smallint', ''],
['month', 'smallint', ''],
['day', 'smallint', '']]
In SparkSQL: hc.sql("DESCRIBE pub.inventory_daily").collect()
[Row(col_name=u'effective_date', data_type=u'string', comment=u''),
Row(col_name=u'listing_skey', data_type=u'int', comment=u''),
Row(col_name=u'car_durable_key', data_type=u'int', comment=u''),
Row(col_name=u'car_id', data_type=u'int', comment=u''),
Row(col_name=u'year', data_type=u'smallint', comment=u''),
Row(col_name=u'month', data_type=u'smallint', comment=u''),
Row(col_name=u'day', data_type=u'smallint', comment=u'')]
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Getting-different-DESCRIBE-results-between-SparkSQL-and-Hive-tp25452.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
