Hello Everyone, I have my parquet files stored on HDFS. I am trying to create a table in Hive Metastore from Spark SQL. I have an Avro schema file from which I generated the parquet files.
I am doing the following to create the table. 1) Firstly create an Avro dummy table from the schema file. spark.sql(""" CREATE TABLE db_test.avro_test ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ('avro.schema.url'='/avro-schema/schema.avsc')""") This step is successful and I have a table created in hive-metastore. 2) Now create an external table with the same schema as the first one and with location pointing to parquet files directory. spark.sql(“”” CREATE EXTERNAL TABLE db_test.parquet_test LIKE db_test.avro_test STORED AS PARQUET LOCATION ‘/parquet-data-dir’ “””) This step is failing. Looks like Spark SQL doesn’t like the word “LIKE” in the create statement. The same statement works fine from the Hive shell. *Can someone please help me to with the parquet table creation from the Avro Schema? * Is this a bug in spark sql that it doesn't parse "LIKE"? Here is the error that the spark is throwing. Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'LIKE' expecting <EOF>(line 1, pos 136) == SQL == CREATE EXTERNAL TABLE db_test.parquet_test LIKE db_test.avro_test STORED AS PARQUET LOCATION ‘/parquet-data-dir’ ---------------------------------------------------------------------^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:239) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:115) -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org