Hi Michael, Yes that was a typo. Thanks for the pointer, it works now.
I had tried that but had one thing off. I had CREATE TEMPORARY TABLE tableName USING parquet OPTIONS ( path '/path/to/file' ) Which caused some issues with the BI tool I'm using, Tableau. It would show the table under tables, but when selected would throw an exception. Removing the "TEMPORARY" appears to address that problem. I do have one remaining concern. If one wants to reflect realtime updates, using this approach, will they show up? I wold think I would need to issue a "REFRESH TABLE tableName" when an update occurs; is that correct? If I have a streaming job that is updating a set of tables, and issues the REFRESH TABLE as the final step in the pipeline do you know if the changes will be reflected in the referenced data by Tableau session? I need to try that next, just thought you or someone on the forum may know answer. Thanks again. -Todd On Mon, Apr 6, 2015 at 4:36 PM, Michael Armbrust <[email protected]> wrote: > Hey Todd, > > In migrating to 1.3.x I see that the >> spark.sql.hive.convertMetastoreParquet is no longer public, so the above no >> longer works. > > > This was probably just a typo, but to be clear, > spark.sql.hive.convertMetastoreParquet is still a supported option and > should work. You are correct that the HiveMetastoreTypes class is now > private (we made a lot of stuff private starting with 1.3 (and the removal > of "alpha") since we are now promising binary compatibility for public > APIs. Your hack seems reasonable, but I'll caution this is not a stable > public API so could break with future upgrades. > > While this will work, is there a better way to achieve this under 1.3.x? > > > If you are only looking for the ability to read this data with Spark SQL > (and not Hive) I suggest you look at the Data Sources API syntax for > creating tables. You don't need to specify the schema at all for self > describing formats like parquet. > > CREATE TABLE tableName > USING parquet > OPTIONS ( > path '/path/to/file' > ) > > Michael > > > On Mon, Apr 6, 2015 at 11:37 AM, Todd Nist <[email protected]> wrote: > >> In 1.2.1 of I was persisting a set of parquet files as a table for use by >> spark-sql cli later on. There was a post here >> <http://apache-spark-user-list.1001560.n3.nabble.com/persist-table-schema-in-spark-sql-tt16297.html#a16311> >> by >> Mchael Armbrust that provide a nice little helper method for dealing with >> this: >> >> /** >> * Sugar for creating a Hive external table from a parquet path. >> */def createParquetTable(name: String, file: String): Unit = { >> import org.apache.spark.sql.hive.HiveMetastoreTypes >> >> val rdd = parquetFile(file) >> val schema = rdd.schema.fields.map(f => s"${f.name} >> ${HiveMetastoreTypes.toMetastoreType(f.dataType)}").mkString(",\n") >> val ddl = s""" >> |CREATE EXTERNAL TABLE $name ( >> | $schema >> |) >> |ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe' >> |STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat' >> |OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat' >> |LOCATION '$file'""".stripMargin >> sql(ddl) >> setConf("spark.sql.hive.convertMetastoreParquet", "true") >> } >> >> In migrating to 1.3.x I see that the >> spark.sql.hive.convertMetastoreParquet is no longer public, so the above no >> longer works. >> >> I can define a helper method that wraps the HiveMetastoreTypes something >> like: >> >> package org.apache.spark.sql.hive >> import org.apache.spark.sql.types.DataType >> >> /** >> * Helper to expose HiveMetastoreTypes hidden by Spark. It is created in >> this name space to make it accessible. >> */ >> object HiveTypeHelper { >> def toDataType(metastoreType: String): DataType = >> HiveMetastoreTypes.toDataType(metastoreType) >> def toMetastoreType(dataType: DataType): String = >> HiveMetastoreTypes.toMetastoreType(dataType) >> } >> >> While this will work, is there a better way to achieve this under 1.3.x? >> >> TIA for the assistance. >> >> -Todd >> > >
