Hey Todd,
In migrating to 1.3.x I see that the spark.sql.hive.convertMetastoreParquet
is no longer public, so the above no longer works.
This was probably just a typo, but to be clear,
spark.sql.hive.convertMetastoreParquet is still a supported option and
should work. You are correct that the HiveMetastoreTypes class is now
private (we made a lot of stuff private starting with 1.3 (and the removal
of alpha) since we are now promising binary compatibility for public
APIs. Your hack seems reasonable, but I'll caution this is not a stable
public API so could break with future upgrades.
While this will work, is there a better way to achieve this under 1.3.x?
If you are only looking for the ability to read this data with Spark SQL
(and not Hive) I suggest you look at the Data Sources API syntax for
creating tables. You don't need to specify the schema at all for self
describing formats like parquet.
CREATE TABLE tableName
USING parquet
OPTIONS (
path '/path/to/file'
)
Michael
On Mon, Apr 6, 2015 at 11:37 AM, Todd Nist tsind...@gmail.com wrote:
In 1.2.1 of I was persisting a set of parquet files as a table for use by
spark-sql cli later on. There was a post here
http://apache-spark-user-list.1001560.n3.nabble.com/persist-table-schema-in-spark-sql-tt16297.html#a16311
by
Mchael Armbrust that provide a nice little helper method for dealing with
this:
/**
* Sugar for creating a Hive external table from a parquet path.
*/def createParquetTable(name: String, file: String): Unit = {
import org.apache.spark.sql.hive.HiveMetastoreTypes
val rdd = parquetFile(file)
val schema = rdd.schema.fields.map(f = s${f.name}
${HiveMetastoreTypes.toMetastoreType(f.dataType)}).mkString(,\n)
val ddl = s
|CREATE EXTERNAL TABLE $name (
| $schema
|)
|ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
|STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat'
|OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat'
|LOCATION '$file'.stripMargin
sql(ddl)
setConf(spark.sql.hive.convertMetastoreParquet, true)
}
In migrating to 1.3.x I see that the
spark.sql.hive.convertMetastoreParquet is no longer public, so the above no
longer works.
I can define a helper method that wraps the HiveMetastoreTypes something
like:
package org.apache.spark.sql.hive
import org.apache.spark.sql.types.DataType
/**
* Helper to expose HiveMetastoreTypes hidden by Spark. It is created in
this name space to make it accessible.
*/
object HiveTypeHelper {
def toDataType(metastoreType: String): DataType =
HiveMetastoreTypes.toDataType(metastoreType)
def toMetastoreType(dataType: DataType): String =
HiveMetastoreTypes.toMetastoreType(dataType)
}
While this will work, is there a better way to achieve this under 1.3.x?
TIA for the assistance.
-Todd
