Hi all, I've read the docs for Spark SQL 2.1.0 but I'm still having issues with the warehouse and related details.
I'm not using Hive proper, so my hive-site.xml consists only of: <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:;databaseName=/mnt/data/spark/metastore_db;create=true</value> I've set "spark.sql.warehouse.dir" in my "spark-defaults.conf", however the location in my catalog doesn't match: scala> spark.conf.get("spark.sql.warehouse.dir") res8: String = file://mnt/data/spark/warehouse scala> spark.conf.get("hive.metastore.warehouse.dir") res9: String = file://mnt/data/spark/warehouse scala> spark.catalog.listDatabases.show(false) +-------+---------------------+-----------------------------+ |name |description |locationUri | +-------+---------------------+-----------------------------+ |default|Default Hive database|file:/home/me/spark-warehouse| +-------+---------------------+-----------------------------+ I've also tried setting "spark.sql.warehouse.dir" to a valid HDFS path to no avail. My application loads both ORC tables and AVRO files (using spark-avro) from HDFS. When I load a table using spark.sql("select * from orc.`my-table-in-hdfs`"), I see WARN ObjectStore: Failed to get database orc, returning NoSuchObjectException. When I load an AVRO file from HDFS using spark.read.avro(filename) , I see WARN DataSource: Error while looking for metadata directory. Any ideas as to what I'm doing wrong? -- Joe Naegele Grier Forensics 410.220.0968 --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org