[jira] [Comment Edited] (SPARK-15034) Use the value of spark.sql.warehouse.dir as the warehouse location instead of using hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/SPARK-15034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15698424#comment-15698424
]
Thomas Sebastian edited comment on SPARK-15034 at 11/26/16 7:54 PM:
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location:
/user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark 2.0 in local mode and hive-1.2.1 pointing to hdfs warehouse
location
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs,
instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir",
"hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
---
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default,
returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler:
AlreadyExistsException(message:Database default already exists)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
was (Author: thomastechs):
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location:
/user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark 2.0 in local mode.
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs,
instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir",
"hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
---
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default,
returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler:
AlreadyExistsException(message:Database default already exists)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
> Use the value of spark.sql.warehouse.dir as the warehouse location instead of
> using hive.metastore.warehouse.dir
>
>
> Key: SPARK-15034
> URL: https://issues.apache.org/jira/browse/SPARK-15034
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Reporter: Yin Huai
>Assignee: Yin Huai
> Labels: release_notes, releasenotes
> Fix For: 2.0.0
>
>
> Starting from Spark 2.0, spark.sql.warehouse.dir will be the conf to set
> warehouse location. We will not use hive.metastore.warehouse.dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15034) Use the value of spark.sql.warehouse.dir as the warehouse location instead of using hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/SPARK-15034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15698424#comment-15698424
]
Thomas Sebastian edited comment on SPARK-15034 at 11/26/16 7:53 PM:
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location:
/user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark 2.0 in local mode.
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs,
instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir",
"hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
---
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default,
returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler:
AlreadyExistsException(message:Database default already exists)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
was (Author: thomastechs):
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location:
/user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark in local mode.
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs,
instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir",
"hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
---
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default,
returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler:
AlreadyExistsException(message:Database default already exists)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
> Use the value of spark.sql.warehouse.dir as the warehouse location instead of
> using hive.metastore.warehouse.dir
>
>
> Key: SPARK-15034
> URL: https://issues.apache.org/jira/browse/SPARK-15034
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Reporter: Yin Huai
>Assignee: Yin Huai
> Labels: release_notes, releasenotes
> Fix For: 2.0.0
>
>
> Starting from Spark 2.0, spark.sql.warehouse.dir will be the conf to set
> warehouse location. We will not use hive.metastore.warehouse.dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
