[
https://issues.apache.org/jira/browse/HIVE-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079007#comment-14079007
]
Chengxiang Li commented on HIVE-7436:
-------------------------------------
[~xuefuz] HADOOP_CONF_DIR is added to HADOOP_CLASSPATH in hadoop-config.sh, so
as HIVE_CONF_DIR in hive-config.sh. if we only load spark configuration file
from classpath, there are 2 choices:
# export SPARK_CONF_DIR, and add it to HADOOP_CLASSPATH manually.
# commit a patch which would add SPARK_CONF_DIR to HADOOP_CLASSPATH in hive
scripts(such as hive-config.sh). export SPARK_CONF_DIR.
my concern about supporting load spark configuration file from SPARK_CONF_DIR
in implementation level is that:
# HADOOP/HIVE/HIVE on TEZ only load configuration file from classpath actually.
# it may introduce more complexity, like what should we do if different spark
configuration file available on SPARK_CONF_DIR and HADOOP_CLASSPATH both?
The way how to configure Hive on Tez is similar as current Hive on Spark. [Hive
on Tez
Configuration|http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.2/bk_installing_manually_book/content/rpm-chap-tez_configure_tez.html]
> Load Spark configuration into Hive driver
> -----------------------------------------
>
> Key: HIVE-7436
> URL: https://issues.apache.org/jira/browse/HIVE-7436
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Chengxiang Li
> Fix For: spark-branch
>
> Attachments: HIVE-7436-Spark.1.patch, HIVE-7436-Spark.2.patch,
> HIVE-7436-Spark.3.patch
>
>
> load Spark configuration into Hive driver, there are 3 ways to setup spark
> configurations:
> # Java property.
> # Configure properties in spark configuration file(spark-defaults.conf).
> # Hive configuration file(hive-site.xml).
> The below configuration has more priority, and would overwrite previous
> configuration with the same property name.
> Please refer to [http://spark.apache.org/docs/latest/configuration.html] for
> all configurable properties of spark, and you can configure spark
> configuration in Hive through following ways:
> # Configure through spark configuration file.
> #* Create spark-defaults.conf, and place it in the /etc/spark/conf
> configuration directory. configure properties in spark-defaults.conf in java
> properties format.
> #* Create the $SPARK_CONF_DIR environment variable and set it to the location
> of spark-defaults.conf.
> export SPARK_CONF_DIR=/etc/spark/conf
> #* Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
> export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH
> # Configure through hive configuration file.
> #* edit hive-site.xml in hive conf directory, configure properties in
> spark-defaults.conf in xml format.
> Hive driver default spark properties:
> ||name||default value||description||
> |spark.master|local|Spark master url.|
> |spark.app.name|Hive on Spark|Default Spark application name.|
> NO PRECOMMIT TESTS. This is for spark-branch only.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
