[ https://issues.apache.org/jira/browse/HIVE-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079007#comment-14079007 ]
Chengxiang Li commented on HIVE-7436: ------------------------------------- [~xuefuz] HADOOP_CONF_DIR is added to HADOOP_CLASSPATH in hadoop-config.sh, so as HIVE_CONF_DIR in hive-config.sh. if we only load spark configuration file from classpath, there are 2 choices: # export SPARK_CONF_DIR, and add it to HADOOP_CLASSPATH manually. # commit a patch which would add SPARK_CONF_DIR to HADOOP_CLASSPATH in hive scripts(such as hive-config.sh). export SPARK_CONF_DIR. my concern about supporting load spark configuration file from SPARK_CONF_DIR in implementation level is that: # HADOOP/HIVE/HIVE on TEZ only load configuration file from classpath actually. # it may introduce more complexity, like what should we do if different spark configuration file available on SPARK_CONF_DIR and HADOOP_CLASSPATH both? The way how to configure Hive on Tez is similar as current Hive on Spark. [Hive on Tez Configuration|http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.2/bk_installing_manually_book/content/rpm-chap-tez_configure_tez.html] > Load Spark configuration into Hive driver > ----------------------------------------- > > Key: HIVE-7436 > URL: https://issues.apache.org/jira/browse/HIVE-7436 > Project: Hive > Issue Type: Sub-task > Components: Spark > Reporter: Chengxiang Li > Assignee: Chengxiang Li > Fix For: spark-branch > > Attachments: HIVE-7436-Spark.1.patch, HIVE-7436-Spark.2.patch, > HIVE-7436-Spark.3.patch > > > load Spark configuration into Hive driver, there are 3 ways to setup spark > configurations: > # Java property. > # Configure properties in spark configuration file(spark-defaults.conf). > # Hive configuration file(hive-site.xml). > The below configuration has more priority, and would overwrite previous > configuration with the same property name. > Please refer to [http://spark.apache.org/docs/latest/configuration.html] for > all configurable properties of spark, and you can configure spark > configuration in Hive through following ways: > # Configure through spark configuration file. > #* Create spark-defaults.conf, and place it in the /etc/spark/conf > configuration directory. configure properties in spark-defaults.conf in java > properties format. > #* Create the $SPARK_CONF_DIR environment variable and set it to the location > of spark-defaults.conf. > export SPARK_CONF_DIR=/etc/spark/conf > #* Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable. > export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH > # Configure through hive configuration file. > #* edit hive-site.xml in hive conf directory, configure properties in > spark-defaults.conf in xml format. > Hive driver default spark properties: > ||name||default value||description|| > |spark.master|local|Spark master url.| > |spark.app.name|Hive on Spark|Default Spark application name.| > NO PRECOMMIT TESTS. This is for spark-branch only. -- This message was sent by Atlassian JIRA (v6.2#6252)