[
https://issues.apache.org/jira/browse/HIVE-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069804#comment-14069804
]
Xuefu Zhang commented on HIVE-7436:
-----------------------------------
[~chengxiang li] I didn't find hive on tez is taking configuration from
tez-site.xml. The main configurations I saw is in HiveConf.java, which are
named more Hive configuration. I assume those can be put in hive-site.xml or
set in command line.
{code}
HIVE_SERVER2_TEZ_DEFAULT_QUEUES("hive.server2.tez.default.queues", ""),
HIVE_SERVER2_TEZ_SESSIONS_PER_DEFAULT_QUEUE("hive.server2.tez.sessions.per.default.queue",
1),
HIVE_SERVER2_TEZ_INITIALIZE_DEFAULT_SESSIONS("hive.server2.tez.initialize.default.sessions",
{code}
Would you mind pointing me to the place where tez-site.xml is processed?
Having a controlled set of configurations in Hive's terms has the benefits of
clarity and simplicity. If we read spark default conf or allow user to specify
any spark configuration in hive-site, it's probably a nightmare for
documentation or explaining strange behaviours because of user setting a
configure that hive never thought of.
Nevertheless, I'm open to processing spark default conf if Tez also does so,
though personally I prefer a control over the set of spark configurations.
{quote}
You're right, SparkContext should be per user session as multi-tenant is
supported in hive. So this means that we can not support changing spark
configurations through hive cli set command.
{quote}
I think this is fine. User can set these configurations in command line (if
admin allows it), but once SparkContext is created upon the first query,
resetting those will not have any effect for subsequent queries until user
disconnects and reconnects. This can be documented.
> Load Spark configuration into Hive driver
> -----------------------------------------
>
> Key: HIVE-7436
> URL: https://issues.apache.org/jira/browse/HIVE-7436
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Chengxiang Li
> Attachments: HIVE-7436-Spark.1.patch, HIVE-7436-Spark.2.patch
>
>
> load Spark configuration into Hive driver, there are 3 ways to setup spark
> configurations:
> # Configure properties in spark configuration file(spark-defaults.conf).
> # Java property.
> # System environment.
> Spark support configuration through system environment just for compatible
> with previous scripts, we won't support in Hive on Spark. Hive on Spark load
> defaults from java properties, then load properties from configuration file,
> and override existed properties.
> configuration steps:
> # Create spark-defaults.conf, and place it in the /etc/spark/conf
> configuration directory.
> please refer to [http://spark.apache.org/docs/latest/configuration.html]
> for configuration of spark-defaults.conf.
> # Create the $SPARK_CONF_DIR environment variable and set it to the location
> of spark-defaults.conf.
> export SPARK_CONF_DIR=/etc/spark/conf
> # Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
> export HADOOP_CLASSPATH=$SPARK_CONF_DIR:$HADOOP_CLASSPATH
> NO PRECOMMIT TESTS. This is for spark-branch only.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
