[
https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274831#comment-14274831
]
Chengxiang Li commented on HIVE-9342:
-------------------------------------
Thanks, [~fangxi.yin] bring this up. While we launch spark client with
spark-submit script, it load configurations from 2 ways: spark configuration
file and command line options. Hive on Spark actually write all spark related
configurations into a property file and add it to spark-submit's
--properties-file option. For the 3 executor options you mentioned, there
should already corresponding configurations, like:
# --num-executors -> spark.executor.instances
# --executor-cores -> spark.executor.cores
# --executor-memory -> spark.executor.memory
So theoreticly, you can configure these properties through hive configuration
file or CLI, while it's possible that these configuration does not work in
certain deploy mode due to spark implementation. I think we shoud verify if it
works in yarn-client or yarn-cluster mode first.
> add num-executors / executor-cores / executor-memory option support for hive
> on spark in Yarn mode
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-9342
> URL: https://issues.apache.org/jira/browse/HIVE-9342
> Project: Hive
> Issue Type: Improvement
> Components: spark-branch
> Affects Versions: spark-branch
> Reporter: Pierre Yin
> Priority: Minor
> Labels: spark
> Fix For: spark-branch
>
> Attachments: HIVE-9342.1-spark.patch
>
>
> When I run hive on spark with Yarn mode, I want to control some yarn option,
> such as --num-executors, --executor-cores, --executor-memory.
> We can append these options into argv in SparkClientImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
