[
https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588851#comment-15588851
]
Peter Cseh commented on OOZIE-2712:
-----------------------------------
Thanks for pointing that out. We've been talking about this and this solution
is not acceptable because:
# it's ugly :)
# Some properties can be merged like the driver classpath, but others, like the
driver-memory shouldn't. If the action contains something like {{
--driver-memory 5g --conf spark.driver.memory=3g }} we sould throw an exception
instead of passing {{--conf spark.driver.memory=5g3g}} to spark.
{{--num-executors}} is also affected, we would merge {{--num-executor 4 --conf
spark.executor.instances=2}} into {{--conf spark.executor.instances=42}} which
is probably not for the best.
I'm revoking my patch, but I'm still open for suggestions how to solve this or
other affected properties that I've missed in Spark documentation.
> Merge Spark opts with two possible options
> -------------------------------------------
>
> Key: OOZIE-2712
> URL: https://issues.apache.org/jira/browse/OOZIE-2712
> Project: Oozie
> Issue Type: Bug
> Reporter: Peter Cseh
> Assignee: Peter Cseh
> Attachments: OOZIE-2712.001.patch
>
>
> Issues like OOZIE-2658 can happen with more properties.
> || --conf || overwritten by ||
> |spark.driver.memory | --driver-memory |
> |spark.driver.extraJavaOptions | --driver-java-options|
> |spark.driver.extraLibraryPath|--driver-library-path |
> |spark.executor.instances|--num-executors|
> |spark.driver.extraClassPath| --driver-class-path|
> We should make sure that one is converted to the other (the current pattern
> is to use --conf versions of these) so we don't run into issues where the
> log4j properties are overwritten. (OOZIE-2705 can be affected by this)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
