Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12115#discussion_r58538995
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -819,12 +819,22 @@ private[spark] class Client(
javaOpts += "-XX:CMSIncrementalDutyCycle=10"
}
- // Include driver-specific java options if we are launching a driver
- if (isClusterMode) {
--- End diff --
For client mode I would rather move this check somewhere more generic in
order to keep it consistent across cluster managers. Right now (without this
patch) it doesn't error if you specify Xmx in the java options running in yarn
client mode, the same thing is true for standalone mode.
So I think we should either just change this back and only check in cluster
mode to keep previous behavior (and its consistent with standalone mode) or we
should move the check into SparkClassCommandBuilder and
SparkSubmitCommandBuilder so they are applied for all cluster managers and it
would check before it ever launches them.
I'm leaning towards the latter and move them into the CommandBuilder
classes. @dhruve can you look into adding a check there?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
