Repository: spark Updated Branches: refs/heads/master ab3302895 -> c20916a5d
[SPARK-25073][YARN] AM and Executor Memory validation message is not proper while submitting spark yarn application **## What changes were proposed in this pull request?** When the yarn.nodemanager.resource.memory-mb or yarn.scheduler.maximum-allocation-mb memory assignment is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb even though in message it shows the memory value of yarn.nodemanager.resource.memory-mb parameter,As the error Message is bit misleading to the user we can modify the same, We can keep the error message same as executor memory validation message. Defintion of **yarn.nodemanager.resource.memory-mb:** Amount of physical memory, in MB, that can be allocated for containers. It means the amount of memory YARN can utilize on this node and therefore this property should be lower then the total memory of that machine. **yarn.scheduler.maximum-allocation-mb:** It defines the maximum memory allocation available for a container in MB it means RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more than total allocated memory of the Node. **## How was this patch tested?** Manually tested in hdfs-Yarn clustaer Closes #22199 from sujith71955/maste_am_log. Authored-by: s71955 <sujithchacko.2...@gmail.com> Signed-off-by: Sean Owen <sean.o...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c20916a5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c20916a5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c20916a5 Branch: refs/heads/master Commit: c20916a5dc4a7e771463838e797cb944569f6259 Parents: ab33028 Author: s71955 <sujithchacko.2...@gmail.com> Authored: Fri Aug 24 08:58:19 2018 -0500 Committer: Sean Owen <sean.o...@databricks.com> Committed: Fri Aug 24 08:58:19 2018 -0500 ---------------------------------------------------------------------- .../yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c20916a5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---------------------------------------------------------------------- diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 75614a4..698fc2c 100644 --- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -344,7 +344,8 @@ private[spark] class Client( if (amMem > maxMem) { throw new IllegalArgumentException(s"Required AM memory ($amMemory" + s"+$amMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " + - "Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.") + "Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " + + "'yarn.nodemanager.resource.memory-mb'.") } logInfo("Will allocate AM container, with %d MB memory including %d MB overhead".format( amMem, --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
