This is an automated email from the ASF dual-hosted git repository.
tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c46c067 [SPARK-30942] Fix the warning for requiring cores to be
limiting resources
c46c067 is described below
commit c46c067f39213df9b3ee5a51e7d7803b867a0d54
Author: Thomas Graves <tgra...@nvidia.com>
AuthorDate: Tue Feb 25 10:55:56 2020 -0600
[SPARK-30942] Fix the warning for requiring cores to be limiting resources
### What changes were proposed in this pull request?
fix the warning for limiting resources when we don't know the number of
executor cores. The issue is that there are places in the Spark code that use
cores/task cpus to calculate slots and until the entire Stage level scheduling
feature is in, we have to rely on the cores being the limiting resource.
Change the check to only warn when custom resources are specified.
### Why are the changes needed?
fix the check and warn when we should
### Does this PR introduce any user-facing change?
A warning is printed
### How was this patch tested?
manually tested spark-shell with standalone mode, yarn, local mode.
Closes #27686 from tgravescs/SPARK-30942.
Authored-by: Thomas Graves <tgra...@nvidia.com>
Signed-off-by: Thomas Graves <tgra...@apache.org>
---
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
.../src/main/scala/org/apache/spark/resource/ResourceProfile.scala | 7 +++----
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index a47136e..f377f13 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -2798,7 +2798,7 @@ object SparkContext extends Logging {
defaultProf.maxTasksPerExecutor(sc.conf) < cpuSlots) {
throw new IllegalArgumentException("The number of slots on an executor
has to be " +
"limited by the number of cores, otherwise you waste resources and "
+
- "dynamic allocation doesn't work properly. Your configuration has " +
+ "some scheduling doesn't work properly. Your configuration has " +
s"core/task cpu slots = ${cpuSlots} and " +
s"${limitingResource} = " +
s"${defaultProf.maxTasksPerExecutor(sc.conf)}. Please adjust your
configuration " +
diff --git
a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
index 2608ab9..5b2476c 100644
--- a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
+++ b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
@@ -168,7 +168,7 @@ class ResourceProfile(
// limiting resource because the scheduler code uses that for slots
throw new IllegalArgumentException("The number of slots on an
executor has to be " +
"limited by the number of cores, otherwise you waste resources
and " +
- "dynamic allocation doesn't work properly. Your configuration
has " +
+ "some scheduling doesn't work properly. Your configuration has "
+
s"core/task cpu slots = ${taskLimit} and " +
s"${execReq.resourceName} = ${numTasks}. " +
"Please adjust your configuration so that all resources require
same number " +
@@ -183,12 +183,11 @@ class ResourceProfile(
"no corresponding task resource request was specified.")
}
}
- if(!shouldCheckExecCores && Utils.isDynamicAllocationEnabled(sparkConf)) {
+ if(!shouldCheckExecCores && execResourceToCheck.nonEmpty) {
// if we can't rely on the executor cores config throw a warning for user
logWarning("Please ensure that the number of slots available on your " +
"executors is limited by the number of cores to task cpus and not
another " +
- "custom resource. If cores is not the limiting resource then dynamic "
+
- "allocation will not work properly!")
+ "custom resource.")
}
if (taskResourcesToCheck.nonEmpty) {
throw new SparkException("No executor resource configs were not
specified for the " +
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
