spark git commit: [SPARK-6050] [yarn] Relax matching of vcore count in received containers.
Repository: spark
Updated Branches:
refs/heads/master 582e5a24c -> 6b348d90f
[SPARK-6050] [yarn] Relax matching of vcore count in received containers.
Some YARN configurations return a vcore count for allocated
containers that does not match the requested resource. That means
Spark would always ignore those containers. So relax the the matching
of the vcore count to allow the Spark jobs to run.
Author: Marcelo Vanzin
Closes #4818 from vanzin/SPARK-6050 and squashes the following commits:
991c803 [Marcelo Vanzin] Remove config option, standardize on legacy behavior
(no vcore matching).
8c9c346 [Marcelo Vanzin] Restrict lax matching to vcores only.
3359692 [Marcelo Vanzin] [SPARK-6050] [yarn] Add config option to do lax
resource matching.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6b348d90
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6b348d90
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6b348d90
Branch: refs/heads/master
Commit: 6b348d90f475440c285a4b636134ffa9351580b9
Parents: 582e5a2
Author: Marcelo Vanzin
Authored: Mon Mar 2 16:41:43 2015 -0600
Committer: Thomas Graves
Committed: Mon Mar 2 16:41:43 2015 -0600
--
.../org/apache/spark/deploy/yarn/YarnAllocator.scala | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6b348d90/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
--
diff --git
a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index 12c62a6..55bfbcd 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -290,8 +290,14 @@ private[yarn] class YarnAllocator(
location: String,
containersToUse: ArrayBuffer[Container],
remaining: ArrayBuffer[Container]): Unit = {
+// SPARK-6050: certain Yarn configurations return a virtual core count
that doesn't match the
+// request; for example, capacity scheduler + DefaultResourceCalculator.
So match on requested
+// memory, but use the asked vcore count for matching, effectively
disabling matching on vcore
+// count.
+val matchingResource =
Resource.newInstance(allocatedContainer.getResource.getMemory,
+ resource.getVirtualCores)
val matchingRequests =
amClient.getMatchingRequests(allocatedContainer.getPriority, location,
- allocatedContainer.getResource)
+ matchingResource)
// Match the allocation to a request
if (!matchingRequests.isEmpty) {
@@ -318,7 +324,7 @@ private[yarn] class YarnAllocator(
assert(container.getResource.getMemory >= resource.getMemory)
logInfo("Launching container %s for on host %s".format(containerId,
executorHostname))
- executorIdToContainer(executorId) = container
+ executorIdToContainer(executorId) = container
val containerSet =
allocatedHostToContainersMap.getOrElseUpdate(executorHostname,
new HashSet[ContainerId])
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-6050] [yarn] Relax matching of vcore count in received containers.
Repository: spark
Updated Branches:
refs/heads/branch-1.3 a83b9bbb2 -> 650d1e7fb
[SPARK-6050] [yarn] Relax matching of vcore count in received containers.
Some YARN configurations return a vcore count for allocated
containers that does not match the requested resource. That means
Spark would always ignore those containers. So relax the the matching
of the vcore count to allow the Spark jobs to run.
Author: Marcelo Vanzin
Closes #4818 from vanzin/SPARK-6050 and squashes the following commits:
991c803 [Marcelo Vanzin] Remove config option, standardize on legacy behavior
(no vcore matching).
8c9c346 [Marcelo Vanzin] Restrict lax matching to vcores only.
3359692 [Marcelo Vanzin] [SPARK-6050] [yarn] Add config option to do lax
resource matching.
(cherry picked from commit 6b348d90f475440c285a4b636134ffa9351580b9)
Signed-off-by: Thomas Graves
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/650d1e7f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/650d1e7f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/650d1e7f
Branch: refs/heads/branch-1.3
Commit: 650d1e7fb13545d0d102de9bb6e11ab4f9ef6359
Parents: a83b9bb
Author: Marcelo Vanzin
Authored: Mon Mar 2 16:41:43 2015 -0600
Committer: Thomas Graves
Committed: Mon Mar 2 16:42:02 2015 -0600
--
.../org/apache/spark/deploy/yarn/YarnAllocator.scala | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/650d1e7f/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
--
diff --git
a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index 12c62a6..55bfbcd 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -290,8 +290,14 @@ private[yarn] class YarnAllocator(
location: String,
containersToUse: ArrayBuffer[Container],
remaining: ArrayBuffer[Container]): Unit = {
+// SPARK-6050: certain Yarn configurations return a virtual core count
that doesn't match the
+// request; for example, capacity scheduler + DefaultResourceCalculator.
So match on requested
+// memory, but use the asked vcore count for matching, effectively
disabling matching on vcore
+// count.
+val matchingResource =
Resource.newInstance(allocatedContainer.getResource.getMemory,
+ resource.getVirtualCores)
val matchingRequests =
amClient.getMatchingRequests(allocatedContainer.getPriority, location,
- allocatedContainer.getResource)
+ matchingResource)
// Match the allocation to a request
if (!matchingRequests.isEmpty) {
@@ -318,7 +324,7 @@ private[yarn] class YarnAllocator(
assert(container.getResource.getMemory >= resource.getMemory)
logInfo("Launching container %s for on host %s".format(containerId,
executorHostname))
- executorIdToContainer(executorId) = container
+ executorIdToContainer(executorId) = container
val containerSet =
allocatedHostToContainersMap.getOrElseUpdate(executorHostname,
new HashSet[ContainerId])
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
