spark git commit: [SPARK-15271][MESOS] Allow force pulling executor docker images

[srowen]

Repository: spark
Updated Branches:
  refs/heads/master a2abb583c -> 0869b3a5f


[SPARK-15271][MESOS] Allow force pulling executor docker images

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, 
manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Mesos agents by default will not pull docker images which are cached
locally already. In order to run Spark executors from mutable tags like
`:latest` this commit introduces a Spark setting
(`spark.mesos.executor.docker.forcePullImage`). Setting this flag to
true will tell the Mesos agent to force pull the docker image (default is 
`false` which is consistent with the previous
implementation and Mesos' default
behaviour).

Author: Philipp Hoffmann 

Closes #14348 from philipphoffmann/force-pull-image.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0869b3a5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0869b3a5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0869b3a5

Branch: refs/heads/master
Commit: 0869b3a5f028b64c2da511e70b02ab42f65fc949
Parents: a2abb58
Author: Philipp Hoffmann 
Authored: Tue Jul 26 16:09:10 2016 +0100
Committer: Sean Owen 
Committed: Tue Jul 26 16:09:10 2016 +0100

--
 .../cluster/mesos/MesosClusterScheduler.scala   | 14 ++---
 .../MesosCoarseGrainedSchedulerBackend.scala|  7 ++-
 .../MesosFineGrainedSchedulerBackend.scala  |  7 ++-
 .../mesos/MesosSchedulerBackendUtil.scala   | 20 ---
 ...esosCoarseGrainedSchedulerBackendSuite.scala | 63 
 .../MesosFineGrainedSchedulerBackendSuite.scala |  2 +
 dev/deps/spark-deps-hadoop-2.2  |  2 +-
 dev/deps/spark-deps-hadoop-2.3  |  2 +-
 dev/deps/spark-deps-hadoop-2.4  |  2 +-
 dev/deps/spark-deps-hadoop-2.6  |  2 +-
 dev/deps/spark-deps-hadoop-2.7  |  2 +-
 docs/_config.yml|  2 +-
 docs/running-on-mesos.md| 12 
 pom.xml |  2 +-
 14 files changed, 110 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0869b3a5/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
--
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
index 39b0f4d..1e9644d 100644
--- 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
+++ 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
@@ -537,16 +537,10 @@ private[spark] class MesosClusterScheduler(
   .addAllResources(memResourcesToUse.asJava)
 offer.resources = finalResources.asJava
 
submission.schedulerProperties.get("spark.mesos.executor.docker.image").foreach 
{ image =>
-  val container = taskInfo.getContainerBuilder()
-  val volumes = submission.schedulerProperties
-.get("spark.mesos.executor.docker.volumes")
-.map(MesosSchedulerBackendUtil.parseVolumesSpec)
-  val portmaps = submission.schedulerProperties
-.get("spark.mesos.executor.docker.portmaps")
-.map(MesosSchedulerBackendUtil.parsePortMappingsSpec)
-  MesosSchedulerBackendUtil.addDockerInfo(
-container, image, volumes = volumes, portmaps = portmaps)
-  taskInfo.setContainer(container.build())
+  MesosSchedulerBackendUtil.setupContainerBuilderDockerInfo(
+image,
+submission.schedulerProperties.get,
+taskInfo.getContainerBuilder())
 }
 val queuedTasks = tasks.getOrElseUpdate(offer.offerId, new 
ArrayBuffer[TaskInfo])
 queuedTasks += taskInfo.build()

http://git-wip-us.apache.org/repos/asf/spark/blob/0869b3a5/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
--
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
index 99e6d39..52993ca 100644
--- 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
+++ 

spark git commit: [SPARK-15271][MESOS] Allow force pulling executor docker images

[srowen]

Repository: spark
Updated Branches:
  refs/heads/master dd784a882 -> 978cd5f12


[SPARK-15271][MESOS] Allow force pulling executor docker images

## What changes were proposed in this pull request?

Mesos agents by default will not pull docker images which are cached
locally already. In order to run Spark executors from mutable tags like
`:latest` this commit introduces a Spark setting
`spark.mesos.executor.docker.forcePullImage`. Setting this flag to
true will tell the Mesos agent to force pull the docker image (default is 
`false` which is consistent with the previous
implementation and Mesos' default
behaviour).

## How was this patch tested?

I ran a sample application including this change on a Mesos cluster and 
verified the correct behaviour for both, with and without, force pulling the 
executor image. As expected the image is being force pulled if the flag is set.

Author: Philipp Hoffmann 

Closes #13051 from philipphoffmann/force-pull-image.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/978cd5f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/978cd5f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/978cd5f1

Branch: refs/heads/master
Commit: 978cd5f125eb5a410bad2e60bf8385b11cf1b978
Parents: dd784a8
Author: Philipp Hoffmann 
Authored: Mon Jul 25 20:14:47 2016 +0100
Committer: Sean Owen 
Committed: Mon Jul 25 20:14:47 2016 +0100

--
 .../cluster/mesos/MesosClusterScheduler.scala   | 14 ++---
 .../MesosCoarseGrainedSchedulerBackend.scala|  7 ++-
 .../MesosFineGrainedSchedulerBackend.scala  |  7 ++-
 .../mesos/MesosSchedulerBackendUtil.scala   | 20 ---
 ...esosCoarseGrainedSchedulerBackendSuite.scala | 63 
 .../MesosFineGrainedSchedulerBackendSuite.scala |  2 +
 dev/deps/spark-deps-hadoop-2.2  |  2 +-
 dev/deps/spark-deps-hadoop-2.3  |  2 +-
 dev/deps/spark-deps-hadoop-2.4  |  2 +-
 dev/deps/spark-deps-hadoop-2.6  |  2 +-
 dev/deps/spark-deps-hadoop-2.7  |  2 +-
 docs/_config.yml|  2 +-
 docs/running-on-mesos.md| 12 
 pom.xml |  2 +-
 14 files changed, 110 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/978cd5f1/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
--
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
index 39b0f4d..1e9644d 100644
--- 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
+++ 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
@@ -537,16 +537,10 @@ private[spark] class MesosClusterScheduler(
   .addAllResources(memResourcesToUse.asJava)
 offer.resources = finalResources.asJava
 
submission.schedulerProperties.get("spark.mesos.executor.docker.image").foreach 
{ image =>
-  val container = taskInfo.getContainerBuilder()
-  val volumes = submission.schedulerProperties
-.get("spark.mesos.executor.docker.volumes")
-.map(MesosSchedulerBackendUtil.parseVolumesSpec)
-  val portmaps = submission.schedulerProperties
-.get("spark.mesos.executor.docker.portmaps")
-.map(MesosSchedulerBackendUtil.parsePortMappingsSpec)
-  MesosSchedulerBackendUtil.addDockerInfo(
-container, image, volumes = volumes, portmaps = portmaps)
-  taskInfo.setContainer(container.build())
+  MesosSchedulerBackendUtil.setupContainerBuilderDockerInfo(
+image,
+submission.schedulerProperties.get,
+taskInfo.getContainerBuilder())
 }
 val queuedTasks = tasks.getOrElseUpdate(offer.offerId, new 
ArrayBuffer[TaskInfo])
 queuedTasks += taskInfo.build()

http://git-wip-us.apache.org/repos/asf/spark/blob/978cd5f1/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
--
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 
b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
index 99e6d39..52993ca 100644
--- 
a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
+++