spark git commit: [SPARK-22994][K8S] Use a single image for all Spark containers.

2018-01-11 Thread vanzin 
Repository: spark
Updated Branches:
  refs/heads/branch-2.3 f624850fe -> b94debd2b


[SPARK-22994][K8S] Use a single image for all Spark containers.

This change allows a user to submit a Spark application on kubernetes
having to provide a single image, instead of one image for each type
of container. The image's entry point now takes an extra argument that
identifies the process that is being started.

The configuration still allows the user to provide different images
for each container type if they so desire.

On top of that, the entry point was simplified a bit to share more
code; mainly, the same env variable is used to propagate the user-defined
classpath to the different containers.

Aside from being modified to match the new behavior, the
'build-push-docker-images.sh' script was renamed to 'docker-image-tool.sh'
to more closely match its purpose; the old name was a little awkward
and now also not entirely correct, since there is a single image. It
was also moved to 'bin' since it's not necessarily an admin tool.

Docs have been updated to match the new behavior.

Tested locally with minikube.

Author: Marcelo Vanzin 

Closes #20192 from vanzin/SPARK-22994.

(cherry picked from commit 0b2eefb674151a0af64806728b38d9410da552ec)
Signed-off-by: Marcelo Vanzin 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b94debd2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b94debd2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b94debd2

Branch: refs/heads/branch-2.3
Commit: b94debd2b01b87ef1d2a34d48877e38ade0969e6
Parents: f624850
Author: Marcelo Vanzin 
Authored: Thu Jan 11 10:37:35 2018 -0800
Committer: Marcelo Vanzin 
Committed: Thu Jan 11 10:37:55 2018 -0800

--
 bin/docker-image-tool.sh| 145 ++
 docs/running-on-kubernetes.md   |  58 
 .../org/apache/spark/deploy/k8s/Config.scala|  17 ++-
 .../org/apache/spark/deploy/k8s/Constants.scala |   3 +-
 .../deploy/k8s/InitContainerBootstrap.scala |   1 +
 .../steps/BasicDriverConfigurationStep.scala|   3 +-
 .../cluster/k8s/ExecutorPodFactory.scala|   3 +-
 .../submit/DriverConfigOrchestratorSuite.scala  |  12 +-
 .../BasicDriverConfigurationStepSuite.scala |   4 +-
 .../InitContainerConfigOrchestratorSuite.scala  |   4 +-
 .../cluster/k8s/ExecutorPodFactorySuite.scala   |   4 +-
 .../src/main/dockerfiles/driver/Dockerfile  |  35 -
 .../src/main/dockerfiles/executor/Dockerfile|  35 -
 .../main/dockerfiles/init-container/Dockerfile  |  24 ---
 .../src/main/dockerfiles/spark-base/Dockerfile  |  50 ---
 .../main/dockerfiles/spark-base/entrypoint.sh   |  37 -
 .../src/main/dockerfiles/spark/Dockerfile   |  52 +++
 .../src/main/dockerfiles/spark/entrypoint.sh|  97 
 sbin/build-push-docker-images.sh| 149 ---
 19 files changed, 348 insertions(+), 385 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/b94debd2/bin/docker-image-tool.sh
--
diff --git a/bin/docker-image-tool.sh b/bin/docker-image-tool.sh
new file mode 100755
index 000..0714063
--- /dev/null
+++ b/bin/docker-image-tool.sh
@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script builds and pushes docker images when run from a release of Spark
+# with Kubernetes support.
+
+function error {
+  echo "$@" 1>&2
+  exit 1
+}
+
+if [ -z "${SPARK_HOME}" ]; then
+  SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+fi
+. "${SPARK_HOME}/bin/load-spark-env.sh"
+
+function image_ref {
+  local image="$1"
+  local add_repo="${2:-1}"
+  if [ $add_repo = 1 ] && [ -n "$REPO" ]; then
+image="$REPO/$image"
+  fi
+  if [ -n "$TAG" ]; then
+image="$image:$TAG"
+  fi
+  echo "$image"
+}
+
+function build {
+  local BUILD_ARGS
+  local IMG_PATH
+
+  if [ ! -f "$SPARK_HOME/RELEASE" ]; then
+

spark git commit: [SPARK-22994][K8S] Use a single image for all Spark containers.

2018-01-11 Thread vanzin 
Repository: spark
Updated Branches:
  refs/heads/master 6d230dccf -> 0b2eefb67


[SPARK-22994][K8S] Use a single image for all Spark containers.

This change allows a user to submit a Spark application on kubernetes
having to provide a single image, instead of one image for each type
of container. The image's entry point now takes an extra argument that
identifies the process that is being started.

The configuration still allows the user to provide different images
for each container type if they so desire.

On top of that, the entry point was simplified a bit to share more
code; mainly, the same env variable is used to propagate the user-defined
classpath to the different containers.

Aside from being modified to match the new behavior, the
'build-push-docker-images.sh' script was renamed to 'docker-image-tool.sh'
to more closely match its purpose; the old name was a little awkward
and now also not entirely correct, since there is a single image. It
was also moved to 'bin' since it's not necessarily an admin tool.

Docs have been updated to match the new behavior.

Tested locally with minikube.

Author: Marcelo Vanzin 

Closes #20192 from vanzin/SPARK-22994.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0b2eefb6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0b2eefb6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0b2eefb6

Branch: refs/heads/master
Commit: 0b2eefb674151a0af64806728b38d9410da552ec
Parents: 6d230dc
Author: Marcelo Vanzin 
Authored: Thu Jan 11 10:37:35 2018 -0800
Committer: Marcelo Vanzin 
Committed: Thu Jan 11 10:37:35 2018 -0800

--
 bin/docker-image-tool.sh| 145 ++
 docs/running-on-kubernetes.md   |  58 
 .../org/apache/spark/deploy/k8s/Config.scala|  17 ++-
 .../org/apache/spark/deploy/k8s/Constants.scala |   3 +-
 .../deploy/k8s/InitContainerBootstrap.scala |   1 +
 .../steps/BasicDriverConfigurationStep.scala|   3 +-
 .../cluster/k8s/ExecutorPodFactory.scala|   3 +-
 .../submit/DriverConfigOrchestratorSuite.scala  |  12 +-
 .../BasicDriverConfigurationStepSuite.scala |   4 +-
 .../InitContainerConfigOrchestratorSuite.scala  |   4 +-
 .../cluster/k8s/ExecutorPodFactorySuite.scala   |   4 +-
 .../src/main/dockerfiles/driver/Dockerfile  |  35 -
 .../src/main/dockerfiles/executor/Dockerfile|  35 -
 .../main/dockerfiles/init-container/Dockerfile  |  24 ---
 .../src/main/dockerfiles/spark-base/Dockerfile  |  50 ---
 .../main/dockerfiles/spark-base/entrypoint.sh   |  37 -
 .../src/main/dockerfiles/spark/Dockerfile   |  52 +++
 .../src/main/dockerfiles/spark/entrypoint.sh|  97 
 sbin/build-push-docker-images.sh| 149 ---
 19 files changed, 348 insertions(+), 385 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0b2eefb6/bin/docker-image-tool.sh
--
diff --git a/bin/docker-image-tool.sh b/bin/docker-image-tool.sh
new file mode 100755
index 000..0714063
--- /dev/null
+++ b/bin/docker-image-tool.sh
@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script builds and pushes docker images when run from a release of Spark
+# with Kubernetes support.
+
+function error {
+  echo "$@" 1>&2
+  exit 1
+}
+
+if [ -z "${SPARK_HOME}" ]; then
+  SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+fi
+. "${SPARK_HOME}/bin/load-spark-env.sh"
+
+function image_ref {
+  local image="$1"
+  local add_repo="${2:-1}"
+  if [ $add_repo = 1 ] && [ -n "$REPO" ]; then
+image="$REPO/$image"
+  fi
+  if [ -n "$TAG" ]; then
+image="$image:$TAG"
+  fi
+  echo "$image"
+}
+
+function build {
+  local BUILD_ARGS
+  local IMG_PATH
+
+  if [ ! -f "$SPARK_HOME/RELEASE" ]; then
+# Set image build arguments accordingly if this is a source repo and not a 
distribution archive.
+