This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 61d9946 [SPARK-26729][K8S] Make image names under test configurable
61d9946 is described below
commit 61d99462a0366b1a9ef3ef966673717a17d2d533
Author: Rob Vesse <rve...@dotnetrdf.org>
AuthorDate: Wed Mar 20 14:28:27 2019 -0700
[SPARK-26729][K8S] Make image names under test configurable
## What changes were proposed in this pull request?
Allow specifying system properties to customise the image names for the
images used in the integration testing. Useful if your CI/CD pipeline or
policy requires using a different naming format.
This is one part of addressing SPARK-26729, I plan to have a follow up
patch that will also make the names configurable when using
`docker-image-tool.sh`
## How was this patch tested?
Ran integration tests against custom images generated by our CI/CD pipeline
that do not follow Spark's existing hardcoded naming conventions using the new
system properties to override the image names appropriately:
```
mvn clean integration-test -pl
:spark-kubernetes-integration-tests_${SCALA_VERSION} \
-Pkubernetes -Pkubernetes-integration-tests \
-P${SPARK_HADOOP_PROFILE} -Dhadoop.version=${HADOOP_VERSION} \
-Dspark.kubernetes.test.sparkTgz=${TARBALL} \
-Dspark.kubernetes.test.imageTag=${TAG} \
-Dspark.kubernetes.test.imageRepo=${REPO} \
-Dspark.kubernetes.test.namespace=${K8S_NAMESPACE} \
-Dspark.kubernetes.test.kubeConfigContext=${K8S_CONTEXT} \
-Dspark.kubernetes.test.deployMode=${K8S_TEST_DEPLOY_MODE} \
-Dspark.kubernetes.test.jvmImage=apache-spark \
-Dspark.kubernetes.test.pythonImage=apache-spark-py \
-Dspark.kubernetes.test.rImage=apache-spark-r \
-Dtest.include.tags=k8s
...
[INFO] --- scalatest-maven-plugin:1.0:test (integration-test)
spark-kubernetes-integration-tests_2.12 ---
Discovery starting.
Discovery completed in 230 milliseconds.
Run starting. Expected test count is: 15
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
Run completed in 8 minutes, 33 seconds.
Total number of tests run: 15
Suites: completed 2, aborted 0
Tests: succeeded 15, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
Closes #23846 from rvesse/SPARK-26729.
Authored-by: Rob Vesse <rve...@dotnetrdf.org>
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
---
.../kubernetes/integration-tests/README.md | 37 ++++++++++++++++++++--
.../dev/dev-run-integration-tests.sh | 31 ++++++++++++++++++
.../kubernetes/integration-tests/pom.xml | 3 ++
.../k8s/integrationtest/KubernetesSuite.scala | 6 ++--
.../deploy/k8s/integrationtest/TestConstants.scala | 3 ++
5 files changed, 75 insertions(+), 5 deletions(-)
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index 73fc058..57c26e1 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -28,7 +28,7 @@ The main useful options are outlined below.
## Using a different backend
The integration test backend i.e. the K8S cluster used for testing is
controlled by the `--deploy-mode` option. By
-default this is set to `minikube`, the available backends are their
perequisites are as follows.
+default this is set to `minikube`, the available backends are their
prerequisites are as follows.
### `minikube`
@@ -46,7 +46,7 @@ environment variable appropriately.
### `cloud`
-These cloud backend configures the tests to use an arbitrary Kubernetes
cluster running in the cloud or otherwise.
+The cloud backend configures the tests to use an arbitrary Kubernetes cluster
running in the cloud or otherwise.
The `cloud` backend auto-configures the cluster to use from your K8S config
file, this is assumed to be `~/.kube/config`
unless the `KUBECONFIG` environment variable is set to override this location.
By default this will use whatever your
@@ -67,6 +67,18 @@ image tag that you have built by other means already, pass
the tag to the test s
where if you still want to use images that were built before by the test
framework:
dev/dev-run-integration-tests.sh --image-tag $(cat target/imageTag.txt)
+
+### Customising the Image Names
+
+If your image names do not follow the standard Spark naming convention -
`spark`, `spark-py` and `spark-r` - then you can customise the names using
several options.
+
+If you use the same basic pattern but a different prefix for the name e.g.
`apache-spark` you can just set `--base-image-name <base-name>` e.g.
+
+ dev/dev-run-integration-tests.sh --base-image-name apache-spark
+
+Alternatively if you use completely custom names then you can set each
individually via the `--jvm-image-name <name>`, `--python-image-name <name>`
and `--r-image-name <name>` arguments e.g.
+
+ dev/dev-run-integration-tests.sh --jvm-image-name jvm-spark
--python-image-name pyspark --r-image-name sparkr
## Spark Distribution Under Test
@@ -190,6 +202,27 @@ to the wrapper scripts and using the wrapper scripts will
simply set these appro
<td><code>docker.io/kubespark</code></td>
</tr>
<tr>
+ <td><code>spark.kubernetes.test.jvmImage</code></td>
+ <td>
+ The image name for the JVM based Spark image to test
+ </td>
+ <td><code>spark</code></td>
+ </tr>
+ <tr>
+ <td><code>spark.kubernetes.test.pythonImage</code></td>
+ <td>
+ The image name for the Python based Spark image to test
+ </td>
+ <td><code>spark-py</code></td>
+ </tr>
+ <tr>
+ <td><code>spark.kubernetes.test.rImage</code></td>
+ <td>
+ The image name for the R based Spark image to test
+ </td>
+ <td><code>spark-r</code></td>
+ </tr>
+ <tr>
<td><code>spark.kubernetes.test.namespace</code></td>
<td>
A specific Kubernetes namespace to run the tests in. If specified then
the tests assume that this namespace
diff --git
a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
index 68f284c..fc17ce6 100755
---
a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
+++
b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
@@ -23,6 +23,10 @@ DEPLOY_MODE="minikube"
IMAGE_REPO="docker.io/kubespark"
SPARK_TGZ="N/A"
IMAGE_TAG="N/A"
+BASE_IMAGE_NAME=
+JVM_IMAGE_NAME=
+PYTHON_IMAGE_NAME=
+R_IMAGE_NAME=
SPARK_MASTER=
NAMESPACE=
SERVICE_ACCOUNT=
@@ -79,6 +83,22 @@ while (( "$#" )); do
EXCLUDE_TAGS="$2"
shift
;;
+ --base-image-name)
+ BASE_IMAGE_NAME="$2"
+ shift
+ ;;
+ --jvm-image-name)
+ JVM_IMAGE_NAME="$2"
+ shift
+ ;;
+ --python-image-name)
+ PYTHON_IMAGE_NAME="$2"
+ shift
+ ;;
+ --r-image-name)
+ R_IMAGE_NAME="$2"
+ shift
+ ;;
*)
break
;;
@@ -119,4 +139,15 @@ then
properties=( ${properties[@]} -Dtest.exclude.tags=$EXCLUDE_TAGS )
fi
+BASE_IMAGE_NAME=${BASE_IMAGE_NAME:-spark}
+JVM_IMAGE_NAME=${JVM_IMAGE_NAME:-${BASE_IMAGE_NAME}}
+PYTHON_IMAGE_NAME=${PYTHON_IMAGE_NAME:-${BASE_IMAGE_NAME}-py}
+R_IMAGE_NAME=${R_IMAGE_NAME:-${R_IMAGE_NAME}-r}
+
+properties+=(
+ -Dspark.kubernetes.test.jvmImage=$JVM_IMAGE_NAME
+ -Dspark.kubernetes.test.pythonImage=$PYTHON_IMAGE_NAME
+ -Dspark.kubernetes.test.rImage=$R_IMAGE_NAME
+)
+
$TEST_ROOT_DIR/build/mvn integration-test -f $TEST_ROOT_DIR/pom.xml -pl
resource-managers/kubernetes/integration-tests -am -Pscala-$SCALA_VERSION
-Pkubernetes -Pkubernetes-integration-tests ${properties[@]}
diff --git a/resource-managers/kubernetes/integration-tests/pom.xml
b/resource-managers/kubernetes/integration-tests/pom.xml
index 91ef0e2..2248482 100644
--- a/resource-managers/kubernetes/integration-tests/pom.xml
+++ b/resource-managers/kubernetes/integration-tests/pom.xml
@@ -148,6 +148,9 @@
<spark.kubernetes.test.master>${spark.kubernetes.test.master}</spark.kubernetes.test.master>
<spark.kubernetes.test.namespace>${spark.kubernetes.test.namespace}</spark.kubernetes.test.namespace>
<spark.kubernetes.test.serviceAccountName>${spark.kubernetes.test.serviceAccountName}</spark.kubernetes.test.serviceAccountName>
+
<spark.kubernetes.test.jvmImage>${spark.kubernetes.test.jvmImage}</spark.kubernetes.test.jvmImage>
+
<spark.kubernetes.test.pythonImage>${spark.kubernetes.test.pythonImage}</spark.kubernetes.test.pythonImage>
+
<spark.kubernetes.test.rImage>${spark.kubernetes.test.rImage}</spark.kubernetes.test.rImage>
</systemProperties>
<tagsToExclude>${test.exclude.tags}</tagsToExclude>
<tagsToInclude>${test.include.tags}</tagsToInclude>
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index 16b8714..91419e8 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -116,9 +116,9 @@ class KubernetesSuite extends SparkFunSuite
sparkHomeDir = Paths.get(sparkDirProp)
require(sparkHomeDir.toFile.isDirectory,
s"No directory found for spark home specified at $sparkHomeDir.")
- image = testImageRef("spark")
- pyImage = testImageRef("spark-py")
- rImage = testImageRef("spark-r")
+ image = testImageRef(sys.props.getOrElse(CONFIG_KEY_IMAGE_JVM, "spark"))
+ pyImage = testImageRef(sys.props.getOrElse(CONFIG_KEY_IMAGE_PYTHON,
"spark-py"))
+ rImage = testImageRef(sys.props.getOrElse(CONFIG_KEY_IMAGE_R, "spark-r"))
val scalaVersion = scala.util.Properties.versionNumberString
.split("\\.")
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/TestConstants.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/TestConstants.scala
index ecc4df7..2b1fd08 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/TestConstants.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/TestConstants.scala
@@ -26,6 +26,9 @@ object TestConstants {
val CONFIG_KEY_KUBE_MASTER_URL = "spark.kubernetes.test.master"
val CONFIG_KEY_KUBE_NAMESPACE = "spark.kubernetes.test.namespace"
val CONFIG_KEY_KUBE_SVC_ACCOUNT = "spark.kubernetes.test.serviceAccountName"
+ val CONFIG_KEY_IMAGE_JVM = "spark.kubernetes.test.jvmImage"
+ val CONFIG_KEY_IMAGE_PYTHON = "spark.kubernetes.test.pythonImage"
+ val CONFIG_KEY_IMAGE_R = "spark.kubernetes.test.rImage"
val CONFIG_KEY_IMAGE_TAG = "spark.kubernetes.test.imageTag"
val CONFIG_KEY_IMAGE_TAG_FILE = "spark.kubernetes.test.imageTagFile"
val CONFIG_KEY_IMAGE_REPO = "spark.kubernetes.test.imageRepo"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
