This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3d400d333050 [SPARK-46780][K8S][TESTS] Improve SBT K8s IT to skip R image build step if not needed 3d400d333050 is described below commit 3d400d333050f25e729db15817cff6940364ccce Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Fri Jan 19 15:00:37 2024 -0800 [SPARK-46780][K8S][TESTS] Improve SBT K8s IT to skip R image build step if not needed ### What changes were proposed in this pull request? This PR aims to allow the users to skip R image build step in SBT if R tests are excluded. ### Why are the changes needed? `setup-integration-test-env.sh` has this feature already and has been used in Maven K8s IT. https://github.com/apache/spark/blob/39f8e1a5953b5897f893151d24dc585a80c0c8a0/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh#L119-L123 This PR aims to improve SBT logic to match the behavior by skipping SparkR test if it's not needed. ### Does this PR introduce _any_ user-facing change? No. This is a developer feature. ### How was this patch tested? Manually tests. **COMMAND** ``` $ build/sbt \ -Pkubernetes \ -Pkubernetes-integration-tests \ -Dtest.exclude.tags=minikube,local,r \ -Dspark.kubernetes.test.deployMode=docker-desktop \ "kubernetes-integration-tests/test" ``` **BEFORE** ``` $ docker images | grep spark kubespark/spark-r dev 443f71dfbf87 17 seconds ago 1.53GB kubespark/spark-py dev 82e3d4a859f6 About a minute ago 1.07GB kubespark/spark dev 2eefd2785313 2 minutes ago 716MB ``` **AFTER** ``` $ docker images | grep spark kubespark/spark-py dev 155ed5f5dd95 10 seconds ago 1.07GB kubespark/spark dev 3edf056c5b33 About a minute ago 716MB ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44807 from dongjoon-hyun/SPARK-46780. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- project/SparkBuild.scala | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 7d399a735888..3123f931cd88 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -995,8 +995,12 @@ object KubernetesIntegrationTests { s"$sparkHome/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile") val pyDockerFile = sys.props.getOrElse("spark.kubernetes.test.pyDockerFile", s"$bindingsDir/python/Dockerfile") - val rDockerFile = sys.props.getOrElse("spark.kubernetes.test.rDockerFile", + var rDockerFile = sys.props.getOrElse("spark.kubernetes.test.rDockerFile", s"$bindingsDir/R/Dockerfile") + val excludeTags = sys.props.getOrElse("test.exclude.tags", "").split(",") + if (excludeTags.exists(_.equalsIgnoreCase("r"))) { + rDockerFile = "" + } val extraOptions = if (javaImageTag.isDefined) { Seq("-b", s"java_image_tag=$javaImageTag") } else { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org