This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 64ac59d7129 [SPARK-45497][K8S] Add a symbolic link file
`spark-examples.jar` in K8s Docker images
64ac59d7129 is described below
commit 64ac59d71296e631ded97b332660d3db5675623a
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Oct 11 00:35:38 2023 -0700
[SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s
Docker images
### What changes were proposed in this pull request?
This PR aims to add a symbolic link file, `spark-examples.jar`, in the
example jar directory.
```
$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail
-n6
total 1620
drwxr-xr-x 1 root root 4096 Oct 11 04:37 .
drwxr-xr-x 1 root root 4096 Sep 9 02:08 ..
-rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar ->
spark-examples_2.12-3.5.0.jar
```
### Why are the changes needed?
Like PySpark example (`pi.py`), we can submit the examples without
considering the version numbers which was painful before.
```
bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
...
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 10000
```
The following is the driver pod log.
```
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
--deploy-mode client
--properties-file /opt/spark/conf/spark.properties
--class org.apache.spark.examples.SparkPi
local:///opt/spark/examples/jars/spark-examples.jar 10000
Files local:///opt/spark/examples/jars/spark-examples.jar from
/opt/spark/examples/jars/spark-examples.jar to
/opt/spark/work-dir/./spark-examples.jar
```
### Does this PR introduce _any_ user-facing change?
No, this is an additional file.
### How was this patch tested?
Manually build the docker image and do `ls`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43324 from dongjoon-hyun/SPARK-45497.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index 02559b8de2d..b80e72c768c 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))
/opt/spark/examples/jars/spark-examples.jar
COPY kubernetes/tests /opt/spark/tests
COPY data /opt/spark/data
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
