This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
The following commit(s) were added to refs/heads/main by this push:
new 8aa8515c infra: spark 4 already comes with spark connect jar (#2887)
8aa8515c is described below
commit 8aa8515cc204c3a1c5939786a1322b3123ef04b2
Author: Kevin Liu <[email protected]>
AuthorDate: Tue Jan 6 03:38:20 2026 -0500
infra: spark 4 already comes with spark connect jar (#2887)
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Check against the `apache/spark:4.0.1` image. Save a couple MBs of
download :D
```
➜ iceberg-python git:(main) docker run --rm apache/spark:4.0.1 ls -la
/opt/spark/jars | grep -i c
onnect
Unable to find image 'apache/spark:4.0.1' locally
4.0.1: Pulling from apache/spark
fdf67ba0bcdc: Already exists
33c2d1bf8782: Already exists
9a562fdfee59: Already exists
b6953a5f4a05: Already exists
c00fd4799efc: Already exists
bf65a9815f44: Already exists
df261ba86031: Already exists
986913f619c8: Already exists
9c58c89601af: Already exists
4f4fb700ef54: Already exists
9da5037c6883: Already exists
Digest:
sha256:acfe7e06e95dd13aa32ee03c464766900d52fba15f58e421f8e1213ec041bb94
Status: Downloaded newer image for apache/spark:4.0.1
drwxr-xr-x 1 spark spark 2330 Sep 2 03:39 connect-repl
-rw-r--r-- 1 spark spark 16577649 Sep 2 03:39 spark-connect_2.13-4.0.1.jar
```
## Are these changes tested?
## Are there any user-facing changes?
<!-- In the case of user-facing changes, please add the changelog label.
-->
---
dev/spark/Dockerfile | 2 --
1 file changed, 2 deletions(-)
diff --git a/dev/spark/Dockerfile b/dev/spark/Dockerfile
index 8adacab8..4b486c90 100644
--- a/dev/spark/Dockerfile
+++ b/dev/spark/Dockerfile
@@ -20,7 +20,6 @@ FROM apache/spark:${BASE_IMAGE_SPARK_VERSION}
# Dependency versions - keep these compatible
ARG ICEBERG_VERSION=1.10.1
ARG ICEBERG_SPARK_RUNTIME_VERSION=4.0_2.13
-ARG SPARK_VERSION=4.0.1
ARG HADOOP_VERSION=3.4.1
ARG SCALA_VERSION=2.13
ARG AWS_SDK_VERSION=2.24.6
@@ -43,7 +42,6 @@ RUN mkdir -p /home/iceberg/spark-events && \
# Required JAR dependencies
ENV JARS_TO_DOWNLOAD="\
-
org/apache/spark/spark-connect_${SCALA_VERSION}/${SPARK_VERSION}/spark-connect_${SCALA_VERSION}-${SPARK_VERSION}.jar
\
org/apache/iceberg/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}/${ICEBERG_VERSION}/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar
\
org/apache/iceberg/iceberg-aws-bundle/${ICEBERG_VERSION}/iceberg-aws-bundle-${ICEBERG_VERSION}.jar
\
org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
\
