lisancao opened a new pull request, #59280: URL: https://github.com/apache/spark/pull/59280
### What changes were proposed in this pull request? This PR installs PyArrow in the PySpark K8s Dockerfile (`resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile`): ```dockerfile pip3 install --break-system-packages --no-cache-dir "pyarrow>=18.0.0" && \ ``` The floor is PySpark's own minimum (`minimum_pyarrow_version`). Only PyArrow, not pandas: that's all the default `@udf` needs, and `@pandas_udf` users opt into pandas explicitly. This follows the discussion on apache/spark-docker#138, where @dongjoon-hyun pointed out that the spark-docker images mirror this Dockerfile, so the change should start here. A SPARK JIRA will follow once my JIRA account is approved. ### Why are the changes needed? Since 4.2.0, regular Python UDFs are Arrow-optimized by default (SPARK-54555). This image ships without PyArrow, so a plain `@udf` fails on it with `ModuleNotFoundError: No module named 'pyarrow'`. ### Does this PR introduce _any_ user-facing change? Yes. Default Python UDFs work on the PySpark image without a custom image. The image grows by about 160 MB (948 MB to 1.11 GB in a local build); installing pandas as well would have added about 140 MB more. ### How was this patch tested? Built the base and PySpark images from this branch's Dockerfiles (eclipse-temurin `25-jre`, Python 3.14) over the 4.3.0-rc1 binary distribution, started a Spark Connect server from each, and ran three UDFs from `pyspark-client` 4.3.0 on Python 3.14: | UDF | before | after | |---|---|---| | `@udf` (default) | `ModuleNotFoundError: No module named 'pyarrow'` | OK | | `@udf(useArrow=False)` | OK | OK | | `@pandas_udf` | `ModuleNotFoundError: No module named 'pandas'` | `ModuleNotFoundError: No module named 'pandas'` (unchanged, by design) | This contribution is my original work, and I license it to the project under the project's open source license. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
