lisancao opened a new issue, #137: URL: https://github.com/apache/spark-docker/issues/137
Since Spark 4.2.0, regular Python UDFs are Arrow-optimized by default ([SPARK-54555](https://issues.apache.org/jira/browse/SPARK-54555); `spark.sql.execution.pythonUDF.arrow.enabled` now defaults to `true`). The `python3` images only install `python3` and `python3-pip` (`r-python.template`), so the Python worker has neither PyArrow nor pandas, and a plain `@udf` with no options fails on the unmodified image: ```python from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate() @F.udf("int") def plus_one(x): return x + 1 spark.range(3).select(plus_one("id")).collect() ``` ``` File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3004, in read_udfs import pyarrow as pa ModuleNotFoundError: No module named 'pyarrow' ``` Server: `apache/spark:4.2.0-scala2.13-java21-python3-ubuntu` running `start-connect-server.sh --wait`; client: `pyspark-client==4.2.0` on Python 3.10. Results: | UDF | `apache/spark:4.2.0-scala2.13-java21-python3-ubuntu` | same image + pyarrow and pandas | |---|---|---| | `@udf` (default) | `ModuleNotFoundError: No module named 'pyarrow'` | OK | | `@udf(useArrow=False)` | OK | OK | | `@pandas_udf` | `ModuleNotFoundError: No module named 'pandas'` | OK | Before 4.2 a plain `@udf` didn't need either library, so this only started breaking with the new default. (SPARK-37554 added them to the release-build image, not these.) **Proposal:** install both in the `HAVE_PY` block of `r-python.template` so 4.2.1 and 4.3.0 pick them up: ```dockerfile PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --no-cache-dir "pyarrow>=18.0.0" "pandas>=2.2.0,<3"; \ ``` The floors are PySpark 4.2's own requirements. `PIP_BREAK_SYSTEM_PACKAGES=1` is there because the Ubuntu 26.04 base in #133 refuses a plain system pip install, and `<3` because PySpark warns about pandas 3.x; tested on both jammy and resolute. It adds roughly 320 MB to the image. That said, it's possible my understanding is off, or that there's a better way to handle this (e.g. documenting it instead of growing the image). Open to either. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
