lisancao commented on PR #59280: URL: https://github.com/apache/spark/pull/59280#issuecomment-6157895837
> So wait @lisancao is this _failing all Python UDFs_ in the container is it just producing an error message and then gracefully falling back? I thought we did auto graceful fallbacks but I could be getting things mixed up. Investigated a bit further. The fallback works fine on classic, but on Connect it fails on the server: the PyArrow check passes on the client (pyspark-client ships PyArrow), but PyArrow isn't installed in this image, so if you use it for your Connect server, the Python worker raises error: No module named 'pyarrow'. The supplement to this, if we don't want to update the docker image (or both) is we could explore a serverside fallback or just make a clearer error for the user advising them to set spark.sql.execution.pythonUDF.arrow.enabled to false if it doesn't have the install -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
