Hi,
we've upgraded from 0.8 to 0.9 and I observe that with the same interpreter
settings,
PySpark no longer works with:
java.io.IOException: Fail to launch python process.
Traceback (most recent call last):
File "/tmp/1615477929423-0/zeppelin_python.py", line 20, in <module>
from py4j.java_gateway import java_import, JavaGateway,
GatewayClient
ModuleNotFoundError: No module named 'py4j'
Comparing logs, I see that for 0.8:
INFO [2021-03-11 15:40:13,565] ({pool-3-thread-5}
PySparkInterpreter.java[createGatewayServerAndStartScript]:265) â
<U+0080><U+0094> pythonExec:
/mnt/conda/envs/zeppelin-pyspark-python3/bin/python
INFO [2021-03-11 15:40:13,585] ({pool-3-thread-5}
PySparkInterpreter.java[setupPySparkEnv]:236) â<U+0080><U+0094> PYTHONPATH:
/usr/lib/spark/python/lib/pyspark.zip:/usr/lib/spark/python/lib/py4j-0.10.7-src.zip:/mnt/zeppelin-0.8.3-SNAPSHOT/../interpreter/lib/python
Whereas 0.9 logs say:
INFO [2021-03-11 15:52:09,428]
({FIFOScheduler-interpreter_293940413-Worker-1}
PythonInterpreter.java[setupPythonEnv]:212) - PYTHONPATH:
/tmp/1615477929423-0
INFO [2021-03-11 15:52:09,428]
({FIFOScheduler-interpreter_293940413-Worker-1}
PythonInterpreter.java[createGatewayServerAndStartScript]:147) - Launching
Python Process Command: /mnt/conda/envs/zeppelin-pyspark-python3/bin/python
/tmp/1615477929423-0/zeppelin_python.py 10.4.2.199 37753
In other words, looks like 0.9 does not add pyspark zips to PYTHONPATH.
Looking at the history, I see a major refactoring in this area:
https://github.com/apache/zeppelin/commit/0a97446a70f6294a3efb071bb9a70601f885840b
But can't quite understand whether this change in behavour is intentional,
and what additional options I might need to set. Does anybody have any
suggestions?
--
Vladimir Prus
http://vladimirprus.com
